This is just a quick blog to introduce my beginners guide to soldering video tutorial:
On the off chance that blogspot screws up the video link, this is the permanent YouTube link: http://youtu.be/2d80i5SNXpg
If you're starting out with electronics and finding soldering frustrating, take a look. It's full of the important tips and techniques that make soldering a breeze.
In case you don't get a chance to view the full 40 minutes, here's some top tips from the video (plus a few extra!):
The most essential piece of kit for soldering (other than a soldering iron) is the third hand - a simple device that will hold your wires in place while you solder. Spark Fun do a good one, but there's loads to choose from (just search for electronics, third hand on google).
Wait for the iron to get properly hot before starting
Cover the tip of the iron in solder, then wipe off the excess before using it. This coats it in fresh solder and makes it better at transferring heat.
Use the heat proof sponge that (probably) comes with the iron to wipe off excess solder when it builds up. If you haven't got a sponge, any heat proof cloth will do.
Always 'tin' your wires/components before trying to join them. This is basically coating the parts you want to join in a thin layer of solder, before actually trying to solder them together.
If you accidentally melt the plastic at the end of the wire and it starts to cover the bits you're joining, trim them off and start again. Solder won't stick to plastic - there's no point trying!
Make sure you've got a bit of ventilation, but there's no need to go nuts over it. I make sure I have the door to my study open so smoke doesn't build up in one room, and if it's warm outside I'll open a window.
And the number 1 rule! Heat the components you're joining, not the solder! You might find you have to touch the solder a bit with the iron to get things going, but the key is making sure that the components are hot. If you don't, the solder won't liquidize onto your components, and instead it'll just form annoying big blobs on the iron itself, which is immensely annoying!
Time for second round of the Wendy light prototype. Last time I got things going (described here), with an ugly but functioning prototype of my light based alarm clock.
Today I'm doing a second round, with more of a focus on the aesthetics and usability of the clock. While I don't expect to end up with something that looks pretty yet, I'm aiming to improve the controls and work out the basic template for the fancy looking final version.
The big change is to turn this big chunk of buttons:
Into something that looks less like an eighties arcade game, and more like an elegant alarm clock that a beautiful lady might have in her bedroom.
I start off using a jig saw and a router to carve out the centre of a block of pine, then cut a square sheet of plastic to sit on top of it.
There's a hole cut out for the IR sensor, but the capacative touch sensors will pick up your finger within about half a centimetre, so they can comfortably sit below the plastic.
Tip: I've spent ages trying to find the best way to cut plastic. After much testing, I've found using a Dremmel with an abrasive attachment the best bet. If you do take this approach though, make absolutely sure you wear safety goggles - it will spit small chunks of hot plastic, and some of those will head for your face. Do not get melted plastic in your eyes!
Next up, the touch sensors:
After a bit of experimentation, I removed the chunky cable attachment and soldered on my own little headers; these are less bulky, which is important now that I'm trying to fit things inside a small and more elegant box.
Tip: If you get a hold of some of these sensors, it's worth knowing they seem very sensitive to power fluctuations - mine stopped working once the IR sensor was plugged in. I shoved a big 250uF capacitor across the power terminals which seemed to fix the problem. Other than that, they work pretty well - hook up Gnd and Vcc to those of the Arduino, and the data output (white cable if you haven't remove the socket) to an IO pin, and you'll get a HIGH signal if a finger is within range.
First step for the innards is to mount all the sensors on a couple of strips of wood that'll sit inside the box:
Next, I hook up all the Vcc and Gnd pins to a little bit of strip board. This allows me to easily attach the big capacitor and connect them to the Vcc and Gnd wires that go to the Arduino (along with the data wire from each sensor):
Tip: If your circuits are ever getting too messy, connecting small chunks of circuit together on a piece of strip board like this is a really good technique. You can cut up strip board with a hack saw, or (as with the plastic) using a Dremmel with an abrasive head (though once again - safety goggles!).
Next up, as I'm wanting this to be a little neater, I'm going to stop trailing 6 wires from bed side table to floor, and instead use a Maplins 9-way data cable to connect the switch box to the Arduino:
Data cables are a great way to keep circuitry tidy, and come in either wrapped form (as above), or as ribbon cables. However, due to the tiny wires inside they can be a little tricky to work with, so I generally end up connecting each end to normal solid core prototyping wire; this allows me to use the data cables to bridge large gaps, but still plug things into prototyping boards or Arduinos at each end.
As I'm tight on space, on the box end I'm going to solder the data cable directly to the cables coming out of the sensors (plus another 2 for Gnd and Vcc).
Soldering wires together is a bit tricky to do well, and often results in fairly delicate connections that are easily broken with an accidental tug. A much neater approach (that I used on the Arduino end) is to solder the data cable to a small piece of strip board:
As you can see, this results in a much neater and stronger connection, that can be wrapped in more insulating tape to strengthen it further. Once this is done, you can easily solder normal solid core wire or pin connectors to the other end, making it easy to plug into an Arduino or prototyping board.
A last bit of work in the shed to drill out the holes and make a cover for the base of the switch box:
Attach everything together, along with some button labels (aka masking type with my writing on):
And it's ready to hook up!
Not a lot to show in terms of actual functional changes, though I may get around to a video of me waving my hand over the sensor to snooze the alarm! Other than that:
I've tweaked the reading lights so the lightest setting is lighter (based on user feedback...),
The central LEDs now stay dimmed for anything other than the alarm
The alarm cancel now triggers a small blue pulse effect, similar to the red/green ones for on/off. This is just to give some confirmation that the touch button worked.
So generally happy with prototype number 2. Final version will be fairly similar I think. The box itself I'll probably make out of white oak, then stain the same colour as the wood the bed is made of. Plus of course I'll have to make a proper box for my side of the bed too! Then a less fancy box for the Arduino under the bed; the LEDs fixed behind the wall; probably some small side buttons so you can set the alarm time and things; Printed button labels (ideally glow-in-the-dark). That should just about round off the Wendy light!
After watching Wendy suffer repeatedly in the mornings, I've decided to build a prototype light based alarm clock. The idea being that (as various research / existing products indicate), waking up as a result of changes in light is much nicer than load angry noises.
The system will initially be simple. I'll preset an MR005-001 clock breakout board (containing a DR1307 chip) to the 'correct' time from my PC, and use it to trigger a NeoPixel led strip.
Research as I undersand it suggests that your circadean rythm is largely dictated by ipRGC sensors in your eyes. These are especially sensitive to blue wave lengths, and not very sensitive to red; my clock will feature a red reading light (that automatically fades off after an hour), and a blue wake up light triggered at 8AM.
The initial step was setting up this real-time clock chip:
The clock, along with some stunning soldering (ahem)
It's a DS1307 real time clock break out board, that communicates with the Arduino over I2C. Fortunately, the folks at Adafruit have been good enough to provide a ready made Arduino library to supplement it, so getting the thing going is pretty easy. Here it is wired up:
The clock hooked up to the I2C pins of an Arduino Mega
And running their demo shows the time/date being regularly printed out just as you'd expect.
Next up, is the very funky neopixel LEDs. Also from Adafruit, these awesome full colour LEDs come in various forms (strips, grids, rings) and are great fun to play with. And of course, in true Adafruit style, a ready made Arduino library exists, so they're pretty much plug-and-play.
2 strips of Neopixels running at full whack
A quick point of note - if you do get some Neopixels (and I highly recommend it), it is really important to read the Adafruit guide. The strips work out of the box, but the information supplied about current requirements and protective components is worth knowing before you even think about plugging anything in. Reading documentation isn't my strong point, but after half an hour working out why my Arduino shut down whenever I turned them on, and 2 strips of destroyed LEDs, I begrudgingly read the docs and felt very silly for not doing so!
If you do blow some LEDs, often it'll only be the first few in the strip. Try chopping off the first few and the odds are the rest of your strip will be fine.
Also worth knowing that these LEDs use up a healthy number of amps on full power, so it'll be worth investing in a decent transformer. I'm running 86 LEDs at full brightness off a 4A supply from oomlout, which is doing the job nicely; For similar reasons, make sure your wiring can take enough current, or you'll end up with melted plastic in your project!
The final job is some basic wiring up of chunky switches, and some bodgy carpentry to mount them in a box. With the LEDs fixed to a chunk of door frame edging and stuck on top of the bed, we're all set.
Chunky buttons on Wendy's side of the bed
Inside of the button box
The Arduino and clock hidden under the bed
The LED strip lit up orange in 'reading light' mode
The circuitry is pretty simple, as there's no clever components going on here. Standard switch wiring up, plus the data line for the LEDs and the I2C lines for the clock:
You can see here 4 separate blocks (each in its own little actual physical wooden box). These break down into:
The main clock circuit, consisting of the Arduino and the DS1307 clock.
The lights, with a small protective circuit; a big capacitor to protect against spikes, and a small resistor to protect the data line.
Switch Box 1 (on Wendy's side of the bed). This is just a collection of grounded push switches, each connected to an IO pin on the Arduino.
Switch Box 2 (on my side of the bed). Due to a lack of chunky switches in my collection, this currently only contains a switch for the reading light, but would eventually have the same set as switch box 1.
And last but not least, a video of the clock in action!
Next up, once I've refined the code a little I'll be updating the system to be a little more aesthetic, and probably implement a few handy extra little features like setting the time/alarm.
(update, if after reading this lovely blog you are so excited you want to see more, the next step is here!)
Over the past few days I've been hacking away at the camera module for the raspberry pi. I made a lot of headway creating a simple and nice api for the camera which is detailed here:
And a video of the whole thing (with description of what's going on!)
The api I designed could use mmal for doing colour conversion and downsampling the image but it was pretty slow and got in the way of opengl. However, I deliberately allowed the user to ask the api for the raw YUV camera data. This is provided as a single block of memory, but really contains 3 separate grey scale textures - one containing the 'luminosity' (Y) and another 2 that contain information to specify the colour of a pixel:
I make a few tweaks to my code to generate these 3 textures:
//lock the chosen frame buffer, and copy it into textures
{
const uint8_t* data = (const uint8_t*)frame_data;
int ypitch = MAIN_TEXTURE_WIDTH;
int ysize = ypitch*MAIN_TEXTURE_HEIGHT;
int uvpitch = MAIN_TEXTURE_WIDTH/2;
int uvsize = uvpitch*MAIN_TEXTURE_HEIGHT/2;
int upos = ysize;
int vpos = upos+uvsize;
ytexture.SetPixels(data);
utexture.SetPixels(data+upos);
vtexture.SetPixels(data+vpos);
cam->EndReadFrame(0);
}
And write a very simple shader to convert from yuv to rgb:
Now I simply run the shader to read in the 3 yuv textures, and write out an rgb one, ending up with this little number:
Good hat yes? Well, hat aside, the next thing to do is provide downsamples so we can run image processing algorithms at different levels. I don't even need a new shader for that, as I can just run the earlier shader, but aiming it at successively lower resolution textures. Here's the lowest one now:
The crucial thing is that in opengl you can create a texture, and then tell it to also double as a frame buffer using the following code:
bool GfxTexture::GenerateFrameBuffer()
{
//Create and bind a new frame buffer
glGenFramebuffers(1,&FramebufferId);
check();
glBindFramebuffer(GL_FRAMEBUFFER,FramebufferId);
check();
//point it at the texture (the id passed in is the Id assigned when we created the open gl texture)
glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,Id,0);
check();
//cleanup
glBindFramebuffer(GL_FRAMEBUFFER,0);
check();
return true;
}
Once you have a texture as a frame buffer you can set it to be the target to render to (don't forget to set the viewport as well):
These features give us a massive range of capability. We can now chain together various shaders to apply multiple levels of filtering, and once the gpu is finished with them the data can read to the cpu and fed into image processing applications such as opencv. This is really handy, as algorithms such as object detection often have to do costly filtering before they can operate. Using the gpu as above we can avoid the cpu needing to do the work.
Thus far I've written the following filters:
Gaussian blur
Dilate
Erode
Median
Threshold
Sobel
Here's a few of them in action:
Enjoy!
p.s. my only annoyance right now is that I still have to go through the cpu to get my data from mmal and into opengl. If anyone knows a way of getting from mmal straight to opengl that'd be super awesome!
pp.s. right at the end, here's a tiny shameless advert for my new venture - http://www.happyrobotgames.com/no-stick-shooter. If you like my writing, check out the dev blog for regular updates on my first proper indie title!
For the past few days I've been messing around with my new raspberry pi camera modules (see earlier blog posts for excessive details) and part of that has involved putting together a nice and easy to use api to access the camera in c++ and read its frames. This post is a guide to installation, an overview of the very simple api and a description of the sample application.
One word of caution - as with any tinkering there is always a chance something will go wrong and result in a dead pi. If this worries you, back up first. I didn't bother, but I didn't have anything on there I was worried about losing!
Installation Make sure you're on a recent Raspberry Pi build, and have a working Camera!
I'm assuming at this point you've got a camera module and it's working. If you've not set it up yet you may need to update your raspberry pi (depends when you bought it). I won't go over this process as it's been described 100 times already, but here's a link to get you going just in case:
That should show you the raspberry pi video feed on screen for 10 seconds.
Get CMake
If you haven't already got it, you'll need cmake for building just about anything:
sudo apt-get install cmake
Download and install the latest 'userland-master'
This is the bit of the raspberry pi OS that contains the code for the camera applications and the various libraries they use. At time of writing it isn't supplied as part of the install, so you need to download, build and install it manually. To do so:
Unzip it into your /opt/vc directory. You should now have a folder called /opt/vc/userland-master with various folders in it such as "host_applications" and "interfaces".
Change to the /opt/vc/userland-master folder, then build it with the following commands:
sudo mkdir build cd build sudo cmake -DCMAKE_BUILD_TYPE=Release .. sudo make sudo make install Test everything worked by running raspivid again. You may see some different messages pop up (I got some harmless errors probably due to the build being so recent), but the crucial thing is that you still get the video feed on screen. Download and build the PiCam API/Samples The api and samples can all be downloaded here: http://www.cheerfulprogrammer.com/downloads/picamtutorial/picamdemo.zip Extract them into a folder in your home directory called 'picamdemo'. You should have a few cpp files in there, plus a make file and some shaders. Change to the folder and build the application with: cmake . make Then run the sample with ./picamdemo If all goes well you should see some text like this:
Compiled vertex shader simplevertshader.glsl:
<some shader code here>
Compiled fragment shader simplefragshader.glsl:
<some shader code here>
mmal: mmal_vc_port_parameter_set: failed to set port parameter 64:0:ENOSYS
mmal: Function not implemented
Init camera output with 512/512
Creating pool with 3 buffers of size 1048576
Init camera output with 256/256
Creating pool with 3 buffers of size 262144
Init camera output with 128/128
Creating pool with 3 buffers of size 65536
Init camera output with 64/64
Creating pool with 3 buffers of size 16384
Camera successfully created
Running frame loop
And your tv should start flicking between various resolutions of the camera feed like this:
(Edit - I've had some reports of the blogger you-tube link not working. You can see the full video here on proper you tube: http://www.youtube.com/watch?v=9bWJBSNxeXk)The API (and what it does!)
PiCam is designed to be very simple but also useful for image processing algorithms. Right now it lets you:
Start up the camera with a given width, height and frame rate
Specify a number of 'levels'. More on that later.
Choose whether to automatically convert the camera feed to RGBA format
Basic Initialisation
All this is done just by calling StartCamera and passing in the right parameters. It returns a pointer to a CCamera object as follows:
That's a 512x512 image at 30hz, with 1 level and rgba conversion enabled.
Reading
Once started you can extract frames from the camera by calling ReadFrame and passing in a buffer:
char mybuffer[512*512*4]
mycamera->ReadFrame(0,mybuffer,sizeof(mybuffer));
ReadFrame will return the number of bytes actually read, or -1 if there was an error. An error occurs either when there is no data available or your buffer is not large enough.
In addition to ReadFrame there are 2 functions: BeginReadFrame and EndReadFrame. These slightly more advanced versions are shown in the demo, and allow you to be more efficient by locking the actual camera buffer, using it, then releasing it. Internally ReadFrame is implemented using these functions.
Shutting down
Once done, call 'StopCamera'
Levels
In image processing it is often useful to have your data provided at different resolutions. Expensive operations need to be performed on low res images to run at a good frame rate, but you may still want higher res versions around for other operations or even just showing on screen. The PiCam api will do this for you automatically (for up to 3 additional levels). If we modify the StartCamera call to this:
The system will automatically generate the main image plus an additional 3 down-sampled ones (at half res, quarter res and 1/8th res). These are then accessed by specifying a level other than 0 in the call to ReadFrame (or BeginReadFrame):
mycamera->ReadFrame(0,mybuffer,sizeof(mybuffer)); //get full res frame
mycamera->ReadFrame(1,mybuffer,sizeof(mybuffer)); //get half res frame
mycamera->ReadFrame(2,mybuffer,sizeof(mybuffer)); //get quarter res frame
mycamera->ReadFrame(3,mybuffer,sizeof(mybuffer)); //get 1/8th res frame
RGBA Conversions
For most purposes you'll want the data in a nice friendly RGBA format, however if you actually want the raw YUV data feed from the camera, specify false as the last parameter to StartCamera and no conversions will be done for you.
The demo application
The picamdemo application consists of the core camera code as these files:
camera.h/camera.cpp
cameracontrol.h/cameracontrol.cpp
mmalincludes.h
A very simple opengl graphics api (which you are welcome to use/modify/change in any way you please):
graphics.h/graphics.cpp
And the main demo app itself:
picam.cpp
Which looks like this:
#include <stdio.h>
#include <unistd.h>
#include "camera.h"
#include "graphics.h"
#define MAIN_TEXTURE_WIDTH 512
#define MAIN_TEXTURE_HEIGHT 512
char tmpbuff[MAIN_TEXTURE_WIDTH*MAIN_TEXTURE_HEIGHT*4];
//entry point
int main(int argc, const char **argv)
{
//should the camera convert frame data from yuv to argb automatically?
bool do_argb_conversion = true;
//how many detail levels (1 = just the capture res, >1 goes down by halves, 4 max)
int num_levels = 4;
//init graphics and the camera
InitGraphics();
CCamera* cam = StartCamera(MAIN_TEXTURE_WIDTH, MAIN_TEXTURE_HEIGHT,30,num_levels,do_argb_conversion);
//create 4 textures of decreasing size
GfxTexture textures[4];
for(int texidx = 0; texidx < num_levels; texidx++)
textures[texidx].Create(MAIN_TEXTURE_WIDTH >> texidx, MAIN_TEXTURE_HEIGHT >> texidx);
printf("Running frame loop\n");
for(int i = 0; i < 3000; i++)
{
//pick a level to read based on current frame (flicking through them every 30 frames)
int texidx = (i / 30)%num_levels;
//lock the chosen buffer, and copy it directly into the corresponding texture
const void* frame_data; int frame_sz;
if(cam->BeginReadFrame(texidx,frame_data,frame_sz))
{
if(do_argb_conversion)
{
//if doing argb conversion just copy data directly
textures[texidx].SetPixels(frame_data);
}
else
{
//if not converting argb the data will be the wrong size so copy it in
//via a temporary buffer just so we can observe something happening!
memcpy(tmpbuff,frame_data,frame_sz);
textures[texidx].SetPixels(tmpbuff);
}
cam->EndReadFrame(texidx);
}
//begin frame, draw the texture then end frame (the bit of maths just fits the image to the screen while maintaining aspect ratio)
BeginFrame();
float aspect_ratio = float(MAIN_TEXTURE_WIDTH)/float(MAIN_TEXTURE_HEIGHT);
float screen_aspect_ratio = 1280.f/720.f;
DrawTextureRect(&textures[texidx],-aspect_ratio/screen_aspect_ratio,-1.f,aspect_ratio/screen_aspect_ratio,1.f);
EndFrame();
}
StopCamera();
}
That's the full code for exploiting all the features of the api. It is designed to loop through each detail level and render them in turn. At the top of the main function you will find a couple of variables to enable argb or change level count, and higher up you can see the frame size settings. Questions? Problems? Comments? I'm happy to answer any questions, hear any comments, and if you hit issues I'd like to fix them. Either comment on this blog or email me (wibble82@hotmail.com) with a sensible subject like 'pi cam problem' (so it doesn't go into the junk mail box!). p.s. right at the end, here's a tiny shameless advert for my new venture - http://www.happyrobotgames.com/no-stick-shooter. If you like my writing, check out the dev blog for regular updates on my first proper indie title!
Right, we've got all kinds of bits working but there's another ingredient I need before the system is just about 'functional'. For image processing I need the camera feed at multiple resolutions, so I can do cheap processing operations on high res feeds, and expensive ones on low res feeds. To do this I use the video splitter component, and have reworked my camera api to:
Create 4 separate outputs, each with its own resizer that does the RGB conversion but generates a different resolution.
Output 0 = full res, output 1 = half res etc
You still use ReadFrame or Begin/EndReadFrame, but now you pass in a 'level' as well
Internally the camera code has become a bit more complex to handle this multi output system but it's mostly just rearranging code.
I won't go into the code here as it was lots of tweaks all over the place and not easy to write. Here is a nice image of 2 of the outputs to make it more clear:
As you can see, in the top image I am at full resolution, however in the lower one it's displaying me at (in this case) 1/8th of the upper resolution. Just to demonstrate it is actually getting all the feeds (and the above isn't just from running the app twice!), this video shows it flicking between them live:
Here's the actual application code for the program above:
//entry point
int main(int argc, const char **argv)
{
printf("PI Cam api tester\n");
InitGraphics();
printf("Starting camera\n");
CCamera* cam = StartCamera(MAIN_TEXTURE_WIDTH, MAIN_TEXTURE_HEIGHT,15);
//create 4 textures of decreasing size
GfxTexture textures[4];
for(int texidx = 0; texidx < 4; texidx++)
textures[texidx].Create(MAIN_TEXTURE_WIDTH >> texidx, MAIN_TEXTURE_HEIGHT >> texidx);
printf("Running frame loop\n");
for(int i = 0; i < 3000; i++)
{
//pick a level to read based on current frame (flicking through them every second)
int texidx = (i / 30)%4;
//lock the chosen frame buffer, and copy it directly into the corresponding open gl texture
const void* frame_data; int frame_sz;
if(cam->BeginReadFrame(texidx,frame_data,frame_sz))
{
textures[texidx].SetPixels(frame_data);
cam->EndReadFrame(texidx);
}
//begin frame, draw the texture then end frame
BeginFrame();
DrawTextureRect(&textures[texidx],-0.9f,-0.9f,0.9f,0.9f);
EndFrame();
}
StopCamera();
}
Note the really crucial point is that my app above is just reading 1 of the levels each frame, however they are all available every frame, so if I chose (and the cpu was available) I could do something with every level. That's really key and undoubtedly what I'll need going forwards. Code for the whole thing is here:
In terms of frame rate it has suffered unfortunately. That image resizer really seems to chew through frame time for some reason. Maybe there's lots of copies going on or something else funky, but it is going disappointingly slow. At 1280x720 the frame rate is probably worse than 10hz when reading the hi res data.
Next up I reckon I'll clean up the api a little - give the user options as to what to enable/disable and make sure everything shuts down right and add a post with a little tutorial on its use. Once that's done, onto gpu acceleration land....
I'm making real progress now getting the camera module simpler and more efficient. My next goal is to rework the camera API to be a more synchronous process (no more callbacks) where the user can simply call 'ReadFrame' to get the next frame.
A Simple Syncronous API
The first step turned out to be pretty simple thanks to the 'queue' structure in mmal. I simply create my own little queue called 'OutputQueue' and change the internal camera callback to be:
void CCamera::OnVideoBufferCallback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
//first, add the buffer to the output queue
mmal_queue_put(OutputQueue,buffer);
}
That code used to lock the buffer, call a callback, then return it to the port for recycling. However now it just pushes the buffer into an output list for processing by the user. Next up, I add a 'ReadFrame' function:
int CCamera::ReadFrame(void* dest, int dest_size)
{
//default result is 0 - no data available
int res = 0;
//get buffer
if(MMAL_BUFFER_HEADER_T *buffer = mmal_queue_get(OutputQueue))
{
//check if buffer has data in
if(buffer->length)
{
//got data so check if it'll fit in the memory provided by the user
if(buffer->length <= dest_size)
{
//it'll fit - yay! copy it in and set the result to be the size copied
mmal_buffer_header_mem_lock(buffer);
memcpy(dest,buffer->data,buffer->length);
mmal_buffer_header_mem_unlock(buffer);
res = buffer->length;
}
else
{
//won't fit so set result to -1 to indicate error
res = -1;
}
}
// release buffer back to the pool from whence it came
mmal_buffer_header_release(buffer);
// and send it back to the port (if still open)
if (VideoCallbackPort->is_enabled)
{
MMAL_STATUS_T status;
MMAL_BUFFER_HEADER_T *new_buffer;
new_buffer = mmal_queue_get(BufferPool->queue);
if (new_buffer)
status = mmal_port_send_buffer(VideoCallbackPort, new_buffer);
if (!new_buffer || status != MMAL_SUCCESS)
printf("Unable to return a buffer to the video port\n");
}
}
return res;
}
This gets the next buffer in the output queue, copies it into memory provided by the user, and then returns it back to the port for reuse, just like the old video callback used to do.
It all worked fine first time, so my actual application code is now as simple as:
//this is the buffer my graphics code uses to update the main texture each frame
extern unsigned char GTextureBuffer[4*1280*720];
//entry point
int main(int argc, const char **argv)
{
printf("PI Cam api tester\n");
InitGraphics();
printf("Starting camera\n");
CCamera* cam = StartCamera(1280,720,15);
printf("Running frame loop\n");
for(int i = 0; i < 3000; i++)
{
BeginFrame();
//read next frame into the texture buffer
cam->ReadFrame(GTextureBuffer,sizeof(GTextureBuffer));
//tell graphics code to draw the texture
DrawMainTextureRect(-0.9f,-0.9f,0.9f,0.9f);
EndFrame();
}
StopCamera();
}
As an added benefit, doing it synchronously means I don't accidentally write to the buffer while it's being copied to the texture, so no more screen tearing! Nice!
A bit more efficient
Now that I'm accessing the buffer synchronously there's the opportunity to get things more efficient and remove a frame of lag. Basically the current system goes:
BeginFrame (updates the main texture from GTextureBuffer - effectively a memcpy)
camera->ReadFrame (memcpy latest frame into GTextureBuffer)
DrawMainTextureRect (draws the main texture)
EndFrame (refreshes the screen)
There's 2 problems here. First up, our read frame call is updating GTextureBuffer after its copied into the opengl texture. This means we're always seeing a frame behind, although that could be easily fixed by calling it before BeginFrame. Worse though, we're doing 2 memcpys - first from camera to GTextureBuffer, and then from GTextureBuffer to the opengl texture. With a little reworking of the api however this can be fixed...
First, I add 'BeginReadFrame' and 'EndReadFrame' functions, which effectively do the same as the earlier ReadFrame (minus the memcpy), but split across 2 function calls:
bool CCamera::BeginReadFrame(const void* &out_buffer, int& out_buffer_size)
{
//try and get buffer
if(MMAL_BUFFER_HEADER_T *buffer = mmal_queue_get(OutputQueue))
{
//lock it
mmal_buffer_header_mem_lock(buffer);
//store it
LockedBuffer = buffer;
//fill out the output variables and return success
out_buffer = buffer->data;
out_buffer_size = buffer->length;
return true;
}
//no buffer - return false
return false;
}
void CCamera::EndReadFrame()
{
if(LockedBuffer)
{
// unlock and then release buffer back to the pool from whence it came
mmal_buffer_header_mem_unlock(LockedBuffer);
mmal_buffer_header_release(LockedBuffer);
LockedBuffer = NULL;
// and send it back to the port (if still open)
if (VideoCallbackPort->is_enabled)
{
MMAL_STATUS_T status;
MMAL_BUFFER_HEADER_T *new_buffer;
new_buffer = mmal_queue_get(BufferPool->queue);
if (new_buffer)
status = mmal_port_send_buffer(VideoCallbackPort, new_buffer);
if (!new_buffer || status != MMAL_SUCCESS)
printf("Unable to return a buffer to the video port\n");
}
}
}
The key here is that instead of returning the buffer straight away, I simply store a pointer to it in BeginReadFrame and return the address and size of the data to the user. In EndReadFrame, I then proceed to unlock and release it as normal.
This means my ReadFrame function now changes to:
int CCamera::ReadFrame(void* dest, int dest_size)
{
//default result is 0 - no data available
int res = 0;
//get buffer
const void* buffer; int buffer_len;
if(BeginReadFrame(buffer,buffer_len))
{
if(dest_size >= buffer_len)
{
//got space - copy it in and return size
memcpy(dest,buffer,buffer_len);
res = buffer_len;
}
else
{
//not enough space - return failure
res = -1;
}
EndReadFrame();
}
return res;
}
In itself that's not much help. However, if I make a tweak to the application so it can copy data straight into the opengl texture and switch it to use BeginReadFrame and EndReadFrame I can avoid one of the memcpys. In addition, by moving the camera read earlier in the frame I lose a frame of lag:
//entry point
int main(int argc, const char **argv)
{
printf("PI Cam api tester\n");
InitGraphics();
printf("Starting camera\n");
CCamera* cam = StartCamera(MAIN_TEXTURE_WIDTH, MAIN_TEXTURE_HEIGHT,15);
printf("Running frame loop\n");
for(int i = 0; i < 3000; i++)
{
//lock the current frame buffer, and copy it directly into the open gl texture
const void* frame_data; int frame_sz;
if(cam->BeginReadFrame(frame_data,frame_sz))
{
UpdateMainTextureFromMemory(frame_data);
cam->EndReadFrame();
}
//begin frame, draw the texture then end frame
BeginFrame();
DrawMainTextureRect(-0.9f,-0.9f,0.9f,0.9f);
EndFrame();
}
StopCamera();
}
Much better! Unfortunately I'm still only at 15hz due to the weird interplay between opengl and the mmal resizing/converting components, but it is a totally solid 15hz at 720p - about 10hz at 1080p. I suspect I'm going to have to ditch the mmal resize/convert components eventually and rewrite them as opengl shaders, but not just yet.
Last time round I got to the point at which I was pulling data out of the camera and using opengl to efficiently render it to screen. There's a few performance improvements to be made, but it runs at a respectable frame rate and could hit 30hz at 720p no problem.
Next up it's time to get the camera feed from the native YUV format into RGB ready for image processing. I'll also be needing to get some down sampled versions of the image, as the more expensive image processing algorithms aren't fast enough to run on a hi def video feed. This'll be a fairly breif post though, as it's late and my brain is going to sleep...
The Image Resizer
My first port of call was the image resize component built into mmal (thanks to a handy tip on this post), which uses the hardware resizer to take an image and... well... resize it! However, as a handy side effect it can also convert between YUV and RGB in the process. At this point massive thanks goes out to Mr Jason Van Cleave, who made all the mmal component documentation available on his web site.
So in short, I need to adjust the camera code so it:
Create an image resize component (I eventually worked out its the "vc.ril.resizer" component)
Connect it to the camera's video output port (the one we're currently displaying)
Set it's input format to that of the video output, and the output format to RGBA. We leave the image sizes the same for now though, so its not really doing any resizing - just the conversion
I do a little code cleanup first so it's easier to add to, plonk in the new code and after a few iterations...
Unfortunately on the first attempt performance appeared to be very poor. Interestingly though if I remove the actual rendering of the image it runs fine. This leads me to believe that the image resizer is chomping through most of the gpu time and consequentially I can't render fast enough. This is really annoying as I frankly don't see why it should be so slow - maybe it's just some interplay between opengl and mmal.
I'll know more about the resizer performance once I get multiple resizers running, generating different downsampled images and we'll see what the actual costs are. If necessary I'm fairly confident I could write a shader that did the convert and downsample quite efficiently. I'm now getting 15hz, which I'm not happy about but it'll do for the moment.
A quick restructure
My next goal is to get multiple outputs at different resolutions coming out of the camera. This allows me to analyse the data at different levels in order to pick and choose where I spend my cpu. It should be doable using the 'video splitter' component, but it raises a few problems in terms of my architecture.
Right now the camera code simply runs, then calls a callback for each frame. Once the splitter is running I'll be receiving blocks of data constantly from different sources and will need a nice way of managing this and providing it in an api to the user. As a result, before going onto the multiple output world, I'm going to adjust the camera code so it internally buffers up the frames and allows the user to read them in a syncronous manner. If I can make use of the queuing system in mmal then I should be able to set it up as follows:
The basic idea is that the camera wraps up all the mmal stuff as usual, but rather than providing a callback, the application simply calls 'ReadFrame' to get the current frame from the camera. It passes in a 'level' to choose the downsampling level (0=full res, 1=half res, 2=quarter res) and obviously a place to put the data.
Internally those output queues will be added to by the internal callbacks on the resizer output ports. Crucially, the resizer buffers will be passed directly into the output queue. A buffer will then only be returned to the resizer when:
The application calls ReadFrame, thus the buffer is no longer needed
An output queue contains more than x (potentially 1) entry, indicating the application isn't reading data fast enough (or at all) so frames can be dropped
This'll all be a lot easier if I can use the mmal queue code, but if not I'll roll my own.
The only problem with this plan is that it involves fiddling around with a complex api and reworking lots of fiddly code, and it's past my bed time. Even coders need sleep, so I'll have to get to downsampling another day.
For the past few days I've been working on getting a pair of raspberry pi camera modules working and accessing their data in a c++ program ready for image processing. Last night I got to the first version of my camera api which can be found here. So far I can:
Start the camera up
Have a callback triggered once per frame that gets past the buffer + its size
Shut the camera down
Very soon I'll get to work on converting the YUV data the camera submits into nice friendly RGB data, and get downsampling of the images going. Both will either need to be done using mmal, or through my own GPU code if they stand a chance of being usable for real time processing. Once they're going I'll be in a great position to get more complex stuff like feature analysis working.
However, while I've got a host of ideas of how to move forwards, the first thing to do is get the output from the camera rendering on screen so I can actually see it in action. As such my next goal is to get opengl going, and render the output from the camera callback to the screen. Initially it'll look like garbage (as it'll actually be yuv data), but it'll be garbage that changes when things move in front of the camera! Once it's working I'll be in a position to do things like downsampling, rgb conversion etc and actually see the results to verify they're functioning correctly.
Getting OpenGL going
I've not endeavered to get opengl working on the pi yet, but there's a couple of examples called hello_triangle and hello_triangle2. On looking at them, hello_triangle2 is both the simplest and most interesting as it uses shaders to do its rendering. I start by copying the graphics init code from hello_triangle2, and then add begin/end frame calls that basically clear the screen and swap the buffers respectively. This rather uninteresting photo is the result:
OK so it's not much, but crucially it shows opengl is operating correctly - I have a render loop that is clearing the screen to blue (and all the while I'm still reading the camera every frame in the background).
Shaders, Buffers and Boxes
I'm not gonna mess with fixed function pipelines and then have to go back and change it to shaders as soon as I want something funky - this is the 21st century after all! As a result I need to get shaders working which I've never done in opengl. From the example it basically seems to be a case of:
Load/set source code for a shader
Compile it, and get a shader id
Create a 'program', and assign it a vertex shader and a fragment shader
So I come up with this code inside a little GfxShader class:
bool GfxShader::LoadVertexShader(const char* filename)
{
//cheeky bit of code to read the whole file into memory
assert(!Src);
FILE* f = fopen(filename, "rb");
assert(f);
fseek(f,0,SEEK_END);
int sz = ftell(f);
fseek(f,0,SEEK_SET);
Src = new GLchar[sz+1];
fread(Src,1,sz,f);
Src[sz] = 0; //null terminate it!
fclose(f);
//now create and compile the shader
GlShaderType = GL_VERTEX_SHADER;
Id = glCreateShader(GlShaderType);
glShaderSource(Id, 1, (const GLchar**)&Src, 0);
glCompileShader(Id);
check();
printf("Compiled vertex shader:\n%s\n",Src);
return true;
}
That just loads up a file, and fires it through the open gl code to create a shader program. Next, I knock together the simplest vertex shader and fragment shader I can think of:
And now it's time to try and render a triangle using those shaders!!! Please note - at time of writing I still don't know if this is going to work, or if those shaders are entirely wrong... Unless I've missed something, it appears the old way of specifying vertices 1 by 1 isn't present in OpenGLES2 (although it's very possible I've missed something), so I'm gonna need to create me a vertex buffer. I knock together these bits to create and draw it...
But glUseProgram is giving me errors so its thinking hat time....
And... an hour of fiddling later I've discovered open gl doesn't return an error if the shader compiling or linking into a program fails. Instead it returns happy success, unless you specifically ask it how things went! Having fixed some compile errors in my earlier shaders I run it and am presented with my first quad:
And after adding offset and scale uniforms and passing them into this draw function...
Hmm - not sure what the glFlush does yet. One for later though. The point is I can make a box anywhere I want:
OK, it's.....
Texture Time
My ultimate goal here is to get the camera texture on screen, which will involve filling an open gl texture with data each frame and then displaying it on a quad like the one above. Before getting that far I'm just gonna try filling a texture with random data each frame and seeing where that gets me...
...half an hour later... well having grasped opengles2 a bit better, that was actually fairly easy. We have a 32x32 random texture (and a code base that's getting messier by the second):
Woohooo!
From camera to texture
This is it folks. If I can get from camera output to something on screen at a decent frame rate then it paves the way for much wonders on this raspberry pi. I'll start with a hacky not-thread-safe approach which will also waste a bit of cpu time doing extra memcpys and generally be bad. But quick to write.
So we've got a callback in the app that is submitting new frames from a separate thread, and a call on the main frame to render a texture on screen. I just need to get the data from the thread into the texure and all will be well. I start by bodgily setting my earlier 'random texture' to be exactly the right size for a 1280x720 camera image, resulting in something a little more 'trippy':
Now to try regenerating that random data each frame - works although very slow. Not even worth uploading the video I made of it really!
However, I now have code that generates some new data, pumps it to open gl and then draws it on screen. All I need to do now is use my camera feed as the 'new data' instead of random numbers. I write a new function that takes a pointer to a buffer and copies it into the block of memory I was previously filling with random numbers. Remember my camera data is still in YUV so it'll not fill a full RGB texture (and will look weird), so I make it keep copying until it fills the buffer - this gives me a good measure of performance. A bit of jiggery pokery and...
Eureka!!!
At 1080p the memcpys involved (probably 2 - one from camera -> scratch space, another from scratch space -> open gl texture) are heavy enough and it hits about 10fps. But at 720p (still easily enough for some tasty image fun) it's in at around 25fps. With a little clever engineering I can remove 1 of those copies, so it'll hit a solid 30fps. Here's a video to show it in action:
Pretty tasty yes? Although please note - when I say 'copies it into the cpu' I mean 'into cpu accesible memory'. One doesn't make sense, the other does...
All code is here, although it's in a bit of a state right now so don't take anything as 'the right way to do it' - especially the graphics bits!
Next Steps
Now that I can see what I'm generating (and can do it at an appreciable frame rate) I'm going to look at using the mmal image resizer component to create downsampled versions of the feed (for different granularity in image processing) and in theory do the rgb conversion (if the documentation is telling the truth...).
