Pi Eyes Stage 4

Last time round I got to the point at which I was pulling data out of the camera and using opengl to efficiently render it to screen. There's a few performance improvements to be made, but it runs at a respectable frame rate and could hit 30hz at 720p no problem.

Next up it's time to get the camera feed from the native YUV format into RGB ready for image processing. I'll also be needing to get some down sampled versions of the image, as the more expensive image processing algorithms aren't fast enough to run on a hi def video feed. This'll be a fairly breif post though, as it's late and my brain is going to sleep...

The Image Resizer

My first port of call was the image resize component built into mmal (thanks to a handy tip on this post), which uses the hardware resizer to take an image and... well... resize it! However, as a handy side effect it can also convert between YUV and RGB in the process. At this point massive thanks goes out to Mr Jason Van Cleave, who made all the mmal component documentation available on his web site.

So in short, I need to adjust the camera code so it:

• Create an image resize component (I eventually worked out its the "vc.ril.resizer" component)
• Connect it to the camera's video output port (the one we're currently displaying)
• Set it's input format to that of the video output, and the output format to RGBA. We leave the image sizes the same for now though, so its not really doing any resizing - just the conversion

I do a little code cleanup first so it's easier to add to, plonk in the new code and after a few iterations...

We're in business! Code here:

http://www.cheerfulprogrammer.com/downloads/pi_eyes_stage4/picam_rgb.zip

Unfortunately on the first attempt performance appeared to be very poor. Interestingly though if I remove the actual rendering of the image it runs fine. This leads me to believe that the image resizer is chomping through most of the gpu time and consequentially I can't render fast enough. This is really annoying as I frankly don't see why it should be so slow - maybe it's just some interplay between opengl and mmal. 

I'll know more about the resizer performance once I get multiple resizers running, generating different downsampled images and we'll see what the actual costs are. If necessary I'm fairly confident I could write a shader that did the convert and downsample quite efficiently. I'm now getting 15hz, which I'm not happy about but it'll do for the moment.

A quick restructure

My next goal is to get multiple outputs at different resolutions coming out of the camera. This allows me to analyse the data at different levels in order to pick and choose where I spend my cpu. It should be doable using the 'video splitter' component, but it raises a few problems in terms of my architecture.

Right now the camera code simply runs, then calls a callback for each frame. Once the splitter is running I'll be receiving blocks of data constantly from different sources and will need a nice way of managing this and providing it in an api to the user. As a result, before going onto the multiple output world, I'm going to adjust the camera code so it internally buffers up the frames and allows the user to read them in a syncronous manner. If I can make use of the queuing system in mmal then I should be able to set it up as follows:

The basic idea is that the camera wraps up all the mmal stuff as usual, but rather than providing a callback, the application simply calls 'ReadFrame' to get the current frame from the camera. It passes in a 'level' to choose the downsampling level (0=full res, 1=half res, 2=quarter res) and obviously a place to put the data. 

Internally those output queues will be added to by the internal callbacks on the resizer output ports. Crucially, the resizer buffers will be passed directly into the output queue. A buffer will then only be returned to the resizer when:

• The application calls ReadFrame, thus the buffer is no longer needed
• An output queue contains more than x (potentially 1) entry, indicating the application isn't reading data fast enough (or at all) so frames can be dropped

This'll all be a lot easier if I can use the mmal queue code, but if not I'll roll my own. 

The only problem with this plan is that it involves fiddling around with a complex api and reworking lots of fiddly code, and it's past my bed time. Even coders need sleep, so I'll have to get to downsampling another day.