PI Vision step 2 - to the metal!

In my last post I got a playstation eye camera working via ffmpeg, but even here I couldn't acheive a respectable frame rate at 640x480. Now perhaps I'm wrong, but it seems to me that the raspberry pi contains enough oomph (perhaps with the gpu involved) to happily decode a std definition web cam feed. The question for me is where is the slow down, why is it happening, and can I fix it?

My first port of call after digging around with ffmpeg was to go a step further back and take a look at using video4linux (aka the linux video capture system) directly. However after compiling a few demos I was unconvinced. I managed to get a lo res feed decoding, and the camera wouldn't even initialize at 640x480. Had I been certain it'd work eventually at a decent speed I might have been willing to put more time in, but the slow down could easily be in v4l.

So what now? Well, I've decided my next port of call is to try to talk to the camera directly. Unless the slow down is the usb communication (which is possible), this should get me close enough to the metal to get a high performance feed. It has the added advantage of letting me monkey around with the usb device directly as well, thus avoiding any nuances with picking up the video0 device (mentioned in earlier post).

Lib USB

A bit of research later, I discover libusb. This handy library lets you talk to the usb port directly, without having to monkey around with writing kernel code. Crossing my fingers, I take a guess at the package name and...

pi@raspberrypi ~ $ sudo apt-get install libusb-1.0

Bingo! I'm informed the library exists, but do I want to get the dev version (maybe this includes the header files). I say yes, it downloads, and there it is in /usr/include/libusb-1.0. Time for some c++. Incidentally, my current approach to coding on the pi is to setup a samba server so I can access the files from my windows PC, then use visual studio to edit. To compile, I currently log in via SSH and do so on the command prompt:

While this isn't exactly wonderful (debugging is reduced to printf), it is simple and efficient, and is serving me fairly well so far! With my main.cpp created, I can build and run it with:

pi@raspberrypi ~/libusbtest $ gcc main.cpp -o main

pi@raspberrypi ~/libusbtest $ ./main

So I'm all set. Lets see if I can use lib usb to find and chat to the ps eye...

After a bit of reading and fiddling, I end up with this simple program. It uses libusb to enumerate all the devices available, get some information on them, and print it out.

#include <stdio.h>
#include <iostream>
#include <libusb-1.0/libusb.h>

libusb_context* UsbContext;

using namespace std;

int main(int argc, const char* argv[])
{
    printf("Initializing lib usb\n");

    int res = libusb_init(&UsbContext);
    printf("libusb_init: %d\n", res);

    libusb_set_debug(UsbContext,3);

    libusb_device** devicelist;
    ssize_t num_devices = libusb_get_device_list(UsbContext, &devicelist);
    for(int i = 0; i < num_devices; i++)
    {
        int bus_number = libusb_get_bus_number(devicelist[i]);
        int device_address = libusb_get_device_address(devicelist[i]);
        int device_speed = libusb_get_device_speed(devicelist[i]);
        
        cout << "Device " << i << ": bus_number=" << bus_number << ", device_address=" << device_address << ", device_speed=" << device_speed << "\n";

        libusb_device_descriptor descriptor;
        libusb_get_device_descriptor(devicelist[i],&descriptor);

        cout << "  bLength=" << (int)descriptor.bLength;
        cout << ", bDescriptorType=" << (int)descriptor.bDescriptorType;
        cout << ", bcdUSB=" << (int)descriptor.bcdUSB;
        cout << ", bDeviceClass=" << (int)descriptor.bDeviceClass;
        cout << "\n  bDeviceSubClass=" << (int)descriptor.bDeviceSubClass;
        cout << ", bDeviceProtocol=" << (int)descriptor.bDeviceProtocol;
        cout << ", bMaxPacketSize0=" << (int)descriptor.bMaxPacketSize0;
        cout << ", idVendor=" << (int)descriptor.idVendor;
        cout << "\n  idProduct=" << (int)descriptor.idProduct;
        cout << ", bcdDevice=" << (int)descriptor.bcdDevice;
        cout << ", iManufacturer=" << (int)descriptor.iManufacturer;
        cout << ", iProduct=" << (int)descriptor.iProduct;
        cout << "\n  iSerialNumber=" << (int)descriptor.iSerialNumber;
        cout << ", bNumConfigurations=" << (int)descriptor.bNumConfigurations;
        cout << "\n";

        libusb_device_handle* devhandle;
        int openres = libusb_open(devicelist[i],&devhandle);
        if(openres == 0)
        {
            cout << "  Device opened\n";

            unsigned char buffer[1024];
            libusb_get_string_descriptor_ascii(devhandle,descriptor.iManufacturer,buffer,1024);
            cout << "  Manufacturer=" << (char*)buffer << "\n";
            libusb_get_string_descriptor_ascii(devhandle,descriptor.iProduct,buffer,1024);
            cout << "  Product=" << (char*)buffer << "\n";
            libusb_get_string_descriptor_ascii(devhandle,descriptor.iSerialNumber,buffer,1024);
            cout << "  SerialNumber=" << (char*)buffer << "\n";

            libusb_close(devhandle);
        }
        else
        {
            cout << "  Could not open device\n";
        }

        cout << "\n";
    }

    libusb_free_device_list(devicelist, 1);
    libusb_exit(UsbContext);
    return 0;
}

I discovered I had to run my application as a super user (i.e. use 'sudo') to access the usb ports, but other than that, success:


pi@raspberrypi ~/libusbtest $ gcc -o main main.cpp -lusb-1.0 -lstdc++
pi@raspberrypi ~/libusbtest $ sudo ./main
Initializing lib usb
libusb_init: 0
Device 0: bus_number=1, device_address=1, device_speed=3
  bLength=18, bDescriptorType=1, bcdUSB=512, bDeviceClass=9
  bDeviceSubClass=0, bDeviceProtocol=1, bMaxPacketSize0=64, idVendor=7531
  idProduct=2, bcdDevice=770, iManufacturer=3, iProduct=2
  iSerialNumber=1, bNumConfigurations=1
  Device opened
  Manufacturer=Linux 3.2.27+ dwc_otg_hcd
  Product=DWC OTG Controller
  SerialNumber=bcm2708_usb

<... bla bla bla more devices ...>


Device 4: bus_number=1, device_address=5, device_speed=3
  bLength=18, bDescriptorType=1, bcdUSB=512, bDeviceClass=9
  bDeviceSubClass=0, bDeviceProtocol=1, bMaxPacketSize0=64, idVendor=5141
  idProduct=16384, bcdDevice=1794, iManufacturer=10, iProduct=11
  iSerialNumber=0, bNumConfigurations=1
  Device opened
  Manufacturer=SONY
  Product=USB 2.0 HUB
  SerialNumber=USB 2.0 HUB

<... bla bla bla more devices ...>

Device 7: bus_number=1, device_address=8, device_speed=3
  bLength=18, bDescriptorType=1, bcdUSB=512, bDeviceClass=0
  bDeviceSubClass=0, bDeviceProtocol=0, bMaxPacketSize0=64, idVendor=5141
  idProduct=8192, bcdDevice=256, iManufacturer=1, iProduct=2
  iSerialNumber=0, bNumConfigurations=1
  Device opened
  Manufacturer=OmniVision Technologies, Inc.
  Product=USB Camera-B3.03.09.3
  SerialNumber=USB Camera-B3.03.09.3

2 interesting results there. The obvious end one - the OmniVision camera interface. I am however also intrigued by what claims to be a Sony USB HUB. Oh well - progress has been made. Next stop I'll focus on that 7th device and try to pull out some more information about it.

Digging Through Drivers

OK, I've spent the past few days digging through code and finding various bits and pieces. After some searching I went through the following process:

• First step, this web site: http://nuigroup.com/forums/viewthread/2921/ in which the first windows drivers for the PS Eye were posted. Crucially, the post states that the play station eye uses the OV534-LB50 chip as its usb interface
• After some scouring, I found the OV534 linux driver source here
• While that source is rather long and contains lots of checks, it references this link - the original reverse engineering of the OV534 usb protocol

Conclusion

I'm now left with a few possible conclusions...

1. The poor performance is due to the cost of getting data from the usb port and into memory. If this is the case, I'm ultimately screwed as I have no idea how to fix it (if it is even fixable!)
2. It's down to slow OV534 drivers. I can potentially fix this, if I can proceed with my earlier plan of directly communicating with the chip via usb.
3. The drivers are fine, but something in video 4 linux is slowing things down. Again, my earlier plan will solve this one - if I can make it work
4. Aside from some oddities in functionality, video 4 linux is fine and it's the fact that everything converts from YUV to RGB on cpu that slows everything down. If that's all it is, just taking the raw video 4 linux data and converting it on gpu should do the job.

My gut right now says to spend a bit of time trying to bypass video 4 linux and go straight to usb as planned. If this turns out to be too tricky I'll give video 4 linux some more love and see if I can get it reading raw YUV data at 640x480 at a decent speed. 

I end this post with a final thought... YUV is 2 bytes per pixel. So for a 640x480 feed at 30fps you have:
     640*480*2*30 bytes per second = 18432000 bytes per second
Or roughly 17MB/s. Should a 600Mhz chip be able to get 17MB/s out of a USB port? I really don't know!