I.MX6 - gstreamer-imx and usb webcam support

Following on from my previous post about gstreamer-imx, this blog covers hooking a usb webcam using the gstreamer-imx plugins. It starts with creating simple pipeline for screen output, next is a pipeline for time lapse video recording. We then progress to streaming to VLC and finally implementing a simple rtsp server. Below is video demonstrating a simple RTSP server running on a AR6MXQ streaming to VLC. The webcam is pointing at the screen and streams the output (720p @10fps). We launch a VLC client window which is located in the lower right side of the same screen (unfortunately this causes an echo effect). As you will see any activity on the screen is streamed to the VLC client window (lower right) albeit with a delay.

The webcam in question was a microsoft HD-3000 which in theory is capable of 720p at 30fps. As with most webcams (which don't encode H264) it outputs YUYV (YUY2) and more importantly MJPEG. On linux you can easily determine what your webcam capabilities are by launching  v4l2-ctl, eg:

v4l2-ctl --list-formats-ext

This post provides a guide to what is possible with a gstreamer-imx plugins and the examples provided should not be treated as production ready. I'm assuming the examples will work on other usb webcams and or potentially other v4l2 devices. However the examples may require alterations depending on your webcam and it's capabilities, therefore background reading on gstreamer is recommended.

Testing was conducted on debian jessie and the target device was a AR6MXQ board provided by BCM Advanced Research. The examples should run on most imx6 devices as there is no device specific dependencies.

While testing with the HD-3000, the webcam  occasionally stop sending a stream if I configured the wrong resolution. The workaround was to unplug the device or reset the usb port  (replace '1-1.3' with the correct usb host and port id for your webcam) eg :

echo 1-1.3 > /sys/bus/usb/drivers/usb/unbind
echo 1-1.3 > /sys/bus/usb/drivers/usb/bind

Output to screen

Now back to the webcam, first step was getting output displayed to the screen. This requires reading the v4l2 source and converting the input so that is compatible with imxeglvivsink. Initial testing revealed decoding MPEG from the webcam performed significantly better than decoding YUY2. Another issue encountered (thanks to Carlos) was imxvpudec outputting Y42B something imeglvivsink currently can't cope with hence the inclusion of imxipuvideotransform element. So here is the final pipeline (webcam input is MPEG 720p at 30fps) :


gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! im
xvpudec ! imxipuvideotransform ! imxeglvivsink sync=false

CPU usage for the above was around 10%, while for YUY2 (720p at 10fps) it rises to 25% with this pipeline:

gst-launch-1.0 v4l2src ! 'video/x-raw,width=1280,height=720,framerate=10/1' ! imxipuvideotransform ! imxeglvivsink sync=false

Simple time lapsed video recording

Now lets implement a simple timelapsed video recorder that outputs to file and screen. I limited to the input MPEG stream to 10fps, reduced the bitrate  and encoded as h264 to reduce the output file size. Additional CPU load occurs due to the inclusion of the clockoverlay element, without the element it is difficult to know when the recording was taken.  Without this clockoverlay element CPU load is < 10%.

gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' ! imx
vpudec ! imxipuvideotransform ! clockoverlay time-format="%Y/%m/%d %H:%M:%S" ! tee name=splitter ! queue ! imxvpuenc_h264 bitrate=1024 ! filesink location=test .mp4 splitter. ! queue ! imxeglvivsink sync=false

The above pipeline generates approximately 400Mb per hour so probably not practical for production use.

Streaming to VLC

To enable streaming to VLC we need to create a 'sdp' file, this instruction VLC to act as a rtp server, below are the contents of the file:

v=0
s=GStreamer
m=video 5000 RTP/AVP 96
c=IN IP4 127.0.0.1
a=type:broadcast
a=rtpmap:96 H264/90000
a=fmtp:96

Save the contents to file eg 'imx6.sdp' and launch VLC,  because there is no h/w acceleration for VLC on the imx6, VLC was running on a PC :

vlc --no-audio imx6.sdp

On the imx6, we are a rtp client submitting h264 payloads to the VLC server (note the inclusion of the host/ip address of the PC) :

gst-launch-1.0 -v v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !
imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 bitrate=1024 ! rtph264pay ! udpsink host=<host/ip of PC> port=5000

Simple RTSP server

And lastly, lets try running an rtp server. Fortunately there is an additional gstreamer plugin (gst-rtsp-server) with rtp support that includes an example test server. The downside is that it needs to be built from sources. You will need to checkout the 1.1.90 tag and build (similar to gstreamer-imx). Once built we can launch the example test server and pass it a pipeline similar to that used when outputting to screen (you need to export the library path so that libgstrtspserver-1.0.so is found) eg:

export ./gst-rtsp-server/gst/rtsp-server/.libs
cd ./gst-rtsp-server/examples/.libs/

./test-launch '(v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay name=pay0 pt=96 )' 

The test server listens on port 8554 therefore the rtsp URL is

rtsp://<host/ip of imx6>:8554/test

You can test by launching VLC and opening a 'Network Stream' to the URL. While streaming the CPU load on the imx6 hovered around 60%. Given this is example code it should be possible to optimise the pipeline/code to bring down this figure.