Sunday, 25 January 2015

RK3288 - Firefly development board

I received this board just over a month ago from the Firefly team and have been keen to assess it development capabilities given its hosts a quad core Cortex A17 (or technically a Cortex A12) processor. 

On board is a Rockchip RK3288 processor which on initial glance has a pretty decent specification:

  1. 4Kx2K H.264/H.265(10-bit) video decoder
  2. 1080p H.264 video encoder
  3. Up to 3840X2160 display resolution
  4. 4Kx2K@60fpsHDMI2.0
  5. eDP 4Kx2K@30fps
  6. According to Rockchip the GPU core is listed as a Mali-T764 GPU although it's reported as a Mali-T760 by the Mali drivers.
  7. Ethernet Controller Supporting 10/100/1000-Mbps 

Given the above I think it is import to clarify what 4Kx2K actually means and the clue is in point 3. Having spent many hours debugging the kernel driver display code it turns out the RK3288 has 2 display controllers know as VOP_BIG and VOP_LIT (I presume for little). VOP_BIG support a maximum resolution of 3840x2160 which equates to 4K UHD (Ultra high definition television) and for VOP_LIT its 2560x1600 (WQXGA). Each controller can be bound to a display interface eg HDMI, LVDS or eDP (sadly missing on the firefly).  If you define 4Kx2K as 4096x2160 also know as DCI 4K then the definitions can be misleading. The numbers also align with H264/VP8/MVC decoding which max at 2160p@24fps (3840x2160), although the HEVC(H265) decoder contradicts this by  supporting 4k@60FPS (4096x2304). What is also interesting is the image processing engine can up scale to 3x input, which would imply 720p can be up scaled to 4K UHD.

The Firefly seems to be based on a Android TV box reference design and it could be argued that its targeted as an Android centric development board. The noticeable peripherals are:

1. VGA support + HDMI
2. On board microphone
3. Wifi Connector (RF Connector SMA Female)
4. IR Sensor
5. eMMC 16Gb

Firefly supply a pre-built dual boot image (on eMMC) that offers Android 4.4
and Ubuntu 14.04.

Android 4.4

Firefly supply the Android SDK sources so that customised images can be built from source. What is nice is that the Android Studio integrates well with the Firefly which eases development of Android Apps especially for newcomers. Furthermore the App can be remote debugged while running on the Firefly. I  suggest that you sign your App with the platform key from the SDK to ease integration when remote debugging. One pitfall to bear in mind is that Android 4.4 implements selinux so you may find accessing I/O via sysfs (eg GPIO) from your Android App is severely restricted. 

Ubuntu 14.04

The Ubuntu image uses a patched version of the Android kernel and unfortunately has no support for GPU/VUP acceleration.

Linux Support

Historically many ARM SOC vendors have side stepped any request for providing meaningful Linux support and instead rely on the developer community to progress this as far as they can. Unfortunately the situation is normally exacerbated by the lack co-operation for GPU/VPU support with SOC vendor. What's clearly not recognised by ARM is that this fragmentation is clearly benefiting Intel with their latest lower power SOC's having far superior out of box Linux support.

As of today I would argue the RK3288 falls midway between no and full Linux support. The reason for this is Rockchips effort to develop a Chromebook device, if you examine the Chromium OS source tree will find numerous patches submitted from Rockchip

So the question becomes can we make use of that source tree? Well my initial aim was to get a minimal kernel booting and ideally test if GPU support was possible. The video below shows the progress made after numerous weeks of attempting to bring a workable kernel. In the video I launch Openbox under X and run es2gear,glmark-es2 and some WebGL samples in Chromium.

Although the video may seem impressive the only GPU acceleration available is through EGL/GLES hence the WebGL examples are accelerated. What is important to bear in mind is that the xf86-video-armsoc driver lacks 2D support for Rockchip therefore this still is fair amount of software rendering in X11 plus I implemented workarounds for broken code. Furthermore performance isn't particular spectacular, es2gear & glmark-es2 numbers are here. Unfortunately I haven't had the time to investigate the cause(s) however further progress may be hinder by the lack of newer Mali drivers from Rockchip/ARM.

For those of you expecting a future Rockchip Chromium OS image to work on an existing RK3288 device you may be disappointed unfortunately the hardware differences make this a slim possibility.

Friday, 7 November 2014

Intel BayTrail - J1900

On of the main challengers to the current generation of ARM SOCs are the Intel BayTrail range. For me the interesting part of the family are E38XX (atom) and J1X00 (celeron) processors which boast  7-10W TDP. In this article I will cover the some initial performance metrics against the J1900 with the Intel Linux software stack. My test device was a low profile MX1900J industrial mini-itx board produced by BCM Advanced Research.

What's nice about the MX1900J :

1. Low profile with a heat sink that is approximately 15mm high.

2. On board DC power jack (12v) hence no need for a separate DC to DC converter board.

3. 4 x USB 3.0

4. Inclusion of LVDS and GPIO support.

5. Dual Ethernet NIC's

What's different about this board is the inclusion of a Display Port connector instead of HDMI along with VGA output. The BIOS has UEFI 64 and legacy support.

From an application developers view point there's quite a few advantages with the x86 platform. Firstly there is the vast amount of existing software that can run of the 'out of the box' or with minimal changes. Another is the shorter ramp up time between set up/configuration of the BSP/kernel/rootfs to actual application development. Lastly I would also argue that Intel do seem to devote a fair amount of resources to open source development therefore the underlying BSP have the potential to keep up with the latest trends (eg Chromium-ozone, Wayland, Tzien).

The J1900 GPU core supports Intel HD 4000 graphics and there are two linux graphics drivers available for the J1900. The lesser known of the two is the EMGD driver (Embedded Media and Graphics Driver) which are closed sourced binaries that are accessible through user space libraries eg libdrm, mesa and libva. The EMGD documentation targets these drivers for Fedora 18 against  Kernel 3.8 and xorg 13.1. Having previously used the EMGD drivers they can be ported to other Linux distros however problems may arise when upgrading or moving to newer  distro versions, where ABI breakages prevent this happening or cause stability issues. Intel prefer EMGD as they claim better 3D performance due to the Unified 3D (UFO) Driver.

The alternative to EMGD is the open source (Intel Linux Graphics) driver which can have better support for later kernels and hence usable on a later Linux distro. The downside may be a slight drop in overall performance and possibly stability. I chose to deploy the open source drivers against a very lightweight Ubuntu 14.04 image. The drivers provides OpenGL 3.3 and OpenGL ES 3.0.

The J1900 GPU core has 4 EU (Execution Units) combined with a maximum GPU frequency of 854Mhz. To given you have ideal of where the J1900 fits in the 'food chain' lets compare the FPS (frame per second) rates of running the WebGL Aquarium Demo on a imx6q, J1900 and an older 1037u celeron.

Number of Fish
1 50 100 500 1000
Platform Screen Resolution Frames Per Second
i.mx6q 1280x720 8 7 7 5 5
J1900 1680x1050 48 48 47 40 33
1037u 1680x1050 60 60 60 60 60

First lets be clear the above results are to be interpreted as a relative comparison. It should not be used as a primary marker for judging one platform to be superior to the other. Each platform has its merits based on a number of factors.  The i.mx6q as expected struggles (even at the lower resolution the rendering was not smooth) mainly due to the lower spec CPU/GPU core , an inefficient X driver and possibly some inefficient code paths in Chromium.  The older 1037u dual core celeron performs well due to the higher GPU frequency 1HGz and 6 EU, the trade off is a higher thermal output at 17W. What I couldn't easily account for was the drop off in the FPS rate at 1000 for the J1900. Below are the results from the BabylonJS train demo which is an intensive WebGL application supporting multiple camera angles, CAM_TRAIN being my favourite. What's interesting is that the FPS rate did not deviate when forcing Chromium to use EGL/GLES instead of OpenGL for the J1900.

Platform Screen Resolution Frames per second
J1900 1680x1050 13
1037u 1680x1050 24

Video playback is available through VAAPI plus libav with h.264/mpeg-2  hardware accelerated encoding/decoding. mplayer and gstreamer 1.0 support is readily available. CPU usage for decoding Big Buck Bunny at 1080p (H.264) was around 13% both in mplayer and using a simple gstreamer 1.0 pipeline. Decoding a 720p usb webcam at 30fps (YUY2) with the output encoded (H.264) to file and displayed to screen using a simple gstreamer pipeline resulted in 15% CPU usage.

Given the recent interest in HTML5 development for embedded platforms I  deployed a development build of Chromium (build 39). Chromium is fast becoming the web container of choice given the recent adoption of its engine in QT (QtWebEngine). HTML5test reported a healthy score of 512 out of 555 against Chromium.  A test HTML5 page with two embedded video files (playing concurrently) along a with a bunch of images (png) and static text ran without hiccup consuming 20% cpu. I briefly ran some demo HTML5 widgets from  zebra , webix and Kendo UI  again these ran smoothly. What should be possible with this platform is the ability to create a HTML5 GUI interface that could drive the rest of the application hosted on the same machine.

On the whole the results look very encouraging and the J1900 seems to offer a good trade off for a fan less solution with decent performance. Furthermore it should provide a relatively smooth route for application development. The main consideration is form factor and it is possible to find the J1900 in 3.5" SBC or Q7 form.

Saturday, 27 September 2014

i.MX6 Efficient font rendering and smoothing scrolling

Here is a short video demonstrating efficient font rendering and smoothing scrolling using OpenGL ES.

You may be forced to adopt this approach if your finding alternatives routes such as Qt5, HTML 5 or GTK+/Cario aren't giving you satisfactory results. The downside is that since you are starting from a low base (this a very low level implementation) it requires a far amount of development effort. On the positive side the code can be made to be as efficient as possible (to reduce power consumption/heat) and can be highly customised, in the above demo we control the screen from a remote PC.

Sunday, 27 July 2014

I.MX6 Developing with WebGL

On the new features in BSP 3.10.17 was support for WebGL. In a nutshell WebGL is javascript API supporting 3D and 2D graphics rendering, more about WebGL can be found here. WebGL is based on Open GL ES 2.0.

WebGL support opens up the possibility of developing and running graphical web based application on the i.mx6 within a browser. Furthermore there's also the possibility of the deploying a LAMP stack to serve the application from the i.mx6.  This opens up the possibility of developing simple games, kiosk applications, interaction user manuals/instructions and signage displays, etc..

To give you a taste of what is possible with the current WebGL implementation, I've put together a number of short videos that run existing WebGL demos/applications. These were run on lightweight Debian rootfs with Chromium browser (under X) on an i.mx6q board as part of a prototyping exercise. Chromium has been tweaked to maximize performance and the screen resolution was 720p (1280x720. Beware that not all WebGL application will run, some will fail because the Vivante libraries currently lack the 'Float Textures' extensions, others because the GPU is not powerful enough to give a decent FPS rate. Apologies for the quality of the videos. I'd advise that a i.mx6 quad processor is used to run WebGL as the examples consume 25-35% CPU under load.

The first video which also sums up what can be done on the i.mx6 with WebGL is the 'Lego build Chrome' demo. This application allows the creation of a Lego structure.

three.js is a javascript library which makes WebGL easier, here are some WebGL/CSS 3D examples which play nicely.

babylon.js is a 3D engine based on WebGL and javascript. The 'Create Your Own Shader' demo allows online editing of shaders.

CopperCube is an editor for creating 3D application, this is the 'Backyard Demo'.

Finally the "undulating-monkey" from aerotwist.

Saturday, 7 June 2014

I.MX6 Debian Jessie (GPU/VPU) 3.10.17_GA

Given that we a finally have a GA release of the 3.10.17 kernel and BSP from Freescale (although there also is a 3.10.31 alpha release in the pipleline). Here is an upgraded the Debian Jessie rootfs. You can download it from here and my previous 2 posts (here and here) cover installation and configurations step. Furthermore you require a kernel based on this release with the appropriate dts file for your i.mx6 device. This is a barebone rootfs with gpu/vpu support  and as previously stated I use this rootfs mainly for development. I haven't observed any major changes since the 3.10.17 beta release.

Friday, 11 April 2014

I.MX6 - gstreamer-imx and usb webcam support

Following on from my previous post about gstreamer-imx, this blog covers hooking a usb webcam using the gstreamer-imx plugins. It starts with creating simple pipeline for screen output, next is a pipeline for time lapse video recording. We then progress to streaming to VLC and finally implementing a simple rtsp server. Below is video demonstrating a simple RTSP server running on a AR6MXQ streaming to VLC. The webcam is pointing at the screen and streams the output (720p @10fps). We launch a VLC client window which is located in the lower right side of the same screen (unfortunately this causes an echo effect). As you will see any activity on the screen is streamed to the VLC client window (lower right) albeit with a delay.

The webcam in question was a microsoft HD-3000 which in theory is capable of 720p at 30fps. As with most webcams (which don't encode H264) it outputs YUYV (YUY2) and more importantly MJPEG. On linux you can easily determine what your webcam capabilities are by launching  v4l2-ctl, eg:

v4l2-ctl --list-formats-ext

This post provides a guide to what is possible with a gstreamer-imx plugins and the examples provided should not be treated as production ready. I'm assuming the examples will work on other usb webcams and or potentially other v4l2 devices. However the examples may require alterations depending on your webcam and it's capabilities, therefore background reading on gstreamer is recommended.

Testing was conducted on debian jessie and the target device was a AR6MXQ board provided by BCM Advanced Research. The examples should run on most imx6 devices as there is no device specific dependencies.

While testing with the HD-3000, the webcam  occasionally stop sending a stream if I configured the wrong resolution. The workaround was to unplug the device or reset the usb port  (replace '1-1.3' with the correct usb host and port id for your webcam) eg :

echo 1-1.3 > /sys/bus/usb/drivers/usb/unbind
echo 1-1.3 > /sys/bus/usb/drivers/usb/bind

Output to screen

Now back to the webcam, first step was getting output displayed to the screen. This requires reading the v4l2 source and converting the input so that is compatible with imxeglvivsink. Initial testing revealed decoding MPEG from the webcam performed significantly better than decoding YUY2. Another issue encountered (thanks to Carlos) was imxvpudec outputting Y42B something imeglvivsink currently can't cope with hence the inclusion of imxipuvideotransform element. So here is the final pipeline (webcam input is MPEG 720p at 30fps) :

gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! im
xvpudec ! imxipuvideotransform ! imxeglvivsink sync=false

CPU usage for the above was around 10%, while for YUY2 (720p at 10fps) it rises to 25% with this pipeline:

gst-launch-1.0 v4l2src ! 'video/x-raw,width=1280,height=720,framerate=10/1' ! imxipuvideotransform ! imxeglvivsink sync=false

Simple time lapsed video recording

Now lets implement a simple timelapsed video recorder that outputs to file and screen. I limited to the input MPEG stream to 10fps, reduced the bitrate  and encoded as h264 to reduce the output file size. Additional CPU load occurs due to the inclusion of the clockoverlay element, without the element it is difficult to know when the recording was taken.  Without this clockoverlay element CPU load is < 10%.

gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' ! imx
vpudec ! imxipuvideotransform ! clockoverlay time-format="%Y/%m/%d %H:%M:%S" ! tee name=splitter ! queue ! imxvpuenc_h264 bitrate=1024 ! filesink location=test .mp4 splitter. ! queue ! imxeglvivsink sync=false

The above pipeline generates approximately 400Mb per hour so probably not practical for production use.

Streaming to VLC

To enable streaming to VLC we need to create a 'sdp' file, this instruction VLC to act as a rtp server, below are the contents of the file:

m=video 5000 RTP/AVP 96
c=IN IP4
a=rtpmap:96 H264/90000

Save the contents to file eg 'imx6.sdp' and launch VLC,  because there is no h/w acceleration for VLC on the imx6, VLC was running on a PC :

vlc --no-audio imx6.sdp

On the imx6, we are a rtp client submitting h264 payloads to the VLC server (note the inclusion of the host/ip address of the PC) :

gst-launch-1.0 -v v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !
imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 bitrate=1024 ! rtph264pay ! udpsink host=<host/ip of PC> port=5000

Simple RTSP server

And lastly, lets try running an rtp server. Fortunately there is an additional gstreamer plugin (gst-rtsp-server) with rtp support that includes an example test server. The downside is that it needs to be built from sources. You will need to checkout the 1.1.90 tag and build (similar to gstreamer-imx). Once built we can launch the example test server and pass it a pipeline similar to that used when outputting to screen (you need to export the library path so that is found) eg:

export ./gst-rtsp-server/gst/rtsp-server/.libs
cd ./gst-rtsp-server/examples/.libs/

./test-launch '(v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay name=pay0 pt=96 )' 

The test server listens on port 8554 therefore the rtsp URL is

rtsp://<host/ip of imx6>:8554/test

You can test by launching VLC and opening a 'Network Stream' to the URL. While streaming the CPU load on the imx6 hovered around 60%. Given this is example code it should be possible to optimise the pipeline/code to bring down this figure.

Thursday, 10 April 2014

I.MX6 - gstreamer-imx

gstreamer-imx is set of gstreamer 1.0 imx6 video plugins principally developed by Carlos Giani a.k.a dv_ (great work!). Once stable these plugins "hopefully" will supersede the Freescale BSP gstreamers 0.10 plugins which are showing their age given that gstreamer 0.10 is longer maintained.

The main driver for this work was to get a usb webcam streaming via these plugins this will be covered in the next post. In meantime below are build instructions for deploying the latest code to the debian rootfs if you fancy experimenting.  I'm finding the debian jessie build useful to jump start prototype development on the imx6 mainly due to the availability of prebuilt packages.

Here is a short video demonstrating the use of color key along with the imxipusink to hide/view video within XFCE. The video only appears for the color key 'black' hence it is visible in the terminal windows and menu. The test device was a AR6MXQ board provide by BCM Advanced Research. As a designer/provider of industrial motherboards BCM have done a good job with their first feature rich ARM development board.

The current debian rootfs already includes an older build of the gstreamer-imx plugins. Upgrading these to a newer release is fairly trivial, you can build the latest sources natively on debian as follows:

1. Remove the existing plugins

 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ 
 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ 
 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ 

 rm /usr/lib/ 
 rm /usr/lib/ 
 rm /usr/lib/ 

2. Build lastest sources

 git clone git://  
 cd gstreamer-imx  
 ./waf configure --prefix=usr --kernel-headers=/usr/include  
 ./waf install  

3. Deploy plugins

 cp usr/lib/gstreamer-1.0/ /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/ /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/ /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/ /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/ /usr/lib  
 cd /usr/lib  
 ln -s  
 ln -s  

4. Reboot

5. Verify the plugins are present

 root@debian-imx6:~# gst-inspect-1.0 | grep imx  
 imxipu: imxipuvideotransform: Freescale IPU video transform element  
 imxipu: imxipusink: Freescale IPU video sink  
 imxvpu: imxvpudec: Freescale VPU video decoder  
 imxvpu: imxvpuenc_h263: Freescale VPU h.263 video encoder  
 imxvpu: imxvpuenc_h264: Freescale VPU h.264 video encoder  
 imxvpu: imxvpuenc_mpeg4: Freescale VPU MPEG-4 video encoder  
 imxvpu: imxvpuenc_mjpeg: Freescale VPU motion JPEG video encoder  
 imxeglvivsink: imxeglvivsink: Freescale EGL video sink  
 imxv4l2src: imxv4l2src: V4L2 CSI Video Source  

6. Test video playback (under X11)

 gst-launch-1.0 playbin uri=file://<video file> video-sink=imxeglvivsink  

Unfortunately the elements/sinks aren't documented so you may need to refer to the source code to determine features or properties. There is also a new plugin for CSI camera sources (imxv4l2src) provided by  Philip Craig.