Saturday, 27 September 2014

i.MX6 Efficient font rendering and smoothing scrolling

Here is a short video demonstrating efficient font rendering and smoothing scrolling using OpenGL ES.



You may be forced to adopt this approach if your finding alternatives routes such as Qt5, HTML 5 or GTK+/Cario aren't giving you satisfactory results. The downside is that since you are starting from a low base (this a very low level implementation) it requires a far amount of development effort. On the positive side the code can be made to be as efficient as possible (to reduce power consumption/heat) and can be highly customised, in the above demo we control the screen from a remote PC.

Sunday, 27 July 2014

I.MX6 Developing with WebGL

On the new features in BSP 3.10.17 was support for WebGL. In a nutshell WebGL is javascript API supporting 3D and 2D graphics rendering, more about WebGL can be found here. WebGL is based on Open GL ES 2.0.

WebGL support opens up the possibility of developing and running graphical web based application on the i.mx6 within a browser. Furthermore there's also the possibility of the deploying a LAMP stack to serve the application from the i.mx6.  This opens up the possibility of developing simple games, kiosk applications, interaction user manuals/instructions and signage displays, etc..

To give you a taste of what is possible with the current WebGL implementation, I've put together a number of short videos that run existing WebGL demos/applications. These were run on lightweight Debian rootfs with Chromium browser (under X) on an i.mx6q board as part of a prototyping exercise. Chromium has been tweaked to maximize performance and the screen resolution was 720p (1280x720. Beware that not all WebGL application will run, some will fail because the Vivante libraries currently lack the 'Float Textures' extensions, others because the GPU is not powerful enough to give a decent FPS rate. Apologies for the quality of the videos. I'd advise that a i.mx6 quad processor is used to run WebGL as the examples consume 25-35% CPU under load.

The first video which also sums up what can be done on the i.mx6 with WebGL is the 'Lego build Chrome' demo. This application allows the creation of a Lego structure.





three.js is a javascript library which makes WebGL easier, here are some WebGL/CSS 3D examples which play nicely.




babylon.js is a 3D engine based on WebGL and javascript. The 'Create Your Own Shader' demo allows online editing of shaders.



CopperCube is an editor for creating 3D application, this is the 'Backyard Demo'.



Finally the "undulating-monkey" from aerotwist.



Saturday, 7 June 2014

I.MX6 Debian Jessie (GPU/VPU) 3.10.17_GA

Given that we a finally have a GA release of the 3.10.17 kernel and BSP from Freescale (although there also is a 3.10.31 alpha release in the pipleline). Here is an upgraded the Debian Jessie rootfs. You can download it from here and my previous 2 posts (here and here) cover installation and configurations step. Furthermore you require a kernel based on this release with the appropriate dts file for your i.mx6 device. This is a barebone rootfs with gpu/vpu support  and as previously stated I use this rootfs mainly for development. I haven't observed any major changes since the 3.10.17 beta release.

Friday, 11 April 2014

I.MX6 - gstreamer-imx and usb webcam support

Following on from my previous post about gstreamer-imx, this blog covers hooking a usb webcam using the gstreamer-imx plugins. It starts with creating simple pipeline for screen output, next is a pipeline for time lapse video recording. We then progress to streaming to VLC and finally implementing a simple rtsp server. Below is video demonstrating a simple RTSP server running on a AR6MXQ streaming to VLC. The webcam is pointing at the screen and streams the output (720p @10fps). We launch a VLC client window which is located in the lower right side of the same screen (unfortunately this causes an echo effect). As you will see any activity on the screen is streamed to the VLC client window (lower right) albeit with a delay.





The webcam in question was a microsoft HD-3000 which in theory is capable of 720p at 30fps. As with most webcams (which don't encode H264) it outputs YUYV (YUY2) and more importantly MJPEG. On linux you can easily determine what your webcam capabilities are by launching  v4l2-ctl, eg:

v4l2-ctl --list-formats-ext

This post provides a guide to what is possible with a gstreamer-imx plugins and the examples provided should not be treated as production ready. I'm assuming the examples will work on other usb webcams and or potentially other v4l2 devices. However the examples may require alterations depending on your webcam and it's capabilities, therefore background reading on gstreamer is recommended.

Testing was conducted on debian jessie and the target device was a AR6MXQ board provided by BCM Advanced Research. The examples should run on most imx6 devices as there is no device specific dependencies.

While testing with the HD-3000, the webcam  occasionally stop sending a stream if I configured the wrong resolution. The workaround was to unplug the device or reset the usb port  (replace '1-1.3' with the correct usb host and port id for your webcam) eg :

echo 1-1.3 > /sys/bus/usb/drivers/usb/unbind
echo 1-1.3 > /sys/bus/usb/drivers/usb/bind



Output to screen


Now back to the webcam, first step was getting output displayed to the screen. This requires reading the v4l2 source and converting the input so that is compatible with imxeglvivsink. Initial testing revealed decoding MPEG from the webcam performed significantly better than decoding YUY2. Another issue encountered (thanks to Carlos) was imxvpudec outputting Y42B something imeglvivsink currently can't cope with hence the inclusion of imxipuvideotransform element. So here is the final pipeline (webcam input is MPEG 720p at 30fps) :


gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! im
xvpudec ! imxipuvideotransform ! imxeglvivsink sync=false


CPU usage for the above was around 10%, while for YUY2 (720p at 10fps) it rises to 25% with this pipeline:

gst-launch-1.0 v4l2src ! 'video/x-raw,width=1280,height=720,framerate=10/1' ! imxipuvideotransform ! imxeglvivsink sync=false


Simple time lapsed video recording


Now lets implement a simple timelapsed video recorder that outputs to file and screen. I limited to the input MPEG stream to 10fps, reduced the bitrate  and encoded as h264 to reduce the output file size. Additional CPU load occurs due to the inclusion of the clockoverlay element, without the element it is difficult to know when the recording was taken.  Without this clockoverlay element CPU load is < 10%.

gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' ! imx
vpudec ! imxipuvideotransform ! clockoverlay time-format="%Y/%m/%d %H:%M:%S" ! tee name=splitter ! queue ! imxvpuenc_h264 bitrate=1024 ! filesink location=test .mp4 splitter. ! queue ! imxeglvivsink sync=false


The above pipeline generates approximately 400Mb per hour so probably not practical for production use.

Streaming to VLC



To enable streaming to VLC we need to create a 'sdp' file, this instruction VLC to act as a rtp server, below are the contents of the file:

v=0
s=GStreamer
m=video 5000 RTP/AVP 96
c=IN IP4 127.0.0.1
a=type:broadcast
a=rtpmap:96 H264/90000
a=fmtp:96


Save the contents to file eg 'imx6.sdp' and launch VLC,  because there is no h/w acceleration for VLC on the imx6, VLC was running on a PC :

vlc --no-audio imx6.sdp

On the imx6, we are a rtp client submitting h264 payloads to the VLC server (note the inclusion of the host/ip address of the PC) :

gst-launch-1.0 -v v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !
imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 bitrate=1024 ! rtph264pay ! udpsink host=<host/ip of PC> port=5000




Simple RTSP server


And lastly, lets try running an rtp server. Fortunately there is an additional gstreamer plugin (gst-rtsp-server) with rtp support that includes an example test server. The downside is that it needs to be built from sources. You will need to checkout the 1.1.90 tag and build (similar to gstreamer-imx). Once built we can launch the example test server and pass it a pipeline similar to that used when outputting to screen (you need to export the library path so that libgstrtspserver-1.0.so is found) eg:

export ./gst-rtsp-server/gst/rtsp-server/.libs
cd ./gst-rtsp-server/examples/.libs/

./test-launch '(v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay name=pay0 pt=96 )' 


The test server listens on port 8554 therefore the rtsp URL is

rtsp://<host/ip of imx6>:8554/test

You can test by launching VLC and opening a 'Network Stream' to the URL. While streaming the CPU load on the imx6 hovered around 60%. Given this is example code it should be possible to optimise the pipeline/code to bring down this figure.

Thursday, 10 April 2014

I.MX6 - gstreamer-imx

gstreamer-imx is set of gstreamer 1.0 imx6 video plugins principally developed by Carlos Giani a.k.a dv_ (great work!). Once stable these plugins "hopefully" will supersede the Freescale BSP gstreamers 0.10 plugins which are showing their age given that gstreamer 0.10 is longer maintained.

The main driver for this work was to get a usb webcam streaming via these plugins this will be covered in the next post. In meantime below are build instructions for deploying the latest code to the debian rootfs if you fancy experimenting.  I'm finding the debian jessie build useful to jump start prototype development on the imx6 mainly due to the availability of prebuilt packages.

Here is a short video demonstrating the use of color key along with the imxipusink to hide/view video within XFCE. The video only appears for the color key 'black' hence it is visible in the terminal windows and menu. The test device was a AR6MXQ board provide by BCM Advanced Research. As a designer/provider of industrial motherboards BCM have done a good job with their first feature rich ARM development board.





The current debian rootfs already includes an older build of the gstreamer-imx plugins. Upgrading these to a newer release is fairly trivial, you can build the latest sources natively on debian as follows:

1. Remove the existing plugins

 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstimxipu.so 
 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstimxvpu.so 
 rm /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstimxeglvivsink.so 

 rm /usr/lib/libgstimxcommon.so 
 rm /usr/lib/libgstimxcommon.so.0 
 rm /usr/lib/libgstimxcommon.so.0.9.1 

2. Build lastest sources

 git clone git://github.com/Freescale/gstreamer-imx.git  
 cd gstreamer-imx  
 ./waf configure --prefix=usr --kernel-headers=/usr/include  
 ./waf  
 ./waf install  

3. Deploy plugins

 cp usr/lib/gstreamer-1.0/libgstimxipu.so /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/libgstimxvpu.so /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/libgstimxeglvivsink.so /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/gstreamer-1.0/libgstimxv4l2src.so /usr/lib/arm-linux-gnueabihf/gstreamer-1.0  
 cp usr/lib/libgstimxcommon.so.0.9.5 /usr/lib  
 cd /usr/lib  
 ln -s libgstimxcommon.so.0.9.5 libgstimxcommon.so.0  
 ln -s libgstimxcommon.so.0.9.5 libgstimxcommon.so  

4. Reboot

5. Verify the plugins are present

 root@debian-imx6:~# gst-inspect-1.0 | grep imx  
 imxipu: imxipuvideotransform: Freescale IPU video transform element  
 imxipu: imxipusink: Freescale IPU video sink  
 imxvpu: imxvpudec: Freescale VPU video decoder  
 imxvpu: imxvpuenc_h263: Freescale VPU h.263 video encoder  
 imxvpu: imxvpuenc_h264: Freescale VPU h.264 video encoder  
 imxvpu: imxvpuenc_mpeg4: Freescale VPU MPEG-4 video encoder  
 imxvpu: imxvpuenc_mjpeg: Freescale VPU motion JPEG video encoder  
 imxeglvivsink: imxeglvivsink: Freescale EGL video sink  
 imxv4l2src: imxv4l2src: V4L2 CSI Video Source  

6. Test video playback (under X11)

 gst-launch-1.0 playbin uri=file://<video file> video-sink=imxeglvivsink  

Unfortunately the elements/sinks aren't documented so you may need to refer to the source code to determine features or properties. There is also a new plugin for CSI camera sources (imxv4l2src) provided by  Philip Craig.

Monday, 24 March 2014

I.MX6 Debian Jessie (GPU/VPU) 3.10.17_beta

Following on from the previous jessie rootfs, this is an updated version with the 3.10.17_beta BSP libraries deployed. It should work on most i.mx6 devices providing you have a valid uboot and kernel. Furthermore you can also use it as a test bed for Wayland/Weston development as per my last post.

One of the primary changes in the beta BSP is the inclusion of xrandr support within the X driver. Unfortunately the driver is partially functioning, for example I couldn't get it change resolution without restarting the X server. Another change is the Vivante driver are updated to 4.6.9p13 (see).

As with the previous rootfs:

1. Download and extract the tar file (md5 e1024db60c74df661e884a28eff104f6) onto your boot media
2. Deploy a compatible uboot on your boot media
3. Deploy 3.10.17 beta kernel (or a kernel with the 4.6.9p13 patches merged) on your boot media
4. If required enable serial console support in /etc/inittab (see)
5. Setup networking as required.

By default the system boots to console mode, refer to the previous post on how to verify the rootfs is functioning correctly.

I've included some scripts to allow switching between the fb, x11 and wayland vivante libraries, these are located in /root and self explanatory.

/root/switch_to_wl.sh
/root/switch_to_fb.sh
/root/switch_to_x11.sh


If you choose to use x11 by default XFCE (lightdm) is disabled. To launch it:

service lightdm start

If you have no screen output (assuming HDMI) then (because of the xrandr feature) you will to have to manually configure the screen resolution in xorg.conf. To do this un-comment the screen SubSection and change to the correct resolution.

    SubSection     "Display"
        Modes      "U:1280x720p-60"
    EndSubSection 


If can't find the correct resolution then check the last output from /var/log/xorg.0.log. The current the resolution chosen by the X driver will be shown by the following statements:

[    38.607] (II) VIVANTE(0): Using user preference for initial modes
[    38.608] (II) VIVANTE(0): Output DISP3 BG using initial mode U:1280x720p-60
[    38.768] (II) VIVANTE(0): imxDisplayPreInit: virtual set 1280 x 720, display



The log file may also list all the possible modes available, it does this by reading EDID (assuming HDMI) which seems to be a hit or miss affair depending on your TV/monitor. For example you may see something similar to this: 

[    35.223] (II) VIVANTE(0): Modeline "U:720x576p-50"x0.0   27.00  720 732 796 864  576 581 586 625 -hsync -vsync -csync (31.2 kHz
e)
[    35.224] (II) VIVANTE(0): Modeline "U:1920x1080p-60"x0.0  148.50  1920 2008 2052 2200  1080 1084 1089 1125 +hsync +vsync -csync
(67.5 kHz e)


Replace the Modes value with correct one from xorg.0.log and restart lightdm. For example:

Modes      "U:1920x1080p-60"

Monday, 3 March 2014

I.MX6 Wayland/Weston (3.10.17_beta)

Among the many items on my to do list was to determine how complete the Wayland support was in latest 3.10.17_beta BSP release. A complete description of Wayland and Weston can be found here.

In short, Weston is a reference implementation of a Wayland compositor. In order to run Weston we need to build Wayland and Weston, as a starting point I found some outdated instructions here. From these instructions and after further investigation it became clear that the freescale Weston implementation relies on a framebuffer backend (compositor-fbdev.c). The wayland libraries seem to be very similar to their framebuffer counterparts and support double and triple buffering.There are 2 possible configurations for the framebuffer backend:
  1. A custom render (GAL2D) which use the GC320 GPU + fbdev. There's a couple of patches that implement the render (gal2d-renderer.c)
  2. Use the built-in gl_render which uses EGL/GLES for composition.
I chose to use the gl_render so that the EGL sample(s) could be run. To run weston the following packages need to built (instructions for native compilation are below).

  1. wayland 
  2. ibxkbcommon 
  3. pixman
  4. cairo
  5. weston
The packages were built for deployment on the debian jessi rootfs which was upgraded to the 3.10.17_beta BSP along a 3.10.17 kernel. Wayland/Weston were built from the master branch which currently are at version 1.4. Here is a short video demonstrating a number of Weston examples running on a UDOO (this was the most convenient device at hand to test with).



Although the video demonstrates a number of Weston examples running well, I encountered high CPU when invoking the  'weston-smoke' example which tests SHM buffer sharing. Furthermore I encountered a few lockups what's unclear is whether these are a result of using the master branches of the deployed packages or a problem within fsl wayland libraries. Given the 3.10.17 BSP is in beta it should be treated as so.

Building and deploying Wayland/Weston

Ensure you have deployed the 3.10.17 Vivante gpu headers and libaries,  ensure symbolic links point to '-wl' libraries eg:

/usr/lib/libVIVANTE.so -> libVIVANTE-wl.so
/usr/lib/libGAL.so -> libGAL-wl.so
/usr/lib/libEGL.so -> libEGL-wl.so
/usr/lib/libEGL.so.1.0 -> libEGL-wl.so


You may need to pull in addition packages to successfully complete compilation. Building cairo/pixman may require pulling in X11 packages. As mentioned these steps were completed on my debian jessi rootfs and should be transferable to other distros.

Build Wayland

export WLD=/usr
git clone git://anongit.freedesktop.org/wayland/wayland
cd wayland
./autogen.sh --prefix=$WLD --disable-documentation
make && make install
cd ..


Build libxkbcommon

git clone git://github.com/xkbcommon/libxkbcommon
cd libxkbcommon
./autogen.sh
make && make install 

cd ..

Build cairo

git clone git://anongit.freedesktop.org/cairo
cd cairo
./autogen.sh glesv2_CFLAGS="-DLINUX=1 -DEGL_API_FB -DEGL_API_WL" --enable-glesv2 --disable-xcb
make && make install
cd ..

Build pixman

git clone git://anongit.freedesktop.org/pixman
cd pixman
./autogen.sh
make && make install
cd .
.

Build Weston

git clone git://anongit.freedesktop.org/wayland/weston
cd weston


We need patch compositor-fbdev.c so that a EGL handle is retrieved from the framebuffer so that gl_render can use it. See my patch http://pastebin.com/x5gumsye

Now export these variables and build

export LDFLAGS="-lwayland-server -lwayland-client -lwayland-server -lwayland-cursor -lpixman-1"
export COMPOSITOR_LIBS="-lGLESv2 -lEGL -lGAL -lwayland-server -lxkbcommon -lpixman-1"
export COMPOSITOR_CFLAGS="-I $WLD/include -I $WLD/include/pixman-1 -L$SDK_DIR/drivers -DLINUX=1 -DEGL_API_FB -DEGL_API_WL"
export CLIENT_CFLAGS="-I $WLD/include -I $WLD/include/cairo -I $WLD/include/pixman-1"
export CLIENT_LIBS="-lGLESv2 -lEGL -lwayland-client -lwayland-cursor -lxkbcommon -lcairo"
export SIMPLE_EGL_CLIENT_CFLAGS="-DLINUX=1 -DEGL_API_FB -DEGL_API_WL -I $WLD/include"
export SIMPLE_EGL_CLIENT_LIBS="-lGLESv2 -lEGL -lwayland-client -lwayland-cursor"
export IMAGE_LIBS="-lwayland-cursor"
export WESTON_INFO_LIBS="-lwayland-client"
export EGL_TESTS_CFLAGS="-DLINUX=1 -DEGL_API_FB -DEGL_API_WL -I $WLD/include"
export EGL_TESTS_LIBS="-lEGL -lGLESv2 -lwayland-egl -lwayland-client -lcairo"
export TEST_CLIENT_LIBS="-lwayland-client -lcairo"


./autogen.sh --prefix=$WLD --disable-setuid-install --disable-x11-compositor --disable-drm-compositor --disable-rpi-compositor --disable-wayland-compositor --disable-weston-launch --disable-libunwind --disable-xwayland --disable-xwayland-test WESTON_NATIVE_BACKEND="fbdev-backend.so"

make && make install

Before we launch weston, edit the default weston.ini and comment out or remove the following lines (there no gnome terminal or chrome to launch):

[launcher]
icon=/usr/share/icons/gnome/24x24/apps/utilities-terminal.png
path=/usr/bin/gnome-terminal


[launcher]
icon=/usr/share/icons/hicolor/24x24/apps/google-chrome.png
path=/usr/bin/google-chrome


Lets enable double buffering (can also set triple buffering):

export FB_MULTI_BUFFER=2

To launch weston, asumming your are runnnig as root user ( and output to a log file):

export XDG_RUNTIME_DIR=/tmp; weston --log=weston.log --use-gl

If weston fails to launch, check the log file weston.log.

The following samples ran without problems:

weston-clickdot
weston-cliptest
weston-dnd
weston-editor
weston-flower
weston-scaler
weston-simple-egl
weston-simple-shm
weston-stacking