I've finally reached a point with reverse engineering where we can start evaluating the usefulness of the NPU for LMMs. I've crafted a basic library (rk3588-npu) with sufficient features for initial integration. A good reference application for integration is llama2.c because its a single C file, the code structure is straightforward to follow and more importantly modify. We will use TinyStories (stories110m) for testing since the models are relatively small, making it easier to troubleshoot when the outputs go a stray. Credit to karpathy for providing llama2.c
Note, I've set the cpu and npu cores to max clk speeds.
We'll start with running run.c against stories110m, this is the fp32
version using the cpu with a single thread (single core). As we see
roughly 9.7 tokens/second.
Next I converted run.c to use fp16 (_Float16) along with the weights. We
see a slight drop in performance to roughly 9.3 tokens/s as arithmetic
operations require a conversion back to fp32.
As with the fp32 version a single core run at a 100% as the code is single thread.
The next step was to offload all FP16 multiplications to the NPU. With a vocabulary size of 32,000, the largest multiplication is 768 x 32,000, with others being 768 x 768, 768 x 2048, and 2048 x 768. For efficient execution, the model weights need to reside entirely in memory, accessible to the NPU. This requires them to be within the 4GB address space, which can be problematic for larger models. In our case, the weights are roughly 256MB, requiring an expansion of the kernel CMA memory allocation to 512MB. Additionally, the weights needed conversion to the NPU format.
The changes result in a additional uplift of roughly 21 to 23 tokens/s depending on the length of the output as per the video below. Conservatively we could say a doubling.
CPU fluctuates between 30-60% for the single. The CPU is still critical for a number of reason:
1. We're still having to rely on memory copies to send/receive the remaining data for the multiplication to occur on the NPU.
2. Invocation of the NPU kernel driver requires CPU cycles.
3. The rest of the llam2.c code stills runs of the CPU.
Although the results look promising we need to bear mind that TinyStories is very small model as per the architecture. Furthermore its fortunate that the converted weights can fit in memory without having to shuffle weights between userspace and physical memory. In additional fp16 format would further limit the possibility for larger models to run efficiently. So conclusion so far there is some uplift but mileage will vary depending on model size and number of layers.
One of the interesting features of the RK3588 is the pcie controller because of it support for a Gen3 X4 link. I'd started looking into using the controller for a forth coming project and subsequently this lead me to the idea of testing the controller against a external GPU card to gain an understanding of it limitations and potential. From what I understand Jeff Geerling has been a similar journey with the RPI CM4 and has had limited success with help from numerous developers. Furthermore there was a Radxa tweet which a gave a teasing glimpse of the working GPU. So lets see what is or isn't possible using a Rock-5b.
I'd managed to get hold of a Radeon R7 520 (XFX R7 250 low-profile) card along a with M.2 Key M Extender Cable to PCIE x16 Graphics Card Riser Adapter. To power the card I'd reused a old LR1007 120W 12VDC ATX board which was to hand. Setup as shown below, we reuse the nvme slot for the m.2 adapter and revert back to an sd card for booting an OS. I'd used the Radxa debian image with a custom compiled Radxa kernel to include the graphics card drivers and fixes. Having reviewed the pcie BAR definitions in the rk3588.dtsi there should be enough address space available for the card to use. After removing the hdmi and mali drivers from kernel config, I initially tried the amdgpu driver but that seems to report an error and no display output
[ 11.844163] amdgpu 0000:01:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring gfx test failed (-110) [ 11.844378] [drm:amdgpu_device_init [amdgpu]] *ERROR* hw_init of IP block <gfx_v6_0> failed -110 [ 11.844383] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_init failed [ 11.844388] amdgpu 0000:01:00.0: amdgpu: Fatal error during GPU init [ 11.844414] amdgpu 0000:01:00.0: amdgpu: amdgpu: finishing device. [ 11.846559] [drm] amdgpu: ttm finalized [ 11.848018] amdgpu: probe of 0000:01:00.0 failed with error -110
The radeon driver fared slightly better with a similar error but at least display output for console login
[ 12.059398] [drm:r600_ring_test [radeon]] *ERROR* radeon: ring 0 test failed (scratch(0x850C)=0xCAFEDEAD) [ 12.059408] radeon 0000:01:00.0: disabling GPU acceleration
The was puzzling as the card relies on pcie memory mapped I/O which the RK3588 should see a standard memory and be able to read/write too. It turns out Peter Geis who was attempting to mainline a pcie driver for the RK3566 and raised 2 issues per this thread which Rockchip replied too. The same issues weren't improved/fixed on the RK3588 as mentioned here . In simple terms for our requirements:
1. For the pcie dma transfers memory allocation are limited to 32bits so a 4GB board might not see an issue. While a 8GB board like mine the kernel could pick an address range above 4GB.
2. AMD cards rely on pcie snooping, there is no CPU snooping on the RK3588 interconnect. So any cache copies of the same device memory block won't get updated to remain in sync.
If we hack the Radeon driver to work around these issues we get:
[ 12.529087] [drm] ring test on 0 succeeded in 1 usecs [ 12.529094] [drm] ring test on 1 succeeded in 1 usecs [ 12.529102] [drm] ring test on 2 succeeded in 1 usecs [ 12.529121] [drm] ring test on 3 succeeded in 8 usecs [ 12.529132] [drm] ring test on 4 succeeded in 3 usecs [ 12.706419] [drm] ring test on 5 succeeded in 2 usecs [ 12.706427] [drm] UVD initialized successfully. [ 12.816582] [drm] ring test on 6 succeeded in 18 usecs [ 12.816625] [drm] ring test on 7 succeeded in 5 usecs [ 12.816627] [drm] VCE initialized successfully. [ 12.816879] [drm:si_irq_set [radeon]] si_irq_set: sw int gfx [ 12.816921] [drm] ib test on ring 0 succeeded in 0 usecs [ 12.816989] [drm:si_irq_set [radeon]] si_irq_set: sw int cp1 [ 12.817028] [drm] ib test on ring 1 succeeded in 0 usecs [ 12.817088] [drm:si_irq_set [radeon]] si_irq_set: sw int cp2 [ 12.817127] [drm] ib test on ring 2 succeeded in 0 usecs [ 12.817185] [drm:si_irq_set [radeon]] si_irq_set: sw int dma [ 12.817224] [drm] ib test on ring 3 succeeded in 0 usecs [ 12.817281] [drm:si_irq_set [radeon]] si_irq_set: sw int dma1 [ 12.817319] [drm] ib test on ring 4 succeeded in 0 usecs [ 13.477677] [drm] ib test on ring 5 succeeded [ 13.984454] [drm] ib test on ring 6 succeeded [ 14.491404] [drm] ib test on ring 7 succeeded ...
[ 14.549296] [drm] Initialized radeon 2.50.0 20080528 for 0000:01:00.0 on minor 1
So potentially we have graphics acceleration ... let try kmstest
rock@rock-5b:~$ kmstest trying to open device 'i915'...failed trying to open device 'amdgpu'...failed trying to open device 'radeon'...done main: All ok!
The 'bus error' indicates a memory alignment issue and turns out to be a bit of a of rabbit hole. To fix the Radeon kernel driver we are ensuring the cards memory is mapped as 'Device memory' type Device-nGnRnE. If it were 'Normal Memory' then unaligned access is allowed. This implies fixing up userspace drivers/applications as these errors are encountered as these applications can directly manlipulate the cards memory. For this particular bus error it was caused by a memcpy in the radeon gallium driver and fixed applied there and as shown in the video kmscube runs
=================================== Using modifier ffffffffffffff Modifiers failed! Using modifier ffffffffffffff Modifiers failed! Rendered 120 frames in 2.000246 sec (59.992635 fps) Rendered 240 frames in 4.000428 sec (59.993577 fps) Rendered 361 frames in 6.016865 sec (59.998019 fps) Rendered 481 frames in 8.017015 sec (59.997390 fps) Rendered 601 frames in 10.017050 sec (59.997704 fps) Rendered 721 frames in 12.017079 sec (59.997942 fps) Rendered 841 frames in 14.017118 sec (59.998067 fps) Rendered 961 frames in 16.017314 sec (59.997574 fps) Rendered 1082 frames in 18.033850 sec (59.998280 fps)
Similiar fixes were applied to glmark2-drm & glmark2-es2-drm to run successfully (1680x1050 resolution) although the terrain scene displayed a bunch of colored bars on the screen.
======================================================= glmark2 2021.12 ======================================================= OpenGL Information GL_VENDOR: AMD GL_RENDERER: AMD VERDE (DRM 2.50.0, 5.10.110-99-rockchip-g6e21553c2116, LLVM 11.0.1) GL_VERSION: 4.5 (Compatibility Profile) Mesa 20.3.5 Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0 samples=0 Surface Size: 1680x1050 fullscreen ======================================================= [build] use-vbo=false: FPS: 939 FrameTime: 1.066 ms [build] use-vbo=true: FPS: 2411 FrameTime: 0.415 ms [texture] texture-filter=nearest: FPS: 1957 FrameTime: 0.511 ms [texture] texture-filter=linear: FPS: 1958 FrameTime: 0.511 ms [texture] texture-filter=mipmap: FPS: 2003 FrameTime: 0.499 ms [shading] shading=gouraud: FPS: 1975 FrameTime: 0.506 ms [shading] shading=blinn-phong-inf: FPS: 1973 FrameTime: 0.507 ms [shading] shading=phong: FPS: 1976 FrameTime: 0.506 ms [shading] shading=cel: FPS: 1974 FrameTime: 0.507 ms [bump] bump-render=high-poly: FPS: 1739 FrameTime: 0.575 ms [bump] bump-render=normals: FPS: 2373 FrameTime: 0.422 ms [bump] bump-render=height: FPS: 2330 FrameTime: 0.429 ms [effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 1254 FrameTime: 0.798 ms [effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 707 FrameTime: 1.415 ms [pulsar] light=false:quads=5:texture=false: FPS: 1338 FrameTime: 0.747 ms [desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 456 FrameTime: 2.194 ms [desktop] effect=shadow:windows=4: FPS: 600 FrameTime: 1.667 ms [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 214 FrameTime: 4.684 ms [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 233 FrameTime: 4.306 ms [buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 347 FrameTime: 2.885 ms [ideas] speed=duration: FPS: 1430 FrameTime: 0.700 ms [jellyfish] <default>: FPS: 806 FrameTime: 1.242 ms [terrain] <default>: FPS: 150 FrameTime: 6.706 ms [shadow] <default>: FPS: 843 FrameTime: 1.188 ms [refract] <default>: FPS: 115 FrameTime: 8.718 ms [conditionals] fragment-steps=0:vertex-steps=0: FPS: 1970 FrameTime: 0.508 ms [conditionals] fragment-steps=5:vertex-steps=0: FPS: 1980 FrameTime: 0.505 ms [conditionals] fragment-steps=0:vertex-steps=5: FPS: 1972 FrameTime: 0.507 ms [function] fragment-complexity=low:fragment-steps=5: FPS: 1979 FrameTime: 0.505 ms [function] fragment-complexity=medium:fragment-steps=5: FPS: 1971 FrameTime: 0.507 ms [loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 1972 FrameTime: 0.507 ms [loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 1972 FrameTime: 0.507 ms [loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 1968 FrameTime: 0.508 ms ======================================================= glmark2 Score: 1450 =======================================================
Next up was to see if startx would run, unfortunately it drops out with a shader compiler error. Looks like glamor is using egl but encounters an opengl shader to compile, requires further investigation.
[ 7916.924] (II) modeset(0): Modeline "360x202"x119.0 11.25 360 372 404 448 202 204 206 211 doublescan -hsync +vsync (25.1 kHz d) [ 7916.924] (II) modeset(0): Modeline "360x202"x118.3 10.88 360 384 400 440 202 204 206 209 doublescan +hsync -vsync (24.7 kHz d) [ 7916.924] (II) modeset(0): Modeline "320x180"x119.7 9.00 320 332 360 400 180 181 184 188 doublescan -hsync +vsync (22.5 kHz d) [ 7916.924] (II) modeset(0): Modeline "320x180"x118.6 8.88 320 344 360 400 180 181 184 187 doublescan +hsync -vsync (22.2 kHz d) [ 7916.925] (II) modeset(0): Output DVI-D-1 status changed to disconnected. [ 7916.925] (II) modeset(0): EDID for output DVI-D-1 [ 7916.939] (II) modeset(0): Output VGA-1 status changed to disconnected. [ 7916.939] (II) modeset(0): EDID for output VGA-1 [ 7916.939] (II) modeset(0): Output HDMI-1 connected [ 7916.939] (II) modeset(0): Output DVI-D-1 disconnected [ 7916.939] (II) modeset(0): Output VGA-1 disconnected [ 7916.939] (II) modeset(0): Using exact sizes for initial modes [ 7916.939] (II) modeset(0): Output HDMI-1 using initial mode 1680x1050 +0+0 [ 7916.939] (==) modeset(0): Using gamma correction (1.0, 1.0, 1.0) [ 7916.939] (==) modeset(0): DPI set to (96, 96) [ 7916.939] (II) Loading sub module "fb" [ 7916.939] (II) LoadModule: "fb" [ 7916.940] (II) Loading /usr/lib/xorg/modules/libfb.so [ 7916.944] (II) Module fb: vendor="X.Org Foundation" [ 7916.944] compiled for 1.20.11, module version = 1.0.0 [ 7916.944] ABI class: X.Org ANSI C Emulation, version 0.4 [ 7916.964] Failed to compile VS: 0:1(1): error: syntax error, unexpected NEW_IDENTIFIER
Lastly I installed vappi to attempt video playback unfortunately even after fixing a couple of bus errors in galmium theres more to fix. So this pretty much sums up the nature of the problem to address. Furthermore this does raise the question is the tweet from Radxa using acclerated graphics given the hardware restrictions of the RK3588.
