11 KiB
KPU
Some notes about the K210 KPU, which is definitely the weirdest, possibly most interesting peripheral on this SoC. Documentation doesn't seem to be available, so the information here has been reconstructed from various vendor source code.
This kind of custom hardware is pretty much impossible to understand without knowledge of the domain, in this case Convolutional Neural Networks on images. My understanding of this is rudimentary (my last brush with it was in uni) so I may be missing some obvious clues here and there.
From the datasheet
The Kendryte datasheet has the following information on the KPU:
KPU is a general-purpose neural network processor with built-in convolution, batch normalization, activation, and pooling operations. It can detect faces or objects in real time. The specific characteristics are as follows:
- Supports the fixed-point model that the mainstream training framework trains according to specific restriction rules
- There is no direct limit on the number of network layers, and each layer of convolutional neural network parameters can be configured separately, including the number of input and output channels, and the input and output line width and column height
- Support for 1x1 and 3x3 convolution kernels
1×1 and 3×3 is not a very wide range of supported convolutions, but maybe the most common ones in this specific application area…
- Support for any form of activation function
This is definitely true, Normalization functions seem to be represented as an array of 16
segments (kpu_activate_table_t).
- The maximum supported neural network parameter size for real-time work is 5MiB to 5.9MiB
- The maximum supported network parameter size when working in non-real time is (flash size - software size)
The flash size specs are somewhat of a red herring as they relate to software instead of hardware: the KPU does not have logic for loading parameters from flash.
Some other source mentions:
64 KPU which are 576bit width, supports convolution kernel. Offers 0.25TOPS@0.3W,400MHz, and when you overclock to 800MHz, it offers 0.5TOPS, meaning you can do object recognition 60fps@VGA.
Clock speed
The KPU is clocked from PLL1, with a divisor between 1 and 16. The usual clock speed in the Sipeed examples is 300, sometimes 400 MHz. According to some mentions in the data sheet it's possible to clock it to 800 MHz.
Overall execution flow
The overall execution flow is that the KPU runs a neural network layer by layer. This happens in a sequential fashion. Each layer can be considered a separate set of instructions for the KPU.
A layer can receive its input in the "AI" memory area (2MB of the memory is reserved for this, from 0x40600000 to 0x407fffff) as well as write its output there. The input and output can consist of multiple channels (R/G/B for example).
It is possible to set an interrupt to notify the host CPU when a specific layer has finished executing.
Looking at lib/drivers/kpu.c in the SDK, function ai_step, many types of CNN layers are
implemented in software instead of executed by the KPU. I suppose they accelerated the
most common multiplication-intensive layers in hardware, which is KL_K210_CONV.
Peripheral layout
The register layout of the peripheral is as follows. Source: lib/drivers/include/kpu.h.
All registers are 64-bit.
| Ofs | Name | Description |
|---|---|---|
| 0x00 | layer_argument_fifo |
Layer arguments (instructions) are submitted here |
| 0x08 | interrupt_status |
Status of pending interrupts |
| 0x10 | interrupt_raw |
|
| 0x18 | interrupt_mask |
Specifies which global interrupts are enabled |
| 0x20 | interrupt_clear |
Clear pending interrupts |
| 0x28 | fifo_threshold |
FIFO interrupt thresholds |
| 0x30 | fifo_data_out |
Data output FIFO read register |
| 0x38 | fifo_ctrl |
Flush FIFOs |
| 0x40 | eight_bit_mode |
Enable 8-bit instead of 16-bit precision |
Layer format
KPU neural network layers are represented by a series of 12 64-bit values,
submitted to the layer argument FIFO one by one. The overall structure of the bit fields is
available in lib/drivers/include/kpu.h.
It looks like the generation of models is supposed to be done offline by a tool called nnscase, which compiles TensorFlow models to a specific internal representation. The k210-specific code parts are k210_ops.cpp and k210_sim_types.h and k210_ops_body.h (serialization and deserialization). src/common/include/kernels/k210/k210_kernels.h (emulation)
0 interrupt_enabe
(register and field names are from the SDK header files, typos are as-is)
bit name
------ ----------------------
0 `int_en` Generate interrupt after layer computation finished
1 `ram_flag` ?
2 `full_add` Set in `kpu_conv2d_output_full_add`
3 `depth_wise_layer` Is a "depth-wise" layer (1 if enabled)
4..63 reserved
"depth-wise" affects many of the computations: it likely means that the layer computation mixes multiple channels so that they cannot be processed one by one.
1 image_addr
bit name
------ ----------------------
0..14 `image_src_addr` Image source address
15 reserved
16..30 `image_dst_addr` Image destination address
31..63 reserved
image_src_addr and image_dst_addr are specified in 64-byte units relative to the base of "AI" memory.
2 image_channel_num
bit name
------ ----------------------
0..9 `i_ch_num` Number of input channels (minus one)
10..31 reserved
32..41 `o_ch_num` Number of output channels (minus one)
42..47 reserved
48..57 `o_ch_num_coef` Number of output channel coefficients (minus one)
58..63 reserved
3 image_size
bit name
------ ----------------------
0..9 `i_row_wid` Input row width (minus one)
10..18 `i_col_high` Input column height (minus one)
19..31 reserved
32..41 `o_row_wid` Output row width (minus one)
42..50 `o_col_high` Output column height (minus one)
51..63 reserved
4 kernel_pool_type_cfg
bit name
------ ----------------------
0..2 `kernel_type` `filter_type_t` (see below)
3 `pad_type` Always 1
4..7 `pool_type` `pool_type_t` (see below)
8 `first_stride` ?
9 `bypass_conv` ?
10 `load_para` Load parameters (1 if enabled)
11..15 reserved
16..23 `dma_burst_size` Always 15
24..31 `pad_value` Padding value
32..63 `bwsx_base_addr` Batch normalization array base address (8-aligned, `kpu_batchnorm_argument_t`)
kpu_filter_type:
value enum
------ -----------
0 1x1
1 3x3
kpu_pool_type:
value enum description
------ -------------- ------------------
0 bypass bypass pooling (filter size 1×1, stride 1)
1 max_2_s2 max pooling (filter size 2×2, stride 2)
2 mean_2_s2 mean pooling (filter size 2×2, stride 2)
3 max_4_s4 max pooling (filter size 4×4, stride 4)
4 mean_4_s4 mean pooling (filter size 4×4, stride 4)
5 left_top_2_s2 pick left top (filter size 2×2, stride 2)
6 right_top_2_s2 pick right top (filter size 2×2, stride 2)
7 left_top_4_s4 pick left top (filter size 4×4, stride 4)
8 mean_2_s1 mean pooling (filter size 2×2, stride 1)
9 max_2_s1 max pooling (filter size 2×2, stride 1)
See kpu_pool2d in src/common/include/kernels/k210/k210_kernels.h,
as well as src/common/include/runtime/k210/k210_runtime_op_utility.h in nncase.
5 kernel_load_cfg
bit name
------ ----------------------
0 `load_coor` Always 1
1..6 `load_time` Parameter load frequency (0=once, 1=per channel?)
7..14 reserved
15..31 `para_size` Parameter (weights) size
32..63 `para_start_addr` Parameter (weights) start address (128-aligned, one byte per weight)
6 kernel_offset
bit name
------ ----------------------
0..3 `coef_column_offset` ?
4..15 `coef_row_offset` ?
16..63 reserved
7 kernel_calc_type_cfg
bit name
------ ----------------------
0..14 `channel_switch_addr` In layout channel length
15 reserved
16..19 `row_switch_addr` In layout row length
20..27 `coef_size` ?
28..30 `coef_group` ?
31 `load_act` Load activation function (1 is enabled)
32..63 `active_addr` Activation function address (256-aligned `kpu_activate_table_t`)
8 write_back_cfg
bit name
------ ----------------------
0..14 `wb_channel_switch_addr` Out layout channel length
15 reserved
16..19 `wb_row_switch_addr` Out layout row length
20..22 `wb_group` Out layout number of groups
23..63 reserved
9 conv_value
bit name
------ ----------------------
0..3 `shr_w` Convolution value shift right w
4..7 `shr_x` Convolution value shift right x
8..31 `arg_w` Convolution value w multiplier
32..55 `arg_x` Convolution value x multiplier
56..63 reserved
10 conv_value2
bit name
------ ----------------------
0..39 `arg_add` Convolution value addition/bias
40..63 reserved
11 dma_parameter
bit name
------ ----------------------
0 `send_data_out` Send data out to DMA (main memory)
1..15 reserved
16..31 `channel_byte_num` Number of bytes per out channel (minus one)
32..63 `dma_total_byte` Number of bytes total out (minus one)