Bringing
Up Heterogeneous RISC-V on Allwinner SoCs (Part 2): Building the Linux
remoteproc Driver and Hardware Verification Suite
In Part
1, we laid the architectural foundation for the
Allwinner T527 / A527 (sun55i) SoC,
derived the physical memory map from the Technical Reference Manual
(TRM), detailed the ITCM/DTCM memory interfaces, and explored the
on-chip memory-mapped debugging paradigm.
In this article (Part 2), we move directly into the code and system bring-up:
- Building the Linux 7.1
sunxi_rproc.cRemoteProc driver with complete multi-segment memory routing across ITCM, DTCM, PubSRAM C, Dedicated MCU SRAM, and DDR carveouts. - Exposing live debugfs trace logs
(
/sys/kernel/debug/remoteproc/remoteproc0/trace0) via.resource_tablewithout dedicated UART cables. - Deploying the all-new
riscv-firmware/appstest suite to systematically prove co-processor boot, memory subsystems, hardware FPU, exception handling, and high-performance IPC paradigms.
1. Building
the Linux remoteproc Driver
(sunxi_rproc.c)
The Linux Remote Processor (remoteproc) framework is the
standard kernel subsystem for managing auxiliary microcontrollers on
heterogeneous SoCs. It provides standardized lifecycle management,
coordinates clock and reset domains, parses standard ELF binaries, and
configures IPC.
┌─────────────────────────────────────────────────────────────────┐
│ Linux User Space Interface │
│ │
│ echo "testBasic.elf" > /sys/class/remoteproc/rproc0/firmware │
│ echo start > /sys/class/remoteproc/rproc0/state │
│ cat /sys/kernel/debug/remoteproc/rproc0/trace0 (Live logs) │
└────────────────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Linux Kernel Driver: drivers/remoteproc/sunxi_rproc.c │
│ - struct rproc_ops sunxi_rproc_ops │
│ - devm_clk_get() / clk_prepare_enable() │
│ - devm_reset_control_get() / reset_control_deassert() │
│ - sunxi_rproc_da_to_va() (Multi-segment memory translation) │
└────────────────────────────────┬────────────────────────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ SRAM Space 0 │ │ SRAM Space 1 │ │ DDR Trace │
│ 256 KB @ │ │ 256 KB @ │ │ Carveout │
│ 0x07280000 │ │ 0x072C0000 │ │ 4 KB @ │
│(reg: r_sram) │ │(reg: r_sram1)│ │ 0x4AE00000 │
│Core:3FFC0000 │ │Core:40000000 │ │ (/trace0) │
│(Reset Vector)│ │(Expansion) │ │ │
└──────────────┘ └──────────────┘ └──────────────┘1.1
Multi-Segment Memory Routing (da_to_va) & The 256 KB
Shift Bug
On the Allwinner T527, the Device Tree node
(sun55i-a523.dtsi) registers the continuous 512 KB SRAM
windows:
r_sram: SRAM Space 0 (0x07280000Host /0x3FFC0000Core, 256 KB) — Primary Boot & Reset Window.r_sram1: SRAM Space 1 (0x072C0000Host /0x40000000Core, 256 KB) — High-speed secondary SRAM bank.
The Linux kernel driver translates device addresses (da)
declared in the ELF program headers to mapped host virtual addresses
(va) inside sunxi_rproc_da_to_va():
static void *sunxi_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len, bool *is_iomem)
{
struct sunxi_rproc *priv = rproc->priv;
/* 1. Dedicated MCU SRAM Space 0 (Host 0x07280000 / Core 0x3FFC0000, 256 KB) */
if (priv->r_sram_va) {
if (da >= 0x3FFC0000 && (da + len) <= (0x3FFC0000 + priv->r_sram_size)) {
if (is_iomem)
*is_iomem = true;
return priv->r_sram_va + (da - 0x3FFC0000);
}
if (da >= priv->r_sram_phys && (da + len) <= (priv->r_sram_phys + priv->r_sram_size)) {
if (is_iomem)
*is_iomem = true;
return priv->r_sram_va + (da - priv->r_sram_phys);
}
}
/* 2. Dedicated MCU SRAM Space 1 (Host 0x072C0000 / Core 0x40000000, 256 KB) */
if (priv->r_sram1_va) {
if (da >= 0x40000000 && (da + len) <= (0x40000000 + priv->r_sram1_size)) {
if (is_iomem)
*is_iomem = true;
return priv->r_sram1_va + (da - 0x40000000);
}
}[!IMPORTANT] The 256 KB Interconnect Shift Bug & Silicon Lockup Root Cause:
In earlier vendor drivers,da = 0x40000000was mistakenly translated topriv->r_sram_va(Host0x07280000, Space 0). However, the Allwinner hardware bus interconnect routes Core DA0x40000000to Space 1 (0x072C0000). Becausesunxi_rproc_prepare()cleared Space 1 withmemset_io(priv->r_sram1_va, 0), booting the core at0x40000000caused the core to fetch zeroes (0x00000000, illegal instruction) and lock up (WORK_MODE_REG 0x07130248 = 0x0000000B).
/* 3. RemoteProc Trace Carveout / DDR Carveouts */ if (priv->trace_va) { if (da >= priv->trace_phys && (da + len) <= (priv->trace_phys + priv->trace_size)) { if (is_iomem) *is_iomem = false; return priv->trace_va + (da - priv->trace_phys); } }
return NULL;}
### 1.2 CCF Clock & Reset Lifecycle Hooks
Clock gating and reset release are tied directly into the Linux Common Clock Framework (CCF):
```c
static int sunxi_rproc_start(struct rproc *rproc)
{
struct sunxi_rproc *priv = rproc->priv;
/* 1. Program Boot Address Register (STA_ADD_REG @ 0x07130204) */
writel(rproc->bootaddr, priv->cfg_va + E907_STA_ADD_REG);
/* 2. Deassert core run reset (RST_BUS_MCU_RISCV_CORE, bit 18) */
reset_control_deassert(priv->rst_core);
dev_info(priv->dev, "XuanTie E907 co-processor started at 0x%08llx\n",
(unsigned long long)rproc->bootaddr);
return 0;
}Because this driver executes inside kernel space with native
ioremap_wc(), we permanently removed
iomem=relaxed from our U-Boot
bootargs, restoring strict physical memory
security (CONFIG_STRICT_DEVMEM).
2. Automatic Trace
Logging via .resource_table
One of the biggest friction points during co-processor bring-up is
having to solder USB-to-UART adapters to physical pins just to read
serial printf output.
The actual resource table in riscv-firmware/common/arch_riscv/resource_table.c
uses a compile-time macro to select between trace-only mode and full
RPMsg + trace mode:
/* Trace buffer in .trace_buffer section (mapped to on-chip SRAM by linker script) */
__attribute__((used, section(".trace_buffer"), aligned(4)))
char g_rproc_trace_buffer[CONFIG_RPROC_TRACE0_LEN];
#ifdef CONFIG_RPROC_RPMSG
/* Full Resource Table: RSC_TRACE + VirtIO VDev (for /dev/rpmsg0) */
__attribute__((used, section(".resource_table"), aligned(4)))
const struct rpmsg_resource_table global_resource_table = {
.ver = 1, .num = 2,
.offset = {
offsetof(struct rpmsg_resource_table, trace),
offsetof(struct rpmsg_resource_table, vdev),
},
.trace = {
.type = RSC_TRACE,
.da = (uint32_t)&g_rproc_trace_buffer[0],
.len = sizeof(g_rproc_trace_buffer),
.name = CONFIG_RPROC_TRACE0_NAME, /* "trace0" */
},
.vdev = {
.type = RSC_VDEV,
.id = VIRTIO_ID_RPMSG,
.num_of_vrings = 2,
/* da = 0: Linux kernel allocates the vring buffers dynamically */
.vring = { {.da=0,.align=VRING_ALIGN,.num=VRING_NUM_DESCS},
{.da=0,.align=VRING_ALIGN,.num=VRING_NUM_DESCS} },
},
};
#else
/* Trace-Only Resource Table (default: no RPMsg overhead) */
__attribute__((used, section(".resource_table"), aligned(4)))
const struct standard_resource_table global_resource_table = {
.ver = 1, .num = 1,
.offset = { offsetof(struct standard_resource_table, trace) },
.trace = {
.type = RSC_TRACE,
.da = (uint32_t)&g_rproc_trace_buffer[0],
.len = sizeof(g_rproc_trace_buffer),
.name = CONFIG_RPROC_TRACE0_NAME,
},
};
#endifKey points:
- The trace buffer lives in a dedicated
.trace_bufferlinker section — not inside the.resource_tablestruct itself. This keeps the struct compact and the buffer optimally placed by the linker. da = 0on the vring entries means Linux allocates the VirtIO ring buffers dynamically at load time. Theda_to_vacallback insunxi_rproc.cmaps them into DDR viaremoteproc_alloc_vring().- When
CONFIG_RPROC_RPMSGis not set (all apps excepttestPingRpmsg), only a singleRSC_TRACEentry is declared — zero VirtIO overhead.
When Linux loads the ELF, it parses the resource table and exposes a live debugfs interface on the ARM host:
# Read live diagnostic logs directly from the running RISC-V core:
cat /sys/kernel/debug/remoteproc/remoteproc0/trace03. The
All-New riscv-firmware/apps Verification Suite
Under riscv-firmware/apps/,
seven progressive test applications validate core boot, memory mapping,
telemetry, exception handling, and inter-processor communication
paradigms:
riscv-firmware/apps/
├── testBasic/ # 1. Sanity boot, PubSRAM execution & live loop counter
├── testStringBinaryTrace0/ # 2. Hardware FPU & combined ASCII + packed binary telemetry
├── testCrash/ # 3. Hardware exception trapping (mtvec) & full register dump
├── testPing/ # 4. Ultra-low-latency Shared Memory SPSC + UIO Doorbell benchmark
│ └── linux/ # Host tools: ping_shm (C++ direct-poll), ping_uio (C++ event-driven UIO) & ping_uio.py (Python)
├── testPingRpmsg/ # 5. Standard Linux VirtIO RPMsg framework echo benchmark
│ └── linux/ # Host tools: ping_rpmsg (C++) & ping_rpmsg.py (Python)
├── testDRAMMsg/ # 6. Hybrid SRAM Control / DDR DRAM Payload buffer pool
│ └── linux/ # Host tool: ping_dram (C++)
└── exampleRiscv/ # 7. Core flight stack telemetry application| Application | Primary Architectural Feature Verified | Host Diagnostic Tool |
|---|---|---|
testBasic |
Boot entry (0x3FFC0000), SRAM
Space 0 execution, MISA probe (0x40901125), Single FPU
verification |
trace0 debugfs |
testStringBinaryTrace0 |
Hardware Single-Precision FPU
(F), packed binary telemetry |
monitor_trace.py |
testCrash |
Machine trap vector (mtvec),
illegal instruction autopsy dump |
trace0 debugfs |
testPing |
Lock-free SPSC in SRAM, Hardware Mailbox Doorbell IRQ | ping_uio /
ping_uio.py |
testPingRpmsg |
Standard VirtIO RPMsg framework
(virtio_rpmsg_bus), /dev/rpmsg0 |
ping_rpmsg /
ping_rpmsg.py |
testDRAMMsg |
Hybrid SRAM control + 1 MB DDR DRAM payload pool, PMP un-cached | ping_dram |
3.1 Step 1:
Sanity Boot & Memory Writes (testBasic)
The testBasic application boots into SRAM Space 0
(0x3FFC0000), writes initial signatures to memory, reads
the hardware MISA and mstatus registers, tests
single-precision hardware float multiplication, and executes an
incrementing counter loop:
/* apps/testBasic/main.cpp */
int main(void) {
// 1. Read standard RISC-V MISA register (CSR 0x301)
uint32_t misa = 0;
asm volatile ("csrr %0, misa" : "=r"(misa));
// 2. Write MISA and status signatures to SRAM
sram_c_loc1[0] = 0xDEADBEEF;
sram_c_loc1[1] = misa;
sram_c_loc2[0] = 0x52495343; // "RISC"
// 3. Initialize In-Memory HAL Trace ring buffer and Timer
hal::Trace::init();
hal::Timer::init();
// 4. Test Hardware Float Multiply
volatile float f_test1 = 12.5f;
volatile float f_test2 = 4.0f;
volatile float f_res = f_test1 * f_test2; // Executed on hardware FPU (F)
uint32_t count = 0;
while (1) {
count++;
sram_c_loc2[1] = count;
hal::Trace::printf("[testBasic] Heartbeat #%u | MISA=0x%08x | count=%u\n",
count, misa, count);
hal::Timer::delay_ms(1000);
}
}- Verification: Reading
/sys/kernel/debug/remoteproc/remoteproc0/trace0reveals live silicon execution:
This proves the core is running cleanly in SRAM Space 0 ([testBasic] Heartbeat #1 | MISA=0x40901125 | count=1 [testBasic] Heartbeat #2 | MISA=0x40901125 | count=20x3FFC0000) without hardware lockup.
3.2
Step 2: Hardware Single FPU & Packed Binary Telemetry
(testStringBinaryTrace0)
The XuanTie E907 on T527 features a hardware single-precision
(F) floating-point unit (MISA = 0x40901125).
testStringBinaryTrace0 executes hardware single-precision
calculations and serializes a 32-byte packed binary
TelemetryPacket alongside formatted ASCII logs:
/* apps/testStringBinaryTrace0/main.cpp */
struct __attribute__((packed)) TelemetryPacket {
uint32_t header_magic; // 0x54454C4D ("TELM")
uint32_t sequence;
uint32_t uptime_ms;
float accel_x; // Hardware float (F, single precision)
float accel_y;
float accel_z;
float sine_wave; // Hardware float (F, single precision)
uint16_t checksum;
uint16_t tail_magic; // 0x55AA
};- Verification: Run
monitor_trace.pyto stream parsed floating-point telemetry and live calculations.
3.3
Step 3: Hardware Exception Trapping & Autopsy
(testCrash)
How does a developer debug a hard fault on a co-processor running without an OS?
testCrash registers a machine-mode exception handler in
the mtvec CSR. After emitting three countdown heartbeats to
trace0, it intentionally executes an illegal instruction
(.word 0x00000000):
/* apps/testCrash/main.cpp */
for (uint32_t i = 1; i <= 3; i++) {
hal::Trace::printf("[testCrash] Normal Heartbeat #%u / 3\n", i);
hal::Timer::delay_ms(1000);
}
hal::Trace::puts("[testCrash] >>> Triggering intentional Illegal Instruction fault NOW <<<\n");
asm volatile(".word 0x00000000"); // Unimplemented opcodeWhen the illegal instruction executes:
- The E907 traps immediately into
hal::CrashHandler::handle. - It captures all 31 General Purpose Registers
(
x1–x31) and key CSRs (mepc,mcause,mtval,mstatus). - It formats and outputs a complete register crash dump to
trace0:================== HARDWARE EXCEPTION AUTOPSY ================== mepc : 0x3FFC0144 (Faulting Instruction Address in SRAM) mcause : 0x00000002 (Illegal Instruction Trap) mtval : 0x00000000 ra : 0x3FFC0188 sp : 0x3FFC5000 gp : 0x3FFC4800 x10(a0): 0x00000003 x11(a1): 0x3FFC2000 ================================================================ - It writes fatal signature
0xDEADF00Dinto SRAM Space 0 (0x3FFFFF00) before halting cleanly.
3.4
Step 4: Ultra-Low-Latency Shared Memory IPC & UIO Doorbell
(testPing)
For high-frequency control loops, traditional kernel messaging
abstractions introduce context switch latency. testPing
implements a direct, zero-copy Single Producer Single Consumer (SPSC)
queue in SRAM C synchronized via Hardware Mailbox Doorbell
interrupts:
/* apps/testPing/main.cpp */
// Check for incoming ping (SRAM flag or Mailbox Channel 1 from Linux)
bool ping_ready = (SHM_CHANNEL->host_doorbell == 1);
if (hal::MsgBox::is_rx_pending(hal::MsgBox::Channel::Channel1)) {
(void)hal::MsgBox::receive(hal::MsgBox::Channel::Channel1);
ping_ready = true;
}
if (ping_ready) {
// Copy payload, record hardware cycle count, and ring host doorbell
SHM_CHANNEL->pong_pkt.riscv_cycles = hal::Timer::get_ticks();
SHM_CHANNEL->riscv_doorbell = 1;
hal::MsgBox::send(hal::MsgBox::Channel::Channel0, 0x01); // Trigger Linux GIC SPI 147
}- Linux Host Companion Tool (
ping_uio): Instead of polling memory and burning 100% of a CPU core, the companion tool opens/dev/uio0and blocks inepoll_wait():# Run 50,000 round-trip ping-pong iterations with event-driven UIO ping_uio -n 50000- Results: Round-trip latency of 1.5 to 2.5 microseconds with 0% idle CPU utilization on the Linux host!
3.5 Step 5:
Standard Linux VirtIO RPMsg (testPingRpmsg)
When standard Linux networking or terminal abstractions are required,
testPingRpmsg connects the XuanTie E907 to the mainline
Linux virtio_rpmsg_bus subsystem using
hal::Rpmsg:
- Announces the Name Service endpoint
"rpmsg-ping-channel"over VirtIO vrings. - The Linux kernel automatically creates
/dev/rpmsg0. - Companion tool
ping_rpmsgsends and receives frames over standard Linux file descriptors (open,read,write):ping_rpmsg -n 5000
3.6
Step 6: High-Bandwidth Hybrid SRAM / DDR Streaming
(testDRAMMsg)
While on-chip SRAM provides zero-wait-state determinism, its capacity
is bounded (128 KB – 256 KB). For high-bandwidth payloads (such as
camera frames, point clouds, or large flight logs),
testDRAMMsg demonstrates a hybrid
architecture:
- Control queues (descriptors, ring pointers, doorbells) reside in fast SRAM C.
- Bulk payload buffers reside in a 1 MB DDR DRAM carveout
(
0x48100000). - The co-processor uses its Physical Memory Protection (PMP) unit to configure the DRAM window as strongly-ordered / non-cacheable, ensuring cache coherency with Linux DMA without manual flushing.
- Companion tool
ping_drambenchmarks transfers up to 4 KB per frame at >100 MB/s throughput.
4. Live Target Workflow & Firmware Switching
4.1 Compiling All Firmware and Companion Tools
From the repository root:
make -C riscv-firmwareThis builds all co-processor ELFs (testBasic.elf,
testStringBinaryTrace0.elf, testCrash.elf,
testPing.elf, testPingRpmsg.elf,
testDRAMMsg.elf) and compiles the host companion binaries
(ping_uio, ping_rpmsg,
ping_dram), staging everything into
riscv-firmware/bin/.
During Buildroot compilation, these binaries are installed directly
into /lib/firmware/ and /usr/local/bin/ on the
target root filesystem.
4.2 Dynamic Runtime Firmware Switching (No Reboots!)
The Linux remoteproc sysfs interface allows stopping,
switching, and starting co-processor firmware on the fly:
# ==============================================================================
# 1. Run Sanity Boot Test
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testBasic.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
# ==============================================================================
# 2. Run Ultra-Low-Latency Shared Memory Benchmark
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testPing.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
ping_uio -n 50000
# ==============================================================================
# 3. Run Standard Linux RPMsg Echo Test
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testPingRpmsg.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
ping_rpmsg -n 5000[!TIP] Scripting RemoteProc Transitions & Hush Token Spacing
When writing shell scripts or boot hooks to automate these firmware toggles (e.g., verifying return codes or checking
/sys/class/remoteproc/remoteproc0/state), ensure conditional checks match strict token spacing:if test "${loaded}" = "1"; thenAs detailed in the Device Tree Overlay guide, accidental whitespace like
test "${loaded}" = " 1"causes silent conditional failures in strict parsers like U-Boot's Hush shell and embedded busybox environments.
Notice that zero /dev/mem or root privilege
poking is used. All hardware interactions are managed cleanly
by the kernel drivers (sunxi_rproc.c,
uio_pdrv_genirq, virtio_rpmsg_bus), ensuring
system stability and maintaining strict memory protection
(CONFIG_STRICT_DEVMEM).
5. What's Next in Part 3
With the sunxi_rproc.c driver and
riscv-firmware/apps verification suite in place:
- The Linux host reliably loads multi-segment ELF binaries into
continuous 512 KB SRAM (Space 0 at
0x3FFC0000and Space 1 at0x40000000) and transparent DDR carveouts. - The
.resource_tableprovides live trace streaming without physical serial debug cables. - Every co-processor subsystem—clocks, resets, hardware single-precision FPU, exception trapping, direct shared memory, and VirtIO RPMsg—is systematically verified on live silicon.
In Part 3, we dive deep into all three IPC paradigms:
- Lock-free Shared SRAM + Hardware Mailbox
(
testPing): HowShmPingChannel,hal::SpscQueue, and event-driven UIO epoll deliver 1.5–2.5 µs round-trip latency. - VirtIO RPMsg (
testPingRpmsg): Standard/dev/rpmsg0integration viahal::Rpmsg. - Hybrid SRAM/DDR (
testDRAMMsg):DramSpscControlBlockdescriptor rings in fast SRAM with a 1 MB DDR carveout for > 100 MB/s bulk streaming.
Series Navigation
- Part 1: Architecture and Memory-Mapped Debugging
- Part 2: Building the Linux
remoteprocDriver and Hardware Verification Suite (You are here) - Part 3: Inter-Processor Communication (IPC) Deep Dive
