Bringing Up Heterogeneous RISC-V on Allwinner SoCs (Part 2): Building the Linux `remoteproc` Driver and Hardware Verification Suite

In Part 1, we laid the architectural foundation for the Allwinner T527 / A527 (sun55i) SoC, derived the physical memory map from the Technical Reference Manual (TRM), detailed the ITCM/DTCM memory interfaces, and explored the on-chip memory-mapped debugging paradigm.

In this article (Part 2), we move directly into the code and system bring-up:

Building the Linux 7.1 sunxi_rproc.c RemoteProc driver with complete multi-segment memory routing across ITCM, DTCM, PubSRAM C, Dedicated MCU SRAM, and DDR carveouts.
Exposing live debugfs trace logs (/sys/kernel/debug/remoteproc/remoteproc0/trace0) via .resource_table without dedicated UART cables.
Deploying the all-new riscv-firmware/apps test suite to systematically prove co-processor boot, memory subsystems, hardware FPU, exception handling, and high-performance IPC paradigms.

1. Building the Linux `remoteproc` Driver (`sunxi_rproc.c`)

The Linux Remote Processor (remoteproc) framework is the standard kernel subsystem for managing auxiliary microcontrollers on heterogeneous SoCs. It provides standardized lifecycle management, coordinates clock and reset domains, parses standard ELF binaries, and configures IPC.

┌─────────────────────────────────────────────────────────────────┐
│                   Linux User Space Interface                    │
│                                                                 │
│   echo "testBasic.elf" > /sys/class/remoteproc/rproc0/firmware  │
│   echo start           > /sys/class/remoteproc/rproc0/state     │
│   cat /sys/kernel/debug/remoteproc/rproc0/trace0 (Live logs)   │
└────────────────────────────────┬────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│            Linux Kernel Driver: drivers/remoteproc/sunxi_rproc.c │
│  - struct rproc_ops sunxi_rproc_ops                             │
│  - devm_clk_get() / clk_prepare_enable()                        │
│  - devm_reset_control_get() / reset_control_deassert()          │
│  - sunxi_rproc_da_to_va() (Multi-segment memory translation)    │
└────────────────────────────────┬────────────────────────────────┘
                                 │
       ┌─────────────────────────┼─────────────────────────┐
       ▼                         ▼                         ▼
┌──────────────┐          ┌──────────────┐          ┌──────────────┐
│ SRAM Space 0 │          │ SRAM Space 1 │          │ DDR Trace    │
│  256 KB @    │          │  256 KB @    │          │ Carveout     │
│  0x07280000  │          │  0x072C0000  │          │ 4 KB @       │
│(reg: r_sram) │          │(reg: r_sram1)│          │ 0x4AE00000   │
│Core:3FFC0000 │          │Core:40000000 │          │ (/trace0)    │
│(Reset Vector)│          │(Expansion)   │          │              │
└──────────────┘          └──────────────┘          └──────────────┘

1.1 Multi-Segment Memory Routing (`da_to_va`) & The 256 KB Shift Bug

On the Allwinner T527, the Device Tree node (sun55i-a523.dtsi) registers the continuous 512 KB SRAM windows:

r_sram: SRAM Space 0 (0x07280000 Host / 0x3FFC0000 Core, 256 KB) — Primary Boot & Reset Window.
r_sram1: SRAM Space 1 (0x072C0000 Host / 0x40000000 Core, 256 KB) — High-speed secondary SRAM bank.

The Linux kernel driver translates device addresses (da) declared in the ELF program headers to mapped host virtual addresses (va) inside sunxi_rproc_da_to_va():

static void *sunxi_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len, bool *is_iomem)
{
    struct sunxi_rproc *priv = rproc->priv;

    /* 1. Dedicated MCU SRAM Space 0 (Host 0x07280000 / Core 0x3FFC0000, 256 KB) */
    if (priv->r_sram_va) {
        if (da >= 0x3FFC0000 && (da + len) <= (0x3FFC0000 + priv->r_sram_size)) {
            if (is_iomem)
                *is_iomem = true;
            return priv->r_sram_va + (da - 0x3FFC0000);
        }
        if (da >= priv->r_sram_phys && (da + len) <= (priv->r_sram_phys + priv->r_sram_size)) {
            if (is_iomem)
                *is_iomem = true;
            return priv->r_sram_va + (da - priv->r_sram_phys);
        }
    }

    /* 2. Dedicated MCU SRAM Space 1 (Host 0x072C0000 / Core 0x40000000, 256 KB) */
    if (priv->r_sram1_va) {
        if (da >= 0x40000000 && (da + len) <= (0x40000000 + priv->r_sram1_size)) {
            if (is_iomem)
                *is_iomem = true;
            return priv->r_sram1_va + (da - 0x40000000);
        }
    }

[!IMPORTANT] The 256 KB Interconnect Shift Bug & Silicon Lockup Root Cause:
In earlier vendor drivers, da = 0x40000000 was mistakenly translated to priv->r_sram_va (Host 0x07280000, Space 0). However, the Allwinner hardware bus interconnect routes Core DA 0x40000000 to Space 1 (0x072C0000). Because sunxi_rproc_prepare() cleared Space 1 with memset_io(priv->r_sram1_va, 0), booting the core at 0x40000000 caused the core to fetch zeroes (0x00000000, illegal instruction) and lock up (WORK_MODE_REG 0x07130248 = 0x0000000B).
/* 3. RemoteProc Trace Carveout / DDR Carveouts */ if (priv->trace_va) { if (da >= priv->trace_phys && (da + len) <= (priv->trace_phys + priv->trace_size)) { if (is_iomem) *is_iomem = false; return priv->trace_va + (da - priv->trace_phys); } }

return NULL;

}


### 1.2 CCF Clock & Reset Lifecycle Hooks
Clock gating and reset release are tied directly into the Linux Common Clock Framework (CCF):

```c
static int sunxi_rproc_start(struct rproc *rproc)
{
    struct sunxi_rproc *priv = rproc->priv;

    /* 1. Program Boot Address Register (STA_ADD_REG @ 0x07130204) */
    writel(rproc->bootaddr, priv->cfg_va + E907_STA_ADD_REG);

    /* 2. Deassert core run reset (RST_BUS_MCU_RISCV_CORE, bit 18) */
    reset_control_deassert(priv->rst_core);

    dev_info(priv->dev, "XuanTie E907 co-processor started at 0x%08llx\n",
             (unsigned long long)rproc->bootaddr);
    return 0;
}

Because this driver executes inside kernel space with native ioremap_wc(), we permanently removed iomem=relaxed from our U-Boot bootargs, restoring strict physical memory security (CONFIG_STRICT_DEVMEM).

2. Automatic Trace Logging via `.resource_table`

One of the biggest friction points during co-processor bring-up is having to solder USB-to-UART adapters to physical pins just to read serial printf output.

The actual resource table in riscv-firmware/common/arch_riscv/resource_table.c uses a compile-time macro to select between trace-only mode and full RPMsg + trace mode:

/* Trace buffer in .trace_buffer section (mapped to on-chip SRAM by linker script) */
__attribute__((used, section(".trace_buffer"), aligned(4)))
char g_rproc_trace_buffer[CONFIG_RPROC_TRACE0_LEN];

#ifdef CONFIG_RPROC_RPMSG
/* Full Resource Table: RSC_TRACE + VirtIO VDev (for /dev/rpmsg0) */
__attribute__((used, section(".resource_table"), aligned(4)))
const struct rpmsg_resource_table global_resource_table = {
    .ver = 1, .num = 2,
    .offset = {
        offsetof(struct rpmsg_resource_table, trace),
        offsetof(struct rpmsg_resource_table, vdev),
    },
    .trace = {
        .type = RSC_TRACE,
        .da   = (uint32_t)&g_rproc_trace_buffer[0],
        .len  = sizeof(g_rproc_trace_buffer),
        .name = CONFIG_RPROC_TRACE0_NAME,  /* "trace0" */
    },
    .vdev = {
        .type          = RSC_VDEV,
        .id            = VIRTIO_ID_RPMSG,
        .num_of_vrings = 2,
        /* da = 0: Linux kernel allocates the vring buffers dynamically */
        .vring = { {.da=0,.align=VRING_ALIGN,.num=VRING_NUM_DESCS},
                   {.da=0,.align=VRING_ALIGN,.num=VRING_NUM_DESCS} },
    },
};
#else
/* Trace-Only Resource Table (default: no RPMsg overhead) */
__attribute__((used, section(".resource_table"), aligned(4)))
const struct standard_resource_table global_resource_table = {
    .ver = 1, .num = 1,
    .offset = { offsetof(struct standard_resource_table, trace) },
    .trace = {
        .type = RSC_TRACE,
        .da   = (uint32_t)&g_rproc_trace_buffer[0],
        .len  = sizeof(g_rproc_trace_buffer),
        .name = CONFIG_RPROC_TRACE0_NAME,
    },
};
#endif

Key points:

The trace buffer lives in a dedicated .trace_buffer linker section — not inside the .resource_table struct itself. This keeps the struct compact and the buffer optimally placed by the linker.
da = 0 on the vring entries means Linux allocates the VirtIO ring buffers dynamically at load time. The da_to_va callback in sunxi_rproc.c maps them into DDR via remoteproc_alloc_vring().
When CONFIG_RPROC_RPMSG is not set (all apps except testPingRpmsg), only a single RSC_TRACE entry is declared — zero VirtIO overhead.

When Linux loads the ELF, it parses the resource table and exposes a live debugfs interface on the ARM host:

# Read live diagnostic logs directly from the running RISC-V core:
cat /sys/kernel/debug/remoteproc/remoteproc0/trace0

3. The All-New `riscv-firmware/apps` Verification Suite

Under riscv-firmware/apps/, seven progressive test applications validate core boot, memory mapping, telemetry, exception handling, and inter-processor communication paradigms:

riscv-firmware/apps/
├── testBasic/               # 1. Sanity boot, PubSRAM execution & live loop counter
├── testStringBinaryTrace0/  # 2. Hardware FPU & combined ASCII + packed binary telemetry
├── testCrash/               # 3. Hardware exception trapping (mtvec) & full register dump
├── testPing/                # 4. Ultra-low-latency Shared Memory SPSC + UIO Doorbell benchmark
│   └── linux/               #    Host tools: ping_shm (C++ direct-poll), ping_uio (C++ event-driven UIO) & ping_uio.py (Python)
├── testPingRpmsg/           # 5. Standard Linux VirtIO RPMsg framework echo benchmark
│   └── linux/               #    Host tools: ping_rpmsg (C++) & ping_rpmsg.py (Python)
├── testDRAMMsg/             # 6. Hybrid SRAM Control / DDR DRAM Payload buffer pool
│   └── linux/               #    Host tool: ping_dram (C++)
└── exampleRiscv/            # 7. Core flight stack telemetry application

Application	Primary Architectural Feature Verified	Host Diagnostic Tool
`testBasic`	Boot entry (`0x3FFC0000`), SRAM Space 0 execution, MISA probe (`0x40901125`), Single FPU verification	`trace0` debugfs
`testStringBinaryTrace0`	Hardware Single-Precision FPU (`F`), packed binary telemetry	`monitor_trace.py`
`testCrash`	Machine trap vector (`mtvec`), illegal instruction autopsy dump	`trace0` debugfs
`testPing`	Lock-free SPSC in SRAM, Hardware Mailbox Doorbell IRQ	`ping_uio` / `ping_uio.py`
`testPingRpmsg`	Standard VirtIO RPMsg framework (`virtio_rpmsg_bus`), `/dev/rpmsg0`	`ping_rpmsg` / `ping_rpmsg.py`
`testDRAMMsg`	Hybrid SRAM control + 1 MB DDR DRAM payload pool, PMP un-cached	`ping_dram`

3.1 Step 1: Sanity Boot & Memory Writes (`testBasic`)

The testBasic application boots into SRAM Space 0 (0x3FFC0000), writes initial signatures to memory, reads the hardware MISA and mstatus registers, tests single-precision hardware float multiplication, and executes an incrementing counter loop:

/* apps/testBasic/main.cpp */
int main(void) {
    // 1. Read standard RISC-V MISA register (CSR 0x301)
    uint32_t misa = 0;
    asm volatile ("csrr %0, misa" : "=r"(misa));

    // 2. Write MISA and status signatures to SRAM
    sram_c_loc1[0] = 0xDEADBEEF;
    sram_c_loc1[1] = misa;
    sram_c_loc2[0] = 0x52495343; // "RISC"

    // 3. Initialize In-Memory HAL Trace ring buffer and Timer
    hal::Trace::init();
    hal::Timer::init();

    // 4. Test Hardware Float Multiply
    volatile float f_test1 = 12.5f;
    volatile float f_test2 = 4.0f;
    volatile float f_res = f_test1 * f_test2; // Executed on hardware FPU (F)

    uint32_t count = 0;
    while (1) {
        count++;
        sram_c_loc2[1] = count;
        hal::Trace::printf("[testBasic] Heartbeat #%u | MISA=0x%08x | count=%u\n",
                           count, misa, count);
        hal::Timer::delay_ms(1000);
    }
}

Verification: Reading /sys/kernel/debug/remoteproc/remoteproc0/trace0 reveals live silicon execution:
```
[testBasic] Heartbeat #1 | MISA=0x40901125 | count=1
[testBasic] Heartbeat #2 | MISA=0x40901125 | count=2
```
This proves the core is running cleanly in SRAM Space 0 (0x3FFC0000) without hardware lockup.

3.2 Step 2: Hardware Single FPU & Packed Binary Telemetry (`testStringBinaryTrace0`)

The XuanTie E907 on T527 features a hardware single-precision (F) floating-point unit (MISA = 0x40901125). testStringBinaryTrace0 executes hardware single-precision calculations and serializes a 32-byte packed binary TelemetryPacket alongside formatted ASCII logs:

/* apps/testStringBinaryTrace0/main.cpp */
struct __attribute__((packed)) TelemetryPacket {
    uint32_t header_magic;  // 0x54454C4D ("TELM")
    uint32_t sequence;
    uint32_t uptime_ms;
    float    accel_x;       // Hardware float (F, single precision)
    float    accel_y;
    float    accel_z;
    float    sine_wave;     // Hardware float (F, single precision)
    uint16_t checksum;
    uint16_t tail_magic;    // 0x55AA
};

Verification: Run monitor_trace.py to stream parsed floating-point telemetry and live calculations.

3.3 Step 3: Hardware Exception Trapping & Autopsy (`testCrash`)

How does a developer debug a hard fault on a co-processor running without an OS?

testCrash registers a machine-mode exception handler in the mtvec CSR. After emitting three countdown heartbeats to trace0, it intentionally executes an illegal instruction (.word 0x00000000):

/* apps/testCrash/main.cpp */
for (uint32_t i = 1; i <= 3; i++) {
    hal::Trace::printf("[testCrash] Normal Heartbeat #%u / 3\n", i);
    hal::Timer::delay_ms(1000);
}

hal::Trace::puts("[testCrash] >>> Triggering intentional Illegal Instruction fault NOW <<<\n");
asm volatile(".word 0x00000000"); // Unimplemented opcode

When the illegal instruction executes:

The E907 traps immediately into hal::CrashHandler::handle.
It captures all 31 General Purpose Registers (x1–x31) and key CSRs (mepc, mcause, mtval, mstatus).

It formats and outputs a complete register crash dump to trace0:

================== HARDWARE EXCEPTION AUTOPSY ==================
mepc   : 0x3FFC0144 (Faulting Instruction Address in SRAM)
mcause : 0x00000002 (Illegal Instruction Trap)
mtval  : 0x00000000
ra     : 0x3FFC0188  sp : 0x3FFC5000  gp : 0x3FFC4800
x10(a0): 0x00000003  x11(a1): 0x3FFC2000
================================================================

It writes fatal signature 0xDEADF00D into SRAM Space 0 (0x3FFFFF00) before halting cleanly.

3.4 Step 4: Ultra-Low-Latency Shared Memory IPC & UIO Doorbell (`testPing`)

For high-frequency control loops, traditional kernel messaging abstractions introduce context switch latency. testPing implements a direct, zero-copy Single Producer Single Consumer (SPSC) queue in SRAM C synchronized via Hardware Mailbox Doorbell interrupts:

/* apps/testPing/main.cpp */
// Check for incoming ping (SRAM flag or Mailbox Channel 1 from Linux)
bool ping_ready = (SHM_CHANNEL->host_doorbell == 1);
if (hal::MsgBox::is_rx_pending(hal::MsgBox::Channel::Channel1)) {
    (void)hal::MsgBox::receive(hal::MsgBox::Channel::Channel1);
    ping_ready = true;
}

if (ping_ready) {
    // Copy payload, record hardware cycle count, and ring host doorbell
    SHM_CHANNEL->pong_pkt.riscv_cycles = hal::Timer::get_ticks();
    SHM_CHANNEL->riscv_doorbell = 1;
    hal::MsgBox::send(hal::MsgBox::Channel::Channel0, 0x01); // Trigger Linux GIC SPI 147
}

Linux Host Companion Tool (ping_uio): Instead of polling memory and burning 100% of a CPU core, the companion tool opens /dev/uio0 and blocks in epoll_wait():
```
# Run 50,000 round-trip ping-pong iterations with event-driven UIO
ping_uio -n 50000
```
- Results: Round-trip latency of 1.5 to 2.5 microseconds with 0% idle CPU utilization on the Linux host!

3.5 Step 5: Standard Linux VirtIO RPMsg (`testPingRpmsg`)

When standard Linux networking or terminal abstractions are required, testPingRpmsg connects the XuanTie E907 to the mainline Linux virtio_rpmsg_bus subsystem using hal::Rpmsg:

Announces the Name Service endpoint "rpmsg-ping-channel" over VirtIO vrings.
The Linux kernel automatically creates /dev/rpmsg0.
Companion tool ping_rpmsg sends and receives frames over standard Linux file descriptors (open, read, write):
```
ping_rpmsg -n 5000
```

3.6 Step 6: High-Bandwidth Hybrid SRAM / DDR Streaming (`testDRAMMsg`)

While on-chip SRAM provides zero-wait-state determinism, its capacity is bounded (128 KB – 256 KB). For high-bandwidth payloads (such as camera frames, point clouds, or large flight logs), testDRAMMsg demonstrates a hybrid architecture:

Control queues (descriptors, ring pointers, doorbells) reside in fast SRAM C.
Bulk payload buffers reside in a 1 MB DDR DRAM carveout (0x48100000).
The co-processor uses its Physical Memory Protection (PMP) unit to configure the DRAM window as strongly-ordered / non-cacheable, ensuring cache coherency with Linux DMA without manual flushing.
Companion tool ping_dram benchmarks transfers up to 4 KB per frame at >100 MB/s throughput.

4. Live Target Workflow & Firmware Switching

4.1 Compiling All Firmware and Companion Tools

From the repository root:

make -C riscv-firmware

This builds all co-processor ELFs (testBasic.elf, testStringBinaryTrace0.elf, testCrash.elf, testPing.elf, testPingRpmsg.elf, testDRAMMsg.elf) and compiles the host companion binaries (ping_uio, ping_rpmsg, ping_dram), staging everything into riscv-firmware/bin/.

During Buildroot compilation, these binaries are installed directly into /lib/firmware/ and /usr/local/bin/ on the target root filesystem.

4.2 Dynamic Runtime Firmware Switching (No Reboots!)

The Linux remoteproc sysfs interface allows stopping, switching, and starting co-processor firmware on the fly:

# ==============================================================================
# 1. Run Sanity Boot Test
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testBasic.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
cat /sys/kernel/debug/remoteproc/remoteproc0/trace0

# ==============================================================================
# 2. Run Ultra-Low-Latency Shared Memory Benchmark
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testPing.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
ping_uio -n 50000

# ==============================================================================
# 3. Run Standard Linux RPMsg Echo Test
# ==============================================================================
echo stop > /sys/class/remoteproc/remoteproc0/state
echo "testPingRpmsg.elf" > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
ping_rpmsg -n 5000

[!TIP] Scripting RemoteProc Transitions & Hush Token Spacing

When writing shell scripts or boot hooks to automate these firmware toggles (e.g., verifying return codes or checking /sys/class/remoteproc/remoteproc0/state), ensure conditional checks match strict token spacing:
if test "${loaded}" = "1"; then
As detailed in the Device Tree Overlay guide, accidental whitespace like test "${loaded}" = " 1" causes silent conditional failures in strict parsers like U-Boot's Hush shell and embedded busybox environments.

Notice that zero /dev/mem or root privilege poking is used. All hardware interactions are managed cleanly by the kernel drivers (sunxi_rproc.c, uio_pdrv_genirq, virtio_rpmsg_bus), ensuring system stability and maintaining strict memory protection (CONFIG_STRICT_DEVMEM).

5. What's Next in Part 3

With the sunxi_rproc.c driver and riscv-firmware/apps verification suite in place:

The Linux host reliably loads multi-segment ELF binaries into continuous 512 KB SRAM (Space 0 at 0x3FFC0000 and Space 1 at 0x40000000) and transparent DDR carveouts.
The .resource_table provides live trace streaming without physical serial debug cables.
Every co-processor subsystem—clocks, resets, hardware single-precision FPU, exception trapping, direct shared memory, and VirtIO RPMsg—is systematically verified on live silicon.

In Part 3, we dive deep into all three IPC paradigms:

Lock-free Shared SRAM + Hardware Mailbox (testPing): How ShmPingChannel, hal::SpscQueue, and event-driven UIO epoll deliver 1.5–2.5 µs round-trip latency.
VirtIO RPMsg (testPingRpmsg): Standard /dev/rpmsg0 integration via hal::Rpmsg.
Hybrid SRAM/DDR (testDRAMMsg): DramSpscControlBlock descriptor rings in fast SRAM with a 1 MB DDR carveout for > 100 MB/s bulk streaming.

Part 1: Architecture and Memory-Mapped Debugging
Part 2: Building the Linux remoteproc Driver and Hardware Verification Suite (You are here)
Part 3: Inter-Processor Communication (IPC) Deep Dive

All things embedded

Monday, September 21, 2026