FreeRTOS trace HW/SW with nw.js application

While working on CleanFlight Linux with a I/O micro-controller for extra peripherals, there is no straight forward way to get performance, trace, and general information about the state of the firmware on the micro-controller.


Goal:

• Provide an open source solution to get metrics on the firmware using traceSWO port.

Implementation:

• Use FreeRTOS trace messages and send them out one of the channels of SWO port
• Also allow setting ETM tracing
• Create a GUI (nw.js) and command line tools to parse and show trace data from FreeRTOS
• GUI will show live line/bar charts of Task performance and I/O
• python tools to parse/mine data allows custom reports.

I have several GD32F103C8 red boards, use these to decode the SWO and send trace information to nw.js application.

Thoughts? suggestions? 

Cleanflight port to Linux (LinuxFlight) status

Created a new CLI (command Line Interface) module based on C++ and uses msgProtocol for subscribing/publishing messages.  This allows components (bash shell) or transports (UDP/TCP (Configurator)) to use the CLI at the same time.  So, now one instance of CLI can process messages from anywhere. For example: Linux has bash shell so, it would be nice not to startup Configurator to get basic info.  Use the bash shell

Here it is:
LinuxFlight>help
status - show status
help
tasks - show task stats
version - show v
LinuxFlight>status

System Uptime: 274 secondsVoltage: 0 * 0.1V (0S battery - NOT PRESENT)CPU Clock=500MHz, GYRO=MPU9250, ACC=MPU9250, MAG=AK8963
Stack size: 790724, Stack address: 0xc10b0CPU:0%, cycle time: 23226, GYRO rate: 43, RX rate: 282, System rate: 10Arming disable flags: TH
LinuxFlight>tasks

LinuxFlight>tasksTask list             rate/hz  max/us  avg/us maxload avgload     total/ms
00 - (         SYSTEM)     10       3       1    0.5%    0.5%         0
01 - (            PID)    444   22177     523  985.1%   23.7%      1270
   - (           GYRO)    889
02 - (          ACCEL)    997   21337     723 2127.7%   72.5%      1822
03 - (       ATTITUDE)     99     327      14    3.7%    0.6%         5
04 - (             RX)    143     166      36    2.8%    1.0%         9
05 - (         SERIAL)    101      87       4    1.3%    0.5%         1
08 - (BATTERY_CURRENT)    149      41       2    1.1%    0.5%         0
09 - ( BATTERY_ALERTS)      4       7       2    0.5%    0.5%         0
10 - (            GPS)     99     129       6    1.7%    0.5%         2
11 - (        COMPASS)     39   22825     756   89.5%    3.4%       158
14 - (      TELEMETRY)    255      39       2    1.4%    0.5%         0
15 - (       LEDSTRIP)    101    2522      74   25.9%    1.2%        30
RX Check Function                   0       0                         0
Total (excluding SERIAL)                      3239.9%
LinuxFlight>


Next is to get the battery voltage reporting and figure out the issue with the loading.

Might have to try the RT kernel but, first load LinuxFlight on to its own processor.  Also, might have to remove CPUFREQ it is known to impact realtime performance.

CleanFlight on Linux using micro-controller for PWM/LCD

Finally have micro-controller sending PWM from a HK-T6A to Linux version of CleanFlight. 

Created a new protocol (protocolMsg), allowing to route messages between software components over UDP or within the app via message bus.

For example, CleanFlight has a PWM parallel RX IO software block and PWM RX driver block.  The PWM RX driver block was replaced with a PWM client protocolMsg handler, the micro-controller is a PWM server protocolMsg handler.  The messages are routed via UDP over USB.  

The goal is to update CleanFlight  CLI and msp to handle message processing from multiple transports, i.e. serial, UDP, TCP. 


CleanFlight Configurator is proper showing receiver status. USB/UDP is running at 28Kb/s for RPM messages.

To Do list:

• validate new Linux serial driver
• a simple libevent shim (same as UDP/TCP shim)
• used for GPS
•  validate new lcdStrip Shim
• allows routing LCD strip messages (LED RGB) to be routed to micro-controller.
• write dshot  shim
• allow sending sdhot motor messages to micro-controller

OrangePI Zero overlay and uboot/armbian

The Armbian has good documentation about how to use device overlays for their uboot scripts.  I wanted to dive a little deeper and understand some of the basics and still use buildroot.

Boot process of uboot, there is a file on the FAT partition boot.src, this is a binary file generated by mkimage.  The source file is in board/orangepi/orangepi-zero boot.sh:
setenv fdt_high ffffffff
setenv load_addr "0x44000000"
echo "Boot script loaded from ${devtype}"

setenv bootargs console=ttyS0,115200 earlyprintk root=/dev/mmcblk0p2 rootwait

fatload mmc 0 $kernel_addr_r zImage
fatload mmc 0 $fdt_addr_r sun8i-h2-plus-orangepi-zero.dtb
fdt addr ${fdt_addr_r}
fdt resize 65536
setenv processorName sun8i-h3-
setenv overlays i2c0  spi-spidev
for overlay_file in ${overlays}; do
    echo " looking for  ${processorName}${overlay_file}.dtbo"
    if fatload mmc 0 ${load_addr} ${processorName}${overlay_file}.dtbo; then
        echo "Applying user provided DT overlay ${overlay_file}.dtbo"
        fdt apply ${load_addr} || setenv overlay_error "true"
    fi
done


bootz $kernel_addr_r - $fdt_addr_r

So, what happens is the device tree overlays are merged in uboot with the dtb file loaded for the board using fdt command line tools.  This allows to change the board dtb without always recompiling it.

Post any questions you have.   So, spi and i2c are now using device tree overlays.

Here is the overlay for SPI:
/dts-v1/;
/plugin/;

/ {
    compatible = "allwinner,sun8i-h3";

    fragment@0 {
        target-path = "/aliases";
        __overlay__ {
            spi0 = "/soc/spi@01c68000";
            spi1 = "/soc/spi@01c69000";
        };
    };

    fragment@1 {
        target = <&spi0>;
        __overlay__ {

            #address-cells = <1>;
            #size-cells = <0>;
            status = "disabled";

            spidev@1 {
                compatible = "linux,spidev";
                reg = <0>;
                spi-max-frequency = <1000000>;
            };
        };
    };

    fragment@2 {
        target = <&spi1>;
        __overlay__ {

            #address-cells = <1>;
            #size-cells = <0>;
            status = "okay";

            spidev@2 {
                compatible = "linux,spidev";
                reg = <0>;
                spi-max-frequency = <25000000>;
            };
        };
    };
};


so, this completes the task of using overlays.  Next DHCP and zeroconfig.

Also, have a modification to EEM USB handler, to improve performance.  Also, thinking of adding a way to use several packets at once based on the size of the packet.

Merge latest changes from Cleanflight

Updated the repo with the latest CleanFlight changes, the merge compiled and ran on x86.  I'm in the middle of updating OrangePI buildroot to use some patches from Armbian and start using device tree overlays.  This makes it easier to add I2C etc without changing the orangePI zero DTS. More on that in the next couple of posts.  i.e basic device overlays using uboot via fdt commands.

Cleanflight configurator is working, the imu processing was not enabled, so, now the graphic is now updating.  

Also move the process to a core, using cpu isolation   (via taskset, etc) to reduce jitter.

Also, working on using dnsmasq as a dhcp server with mdev, this allows I/O board to get a IP address and try zeroconfig/DNS-Based Service Discovery.  The goal is to use names, instead of hard coded IP addresses, so when moving to WiFi networking there will not be a conflict.   Confusing?  I'm too ;)

Updated the I/O EEM packet handling to increase the performance, takes about 23us per 64 byte USB packet.  The goal is to have a 512 UDP packet take less then a 1ms. 

Looked at the GPS handling, right now, GPS parsing is handled on the I/O card, tho goal being to reduce processing in the main loop.  Still looking at the code and how RTH and stabilization process the GPS information.

Tasks Order:

• Update Kernel/u-boot to use overlays
• 4.13 kernel is booting along with u-boot (Now)
• updating boot.src to process device tree overlays  (in progress)
• Dynamic IP addressing 
• Test DHCP server first
• Add zeroconfig with MDNS on lwIP 
•  
•  Retest EEM USB Ethernet
• Add LED API 
• Cleanflight uses two LED APIs will use the basic one, i.e 3 LEDs and test the speed of UDP updating LEDs from Linux
• Add PWM reciever API
• The I/O is processing PWM from receiver, will forward this info onto Cleanflight  validate the configurator is working.

Cleanflight repo updated with OrangePI Zero Flight Controller

Updated https://github.com/tcmichals/cleanflight repo with current code for the OrangePI flight controller.

• Replaced the cleanflight scheduler with libevent
• all timers are based on libevent timers
• in process of adding events for:
• serial port
• sockets
• remote transmitters
• Instead of using hard timers for the above events, timer is more of a timeout/guard.  The event, will be posted to wake up the task instead of a timer/poll event.  
• For logging this will be moved into another thread using a message   So, sdcard write will not delay main processing loop, 

 Adding exti (external gpio interrupts).  Creating separate read file descriptors for each IO then push each one to libevent.  This way, when IO triggers libevent will dispatch the proper handler.   See if 4Khz is possible.

using top:
143   134 root     S     5804   1%   0% ./obj/main/cleanflight_LINUX.elf

Also added a command allowing to get basic info:

Task list             rate/hz  max/us  avg/us maxload avgload     total/ms
00 - (         SYSTEM)     10       2       1    0.5%    0.5%         0
01 - (            PID)     50     151     125    1.2%    1.1%        96
   - (           GYRO)    101
02 - (          ACCEL)    101     140     120    1.9%    1.7%        95
03 - (       ATTITUDE)    101       5       2    0.5%    0.5%         1
04 - (             RX)     50      18      10    0.5%    0.5%         4
05 - (         SERIAL)    101      12       2    0.6%    0.5%         1
06 - (       DISPATCH)
07 - (BATTERY_VOLTAGE)
08 - (BATTERY_CURRENT)     50       1       0    0.5%    0.0%         0
09 - ( BATTERY_ALERTS)      4       4       1    0.5%    0.5%         0
10 - (            GPS)
11 - (        COMPASS)
12 - (           BARO)
13 - (       ALTITUDE)
14 - (      TELEMETRY)    101       1       1    0.5%    0.5%         0
15 - (        RCSPLIT)
RX Check Function                   0       0                         0
Total (excluding SERIAL)                         6.1%    5.3%
LinuxFlight>

 I have to look into why some of the numbers a little high for PID and ACCEL.  Also add memory locking and add high FIFO scheduling.  But, looking good.

CleanFlight update to Linux

Making slow progress, on updating CleanFlight to access SPI, I2C and serial port on Linux.

Progress:

• Updating bus_spi_linux.c to support SPI.
• Still updating access functions based on previous work using LinuxFlight
• Update scheduler to use libevent
• Some of the issues with the current scheduler:
• Does not use events, but polling for status for event block
• Uses it's own timer framework
• Created a new scheduler based on libevent
• Using libevent timer framework
• Adding a way for HW events and SW events to trigger a fc_task to wake.
• Need some way to schedule based on priority, but will put this off, due progessing loop will not pause very long to progress a event, execept for I2C and SPI events.  Will change this next round to have another HW scheduling thread.
• Need to branch LinuxFlight and push the new round of changes to to github.  With the support of SITL in cleanflight, minimal changes are needed. 

IO Board:

• UDP via USB is not the fastest, about 500us for a 512 byte packet.  The main issue is full speed USB uses 64 bytes per interrupt.  So, it takes 8 USB packets to transfer one UDP 512 packet.  Once micro-controllers move to high speed USB, then USB packet is 512 bytes. 
• Able to decode PWM signals
• logging framework is done.

General C/C++ lambda's and calling C code before main

I've been working on a few small projects, EtherCAT and QuadCopter IO board with USB EEM using STM32 micro-controllers. During this projects found couple interesting C++/C concepts:

• Using Lambda functions for C callbacks
• Simplifies static C++ functions to allowing C to call a C++ functions.
• using GCC init to invoke C functions from C++ initialization.
• One issue in creating projects is how to control or startup threads, init code etc in the proper order without #include every function.

Lambda Functions for C callbacks

lwIP has an asynchronous interface allowing the programmer not to create a thread to wait on socket events, all of the events occur within lwIP worker thread.  Using the same thread alleviates locking

err_t            tcp_connect (struct tcp_pcb *pcb, const ip_addr_t *ipaddr,

                              u16_t port, tcp_connected_fn connected);

The above function starts the TCP connection with the IP address in the function.  So, function requires a C callback function.  

 err = tcp_connect(m_conn, &ipaddr, PORT, [](void *arg, struct tcp_pcb *tpcb, err_t err) -> err_t

                                         {

                                              class *pToClass = reinterpret_cast<class *>(args));

                                              return pToClass->connect(tpcb, err);

                                         }

From the code above, a lambda is used without any capture arguments, this allows the lambda to be callable from C.  So, now the connect function can also be a protected function.  So, no static function needs to be declared. 

Using C++ constructor to invoke a C function before main

Here is a simple function:

static void _startUpUdpServices(void) __attribute__((constructor (102)));

void _startUpUdpServices(void) 

{

    regStartUpFunction(StartUpPriority::_6, "udpServices", (startUpFunctor_t)startUpUDPServices);

}

The __attribute__((constructor (102))); Puts the function in the .init section and orders it to be at 102.  So _startUpUdpServices function will be called before main.  This makes it easy to register a system without calling the function.  nothing is worse then spegitti code to debug.  Now, it is possible to order using higher numbers.  I took a little more time and create anoterh class to register function and order them.

If you have any questions let me know...

Update ( dRonin communication to secondary board via UDP (USB EEM))

EEM USB is now working, there are some performance tweaks still needed, but, it is working.

Next start on a simple protocol to setup pins and pwm modes, ie, oneshot, or pwm out.  The goal is to use micropython and json for pin maping, and have a work task handle the I/O.  

Next week, will checkout TCP performance using iperf and Lwip netperf app.

Pushing forward.  

dRonin communication to secondary board via UDP (USB EEM)

There is a similar project, called FlyingPI, but instead of using Raspberry PI, this project is using the OrangePI (Zero)/(Zero plus2) 

The interface between the micro-controller and flight controller will be USB (Full-Speed).  (Soon there will be STM32 high speed USB  boards out, but as of now, it is USB Full Speed).  The protocol will be EEM-USB (Ethernet-Emulation-Model), it is simple, just two bulk end-points.  At this time I have a basic STM class driver setup and some what talking to Linux.   An Ethernet interface is created and DHCP messages are streaming from the Linux. So, at-least the USB descriptors are working etc.

Why use USB?  Why use Ethernet? The goal is to make the flight controller interface generic, i.e UDP This way different boards or platforms can use standard sockets instead of SPI, Serial, etc. 

The main issue at this time is RAM space the board only has 138K of RAM, so, buffer space will be limited, but, should work.

Moving forward...

orangePI Zero dRonin

I've been making slow progress on getting dRonin running on the orangePI Zero; work, and other spring outside activities.

Status:

Sparkfun MPU9250 and MS5611 is connected to the zero and dRonin is loading on the board.  just debugging general start up issues and now getting to testing sensors.  Need to configure starting dRonin with command line options for SPI and I2C.  Once this is done, add the microcontroller board for motor control, pwm input, etc.  Making progress.

General update

It has been several months since the last post; several projects starting up at work, and several projects around the house keeping me busy.

Here is the plan over the next few weeks:

1. New OrangePI Zero buildroot tutorial 
1. Basic configuration 
2. Patches (from Armbian) for WiFi 
3. How to configure kernel for RT_PREEMPT 
1. Best way to configure OS for real-time multicore response
2. Updates on BetaFlight conversion to Linux (never ending)
1. Issues in porting
1.  creating a HAL (Hardware abstraction Library)
1. SPI 
2. Serial
3. I2C
4. GPIO
5. Interrupts
6. Timers
2. Goal is to be able to run BetaFlight 
1. Linux
2. Micro-controller 
1. Must have I2C flash or SDCard
2. Use FreeRTOS
1.  Zephyr RTOS is interesting more of a Linux based RTOS (ITS NOT LINUX)
3. Also doing a for arducopter 
3. Complete Udoo NEO M4 port OpenAMP
1. Update MQX to use OpenAMP
1. Did a port a while back need update this

Yes this is a large list, but there are 25 hours in a day ! ;) yes a management theorem.

If there is any questions please ask.. always looking for idea's for postings