My embedded RTOS notes

Started 8Jan2019, updated 9Jan2019

This page is in group Technology and is a blog note where I will try to have a fresh view (as of 2019) of what might be going on with concurrency and Real-Time Operating Systems (RTOS) in the small embedded systems world. Here I would think mostly of what might run on platforms like ARM-based Arduino boards. I work mostly with XMOS boards and program in the XC language, so I would tend to view this through those glasses. So I would think of a big loop in main as only a single-threaded start in the boy’s (or girl’s) room, and would want real support for processes and then the “thread-safer” would be the better. Remember that a small embedded system landed on a comet in 2014 (see below), and many of them are indeed safety-critical.

Intro

This note started its life as cut and paste from an earlier note: My WiFi101 library port. There in particular, concurrency in the Arduino world was discussed. I guess that also the My SafeRTOS notesIEC 61508 and (safe?) concurrency and IEC 61508 and programming paradigms may relate to this note.

Arduino concurrency

Looping as fast as possible leaves no room for concurrency

SimpleWebServerWiFi_Teig.ino contains loop(). It just loops. I made a counter (unsigned long) and printed out its value every one mill counts. It took about 56 seconds to count to 5 mill, so it’s about 11.2 μs to loop around just counting up that value. It runs as fast as it can.

This is bad architecture. Lucky for the demos that there are no background or foreground tasks that have own state. There is no task / process / thread here. It’s as single threaded as possibly possible.

Disclaimer: Of course the ARM processors have a lot of hardware blocks that would live their own lives. And they would communicate with the main loop of running objects through interrupts. This is concurrency that all processors would need and have, and they have had it since 1954? They certainly make even the simplest of processors do more than one thing quite seamlessly. But it’s not concurrency at the occam, Ada, go, XC etc. level that I want to have here. The quite new (2017) Apache mynewt OS is also interesting in this context. I do not want my aquarium code and the communication to be in some one outer loop. One could say that each process, task or thread consists of local loops all over the place in the software. Then there is a scheduler that controls when each loop is run. Basically this is the basic theme of all of my Technology blogs and papers. But there’s more to it:

Arduino Scheduler concurrency is low-level yield-based

Arduino supplies an “experimental” Scheduler,  Scheduler (see https://www.arduino.cc/en/Reference/Scheduler) that’s “cooperative” (they don’t cooperate on data, just on passing of control). It’s based on starting an additional loop with Scheduler.startLoop(new_loop). Then new_loop must run a Scheduler.yield so that the original loop also will run. This is the lowest level of concurrency with no support for communication between them and no structured way to avoid busy poll.

Have a look at their blinking LED example (here). No no-skew timer. No communication between the loops. No communication into or out of the loops. Besides, all cycles are burned. And the cognitive understanding av what a task is probably isn’t easy to grasp in the light of how to next solve the missing points.

Later, working with the port of the RadioHead library My aquarium’s data radioed through the shelf, I discovered lots of YIELD; in the code. Like in file RHGenericDriver.cpp:

// Blocks until a valid message is received
void RHGenericDriver::waitAvailable()
{
    while (!available())
	YIELD;
}

Now it can do other things instead of blocking. (Aside: I have discussed blocking in Not so blocking after all.) In file RadioHead.h YIELD is defined as yield(); for Arduino is 1.55 or more. I then searched my installed libraries and found the Scheduler files. In Scheduler.h:

// Copyright (C) 2012 The Android Open Source Project
class SchedulerClass {
public:
	SchedulerClass();
	static void startLoop(SchedulerTask task, uint32_t stackSize = 1024);
	static void start(SchedulerTask task, uint32_t stackSize = 1024);
	static void start(SchedulerParametricTask task, void *data, uint32_t stackSize = 1024);

	static void yield() { ::yield(); };
};

We see that it’s based on what we saw above, new Loops may be started as tasks. Looking inside Scheduler.cpp then coopTaskStart, coopDoYield are assembly code, for ARM Arduinos (I think).

Adding YIELD; or yield(); to my .ino top level file for testing the RadioHead library did compile. I use ARM Arduinos. It compiles, yes, but yield(); probably is empty code. I edited in Scheduler.cpp and made sure it would not compile. It did. I then did Sketch | Include Library | Scheduler, and #include <Scheduler.h> arrived in the code. Then it didn’t compile before I had fixed the code again. So, in the RadioHead library blocking seems to block since I cannot find and start or startLoop in the code that would give the code any place to yield to.

I have queried about the sanity of the above inferring on the radiohead-arduino group, see YIELD; in RadioHead code.

Later I also found Ivan Seidel’s ArduinoThread (https://github.com/ivanseidel/ArduinoThread) and downloaded it. It’s a wrapper around callbacks for «blinking of LEDs» and it does not use C++11’s std::thread class. I have found no Arduino stuff that uses those native threads. Also, there is no communication between them.

I will not do my whole story here. Almost all of my technical blog notes deal with concurrency. Just search for yield, generator, CSP, concurrency or blocking in the search window above, or visit the main Technology blog note list here and see if you find a thread to pull.

That being said, to me this Scheduler.yield may be a starting point for building a CSP-type Scheduler on top. This may be interesting in the future.

Zephyr operating system

Update Jan2019: I have been pointed to the Zephyr operating system. See Wikipedia, here – where it says that it was originally developed as Rocket by Wind River Systems, for IoT, then it became a project for the Linux Foundation. It is based on Wind River’s Microkernel Profile for VxWorks.

Aside: The operating system came from Eric Verhult’s company Eonic Systems that had developed the RTOS Virtuoso at the time they were aquired by Wind River. Wind River then developed Virtuoso into Rocket. I have met Verhulst and his people on several conferences. The first time I met him was at a conference that we arranged in Trondheim around 1992, where he told about what was later to become Virtuoso. Virtuoso had been inspired by occam on transputers, a “running subset of CSP“. Even now Zephyr threads with negative priority are “cooperative threads” and non-preemptive. Some of these people are now active in Altreonic and kurt.mobi. Along the way they did a formal analysis of their operating system (search for Verhulst here). And: “The most recent example of the technology’s success is the successful Philae Landing on Comet Churyumov–Gerasimenko and the accompanying Rosetta Orbite”, (they claim it, of all places here). This was in 2014. I have talked with Verhulst after this and he said he’d almost forgotten that the code was on its way to that comet. After all, it had been launched almost 11 years earlier

Zephyr is supported by corporations like Intel, Linaro, NXP and Nordic Semiconductor. But in this context I guess that the most important factor is that (from Wikipedia) “A BSD licensed fork occurs in the Arduino 101 software source package from Intel”. Also see Zephyr’s home page at www.zephyrproject.org. Observe that Wind River Systems since 2009 has been a wholly owned subsidiary of Intel Corporation.

A comment from Eric Verhulst

On 8Jan2019 I had this comment from Erik Verhulst (at Altreonic and kurt.mobi). I was allowed to copy/paste it here:

“Zephyr is indeed the old Virtuoso RTOS (v.3.1), after WRS refactored the API à la posix. Along the way they claim they developed the original code, but if you look in the source code, you know better.

The reality is that the RTOS at that time was become “bloated”. Some engineers with a C++ background had been adding too many nice-to-haves.

We developed OpenComRTOS from 2004/2005 from scratch using formal methods (see the book) but with a similar philosophy and even cleaner API/ semantics. It s a completely new development with a graphical modelling front-end and many meta-models based front-end and code generators. We use XML for this.

The formal development resulted in a code size reduction of a factor 5 to 10 (depending on the target) and the official introduction of “hubs” (a bit like active CSP channels).

A few years ago, we decided to rebrand OpenComRTOS as VirtuosoNext. This went together with the support for fine grain partitioning (each task can run protected in its own memory space) and real-time fault tolerance (e.g. we can recover from exceptions in microseconds).

Now today, being tired of hearing “too advanced for us” (the real reason is in-house job protection, not understanding the real benefits (of small code and trustworthiness, productivity), or not wanting to change what free source and Linux claim to offer) we stopped active prospecting and use it now for our own benefit in the KURT vehicle concept. Fault tolerant by design but that also is difficult to explain to people who think autonomous driving software can be upgraded over the air.

The real killer: the law of Moore (which made software writers lazy) and education (high in the cloud and assuming that a PC is like an embedded device).

I see some signs that some people become aware that their current approach is not really scalable, but should I care?” (Eric Verhulst, 8Jan2019)