Tock Processes
This document explains how application code works in Tock. This is not a guide to writing applications, but rather documentation of the overall design of how applications are implemented in Tock.
Overview of Processes in Tock
Processes in Tock run application code meant to accomplish some type of task for the end user. Processes run in user mode. Unlike kernel code, which runs in supervisor mode and handles device drivers, chip-specific details, as well as general operating system tasks, application code running in processes is independent of the details of the underlying hardware (except the instruction set architecture). Unlike many existing embedded operating systems, in Tock processes are not compiled with the kernel. Instead they are entirely separate code that interact with the kernel and each other through system calls.
Since processes are not a part of the kernel, application code running in a process may be written in any language that can be compiled into code capable of running on a microcontroller. Tock supports running multiple processes concurrently. Co-operatively multiprogramming is the default, but processes may also be time sliced. Processes may share data with each other via Inter-Process Communication (IPC) through system calls.
Processes run code in unprivileged mode (e.g., user mode on Cortex-M or RV32I microcontrollers). The Tock kernel uses hardware memory protection (an MPU on CortexM and a PMP on RV32I) to restrict which addresses application code running in a process can access. A process makes system calls to access hardware peripherals or modify what memory is accessible to it.
Tock supports dynamically loading and unloading independently compiled applications. In this setting, applications do not know at compile time what address they will be installed at and loaded from. To be dynamically loadable, application code must be compiled as position independent code (PIC). This allows them to be run from any address they happen to be loaded into.
In some cases, applications may know their location at compile-time. This happens, for example, in cases where the kernel and applications are combined into a single cryptographically signed binary that is accepted by a secure bootloader. In these cases, compiling an application with explicit addresses works.
Tock supports running multiple processes at the same time. The maximum number of processes supported by the kernel is typically a compile-time constant in the range of 2-4, but is limited only by the available RAM and Flash resources of the chip. Tock scheduling generally assumes that it is a small number (e.g., uses O(n) scheduling algorithms).
System Calls
System calls are how processes and the kernel share data and interact. These could include commands to drivers, subscriptions to callbacks, granting of memory to the kernel so it can store data related to the application, communication with other application code, and many others. In practice, system calls are made through library code and the application need not deal with them directly.
For example, consider the following system call that sets a GPIO pin high:
int gpio_set(GPIO_Pin_t pin) {
return command(GPIO_DRIVER_NUM, 2, pin);
}
The command system call itself is implemented as the ARM assembly instruction
svc (service call):
int __attribute__((naked))
command(uint32_t driver, uint32_t command, int data) {
asm volatile("svc 2\nbx lr" ::: "memory", "r0");
}
A detailed description of Tock's system call API and ABI can be found in TRD104. The system call documentation describes how the are implemented in the kernel.
Upcalls and Termination
The Tock kernel is completely non-blocking, and it pushes this asynchronous behavior to userspace code. This means that system calls (with one exception) do not block. Instead, they always return very quickly. Long-running operations (e.g., sending data over a bus, sampling a sensor) signal their completion to userspace through upcalls. An upcall is a function call the kernel makes on userspace code.
Yield system calls are the exception to this non-blocking rule. The yield-wait system call blocks until the kernel invokes an upcall on the process. The kernel only invokes upcalls when a process issues the yield system call. The kernel does not invoke upcalls at arbitrary points in the program.
For example, consider the case of when a process wants to sleep for 100 milliseconds. The timer library might break this into three operations:
- It registers an upcall for the timer system call driver with a Subscribe system call.
- It tells the timer system call driver to issue an upcall in 100 milliseconds by invoking a Command system call.
- It calls the yield-wait system call. This causes the process to block until the timer upcall executes. The kernel pushes a stack frame onto the process to execute the upcall; this function call returns to the instruction after yield was invoked.
When a process registers an upcall with a call to a Subscribe system call, it
may pass a pointer userdata. The kernel does not access or use this data: it
simply passes it back on each invocation of the upcall. This allows a process to
register the same function as multiple upcalls, and distinguish them by the data
passed in the argument.
It is important to note that upcalls are not executed until a process calls
yield. The kernel will enqueue upcalls as events occur within the kernel,
but the application will not handle them until it yields.
Applications which are "finished" should call an Exit system call. There are two variants of Exit: exit-terminate and exit-restart. They differ in what they signal to the kernel: does the application wish to stop running, or be rebooted?
Inter-Process Communication
Inter-process communication (IPC) allows for separate processes to communicate directly through shared buffers. IPC in Tock is implemented with a service-client model. Each process can support one service. The service is identified by the name of the application running in the process, which is included in the Tock Binary Format Header for the application. A process can communicate with multiple services and will get a unique handle for each discovered service. Clients and services communicate through shared buffers. Each client can share some of its own application memory with the service and then notify the service to instruct it to parse the shared buffer.
Services
Services are named by the package name included in the app's TBF header. To
register a service, an app can call ipc_register_svc() to setup a callback.
This callback will be called whenever a client calls notify on that service.
Clients
Clients must first discover services they wish to use with the function
ipc_discover(). They can then share a buffer with the service by calling
ipc_share(). To instruct the service to do something with the buffer, the
client can call ipc_notify_svc(). If the app wants to get notifications from
the service, it must call ipc_register_client_cb() to receive events from when
the service when the service calls ipc_notify_client().
See ipc.h in libtock-c for more information on these functions.
Application Entry Point
An application specifies the first function the kernel should call by setting
the variable init_fn_offset in its TBF header. This function should have the
following signature:
void _start(void* text_start, void* mem_start, void* memory_len, void* app_heap_break);
Process RAM and Flash Memory
The actual process binary and TBF header are stored in nonvolatile flash. This flash region is fixed when the application is installed.
When a process is loaded by the kernel, the process is assigned a fixed, contiguous region of memory in RAM. This is the entire amount of memory the process can use during its entire lifetime. This region includes the typical memory regions for a process (i.e. stack, data, and heap), but also includes the kernel's grant region for the process and the process control block.
Process RAM is memory space divided between all running apps. The figure below shows the memory space of a process.
The Tock kernel tries to impart no requirements on how a process uses its own accessible memory. As such, a process starts in a very minimal environment, with an initial stack sufficient to support a syscall, but not much more. Application startup routines should first move their program break to accommodate their desired layout, and then setup local stack and heap tracking in accordance with their runtime.
Stack and Heap
Applications can specify their working memory requirements by setting the
minimum_ram_size variable in their TBF headers. Note that the Tock kernel
treats this as a minimum, depending on the underlying platform, the amount of
memory may be larger than requested, but will never be smaller.
If there is insufficient memory to load your application, the kernel will fail during loading and print a message.
If an application exceeds its allotted memory during runtime, the application will crash (see the Debugging section for an example).
Isolation
The kernel limits processes to only accessing their own memory regions by using hardware memory protection units. On Cortex-M platforms this is the MPU and on RV32I platforms this is the PMP (or ePMP).
Before doing a context switch to a process the kernel configures the memory protection unit for that process. Only the memory regions assigned to the process are set as accessible.
Flash Isolation
Processes cannot access arbitrary addresses in flash, including bootloader and kernel code. They are also prohibited from reading or writing the nonvolatile regions of other processes.
Processes do have access to their own memory in flash. Certain regions, including their Tock Binary Format (TBF) header and a protected region after the header, are read-only, as the kernel must be able to ensure the integrity of the header. In particular, the kernel needs to know the total size of the app to find the next app in flash. The kernel may also wish to store nonvolatile information about the app (e.g. how many times it has entered a failure state) that the app should not be able to alter.
The remainder of the app, and in particular the actual code of the app, is considered to be owned by the app. The app can read the flash to execute its own code. If the MCU uses flash for its nonvolatile memory the app can not likely directly modify its own flash region, as flash typically requires some hardware peripheral interaction to erase or write flash. In this case, the app would require kernel support to modify its flash region.
RAM Isolation
For the process's RAM region, the kernel maintains a brk pointer and gives the
process full access to only its memory region below that brk pointer. Processes
can use the Memop syscall to increase the brk pointer. Memop syscalls can
also be used by the process to inform the kernel of where it has placed its
stack and heap, but these are entirely used for debugging. The kernel does not
need to know how the process has organized its memory for normal operation.
All kernel-owned data on behalf of a process (i.e. grant and PCB) is stored at the top (i.e. highest addresses) of the process's memory region. Processes are never given any access to this memory, even though it is within the process's allocated memory region.
Processes can choose to explicitly share portions of their RAM with the kernel
through the use of Allow syscalls. This gives capsules read/write access to
the process's memory for use with a specific capsule operation.
Debugging
If an application crashes, Tock provides a very detailed stack dump. By default, when an application crashes Tock prints a crash dump over the platform's default console interface. When your application crashes, we recommend looking at this output very carefully: often we have spent hours trying to track down a bug which in retrospect was quite obviously indicated in the dump, if we had just looked at the right fields.
Note that because an application is relocated when it is loaded, the binaries
and debugging .lst files generated when the app was originally compiled will not
match the actual executing application on the board. To generate matching files
(and in particular a matching .lst file), you can use the make debug target
app directory to create an appropriate .lst file that matches how the
application was actually executed. See the end of the debug print out for an
example command invocation.
---| Fault Status |---
Data Access Violation: true
Forced Hard Fault: true
Faulting Memory Address: 0x00000000
Fault Status Register (CFSR): 0x00000082
Hard Fault Status Register (HFSR): 0x40000000
---| App Status |---
App: crash_dummy - [Fault]
Events Queued: 0 Syscall Count: 0 Dropped Callback Count: 0
Restart Count: 0
Last Syscall: None
╔═══════════╤══════════════════════════════════════════╗
║ Address │ Region Name Used | Allocated (bytes) ║
╚0x20006000═╪══════════════════════════════════════════╝
│ ▼ Grant 948 | 948
0x20005C4C ┼───────────────────────────────────────────
│ Unused
0x200049F0 ┼───────────────────────────────────────────
│ ▲ Heap 0 | 4700 S
0x200049F0 ┼─────────────────────────────────────────── R
│ Data 496 | 496 A
0x20004800 ┼─────────────────────────────────────────── M
│ ▼ Stack 72 | 2048
0x200047B8 ┼───────────────────────────────────────────
│ Unused
0x20004000 ┴───────────────────────────────────────────
.....
0x00030400 ┬─────────────────────────────────────────── F
│ App Flash 976 L
0x00030030 ┼─────────────────────────────────────────── A
│ Protected 48 S
0x00030000 ┴─────────────────────────────────────────── H
R0 : 0x00000000 R6 : 0x20004894
R1 : 0x00000001 R7 : 0x20004000
R2 : 0x00000000 R8 : 0x00000000
R3 : 0x00000000 R10: 0x00000000
R4 : 0x00000000 R11: 0x00000000
R5 : 0x20004800 R12: 0x12E36C82
R9 : 0x20004800 (Static Base Register)
SP : 0x200047B8 (Process Stack Pointer)
LR : 0x000301B7
PC : 0x000300AA
YPC : 0x000301B6
APSR: N 0 Z 1 C 1 V 0 Q 0
GE 0 0 0 0
EPSR: ICI.IT 0x00
ThumbBit true
Cortex-M MPU
Region 0: base: 0x20004000, length: 8192 bytes; ReadWrite (0x3)
Region 1: base: 0x30000, length: 1024 bytes; ReadOnly (0x6)
Region 2: Unused
Region 3: Unused
Region 4: Unused
Region 5: Unused
Region 6: Unused
Region 7: Unused
To debug, run `make debug RAM_START=0x20004000 FLASH_INIT=0x30059`
in the app's folder and open the .lst file.
Applications
For example applications, see the language specific userland repos:
- libtock-c: C and C++ apps.
- libtock-rs: Rust apps.