Since the 21st century, manned space industry is developing faster and faster all over the world; spacecraft with various uses continue to emerge. In recent years, the scale, complexity, and importance of embedded software systems in the aerospace industry have all increased rapidly. The real-time of the embedded operating system in the aerospace field are getting more and more important. Therefore, the embedded systems mounted on these products require real-time processing in terms of multi-tasks, preemptive scheduling, communication and synchronization between tasks, and communication between tasks and interrupts. Therefore, real-time performance is one of the important performances of the space embedded operating system. Embedded systems need to respond to different statuses in a timely manner during execution of the task, the execution result must be completed within the given time [1-2].

The embedded operating system in the aerospace field not only requires high real-time performance, but also needs good support for high-speed interfaces such as 10GbE. In the presence of loads from other organizations and agencies, they can also ensure the safety of their own data. The Linux system has been increasingly used in aerospace due to features such as open source, stability, portability, and reduce the total cost of a project. In May 2013, the United Space Alliance (USA) replaced the system with Linux due to ISS infection. In January 2014, JAXA of Japan’s Aerospace Exploration and Development Agency transferred the satellite monitoring function of the Sun Observation Satellite “Sunrise” (SOLAR-B), a US-Japan cooperative, and GEOTAIL, a magneto-antimony tail observation satellite, to the Linux platform. When Linux was originally designed, it was not the real-time nature of the system whose primary concern was the fair scheduling, the maximization of throughput, and the sharing of resources, so it is essentially a time-sharing operating system [3]. The real-time issue of Linux is mainly reflected in:

(1) Kernel is not completely preemptable

Though preemption has been introduced since version 2.6, there are still a lot areas in the kernel where preemption is prohibited.

(2) Roughness of clock particles [4]

Kernel clock has been raised to 1000Hz since version 2.6, but high clock precision will lead to system throughput degradation.

(3) Interruption problem [5]

In order to protect the interrupt context, the kernel will shut down the system temporarily after receiving an interrupt.

(4) Irrational task scheduling [6]

Standard Linux is not a real time system. The kernel does not protect the priority rollover.

Therefore, it is an inevitable trend to optimize the real-time performance of Linux to meet the industry demand. Under the general trend that Linux gradually becomes the main embedded operating system, it is an inevitable trend to optimize the real-time performance of Linux to meet the industry demand.

In recent years, many research institutions and universities have used numerous ideas and technical solutions to make great contributions to the improvement of Linux’s real-time performance, enabling them to be applied in various complex environments. In order to improve the real-time performance of the Linux operating system and meet the needs of real-time applications, the Linux kernel need to be optimized in real time. The real-time systems are classified as hard or soft real-time systems based on the system’s timing constraints [7].

At present, there are mainly two kinds of real-time optimized method for the Linux kernel [8]: dual-core and adding modifications to Linux kernel.

2.1.1. Dual-Core

This method is implemented by adding a hard real-time microkernel at the very bottom of the Linux operating system. It is the realization of two-kernel [9]. The real-time kernel is responsible for managing the hardware and scheduling hard real-time tasks. The Linux kernel itself runs as the lowest-priority task in the system. The Linux kernel is dispatched only when there are no real-time tasks to run. Since the real-time core is very small and does not increase the overall system load, all of these provide powerful guarantees for developing real-time demanding real-time software.

Therefore, the application program of the Linux system itself can run without change. The scheduling of hard real-time tasks can also show good performance, and the scheduling time is also predictable. This optimized method provides a communication method for data exchange between the real-time kernel and the Linux kernel, i.e. shared memory and FIFO device interfaces. The real-time Linux kernel guarantees accurate real-time performance and requires all non-real-time tasks on the Linux kernel to be able to support preemptive scheduling. The disadvantage of this method is that there is no memory protection between the real-time task and the Linux kernel, so any failure of real-time tasks may cause the crash of the entire system, while real-time tasks cannot access to standard Linux API. Dual Kernel RTOS tends to have a long integration time for new hardware as the internal design of differentiating between real-time and non-real-time Interrupt Requests (IRQ) [10]. Typical examples are RTAI and Xenomai.

2.1.2. Adding Modifications to Linux Kernel

This method is to modify the Linux kernel to make it real-time and run as a real-time operating system directly [11]. Real-time patch increases the preemption point of the Linux kernel and solves the priority flip problem through priority inheritance. The kernel was modified based on the existing Linux operating system support for real-time software development. This method partially modifies the Linux source code and makes it become a true real-time operating system.

This method is also in line with the open source nature of Linux itself. Linux based on Preempt-RT and Monta Vista-Linux are emblems of this method. However, the performance of real-time Linux based on this technology is slightly inferior to that of real-time Linux based on dual kernel structure. The test results also illustrate this view.

By investigating the current relevant patches and considering the current selection of hardware and processor in the space station stage and the high requirements for stability and safety in the aerospace field, we have selected the following several patches as the main alternatives for real-time optimization of Linux in the space station phase. The following detailed analysis is performed.

2.2.1. RTAI

RTAI (Real-Time Application Interface) is a GPL-compliant open source project initiated by the Department of Aerospace Engineering of Polytechnic University of Milan in Italy [12]. RTAI is a hard real-time extension of the Linux kernel, using the ultra-kernel design method, through the Adaptive Domain Environment for Operating System(ADEOS) ideas to achieve [13]. It is a hard real-time extension of the Linux kernel, and all of its features are seamlessly accessible through the GNU/Linux environment. It uses virtual interrupt technology to run Linux as a real-time task with the lowest priority. Real-time tasks and non-real-time tasks in the system run on RTAI and the original Linux respectively. At present, RTAI/Linux system based on ADEOS is mainly used. It consists of hardware, ADEOS, Linux kernel and RTAI real-time kernel.

The implementation of RTAI mainly includes two key technologies: interrupt virtualization and fine-grained clock. At present, RTAI has been updated to version 5.0.1 and supports multiple architectures. It is one of the effective solutions to improve the real-time performance of Linux.

2.2.2. Xenomai

Xenomai is a free software project based on real-time operating system framework organized in 2001. It completely complies with the GNU/Linux free software protocol. The purpose is to create APIs that can be used for various real-time operating systems based on the Linux platform. Xenomai makes up for the lack of real-time performance of the Linux kernel through auxiliary kernel technology. Xenomai real-time kernel provides rich functions for the development of strong real-time applications, including real-time thread scheduling and management, user space real-time task support, interrupt services, thread synchronization services, real-time object registration services and dynamic memory applications.

The difference between Xenomai and RTAI is their goals. The goal of RTAI is to increase the real-time performance of the operating system as much as possible. It is acceptable to sacrifice other features, such as certain portability and compatibility. The goal of Xenomai is designed to improve the portability and compatibility of system real-time task programs. To some extent, it is acceptable to weaken the requirements of real-time operating systems.

Xenomai has maintained frequent project updates. In recent years, Xenomai’s official website has released an average of more than five versions each year. The activity of the project ensures the continuous addition of new features, bug fixes, and support for the latest Linux kernel version.

2.2.3. RT Patch

The RT patch (Real-time Preemption Patch) is updated and maintained by Ingo Molnar and Thomas Gleixner, and its stability testing is carried out by the Open Source Automation Development Lab (OSADL). The core idea of RT patch is to increase the preempt ability of the kernel itself. Based on the original low-latency patch and preemption patch, RT patch adds new features such as interrupted threading, high-precision clocking as well as priority inheritance, and changes the Linux kernel to a completely preemptive kernel, which enables the kernel to obtain hard real-time capabilities.

The prominent advantage of the Linux RT patch compared to the dual-core RTOS is that it follows the POSIX standard. The differences between using RT patches and using non-real-time systems in applications and drivers are small. Therefore, it is easier to make related development on the platform by using this patch than the dual-core patches.

At the manned space station stage, it may contain a large number of scientific experiments units in the spacecraft, such as microgravity fluid experiments, materials processing experiments, ecological experiments, etc. Each experimental unit also contains many tasks, and each task has a very complex communication relationship. Take the microgravity fluid experiments for example; there is a gateway control unit, storage control unit, and data management unit, etc.

Figure 1 shows the hierarchical structure of the based on embedded Linux in microgravity physics experiments. It can be seen from the figure that each task of the user layer interacts through the data flow and the control flow, and calls the kernel of Linux through API.

For multi-task and high concurrency in space station mission, we take microgravity fluid experiments as an example. From Figure 1, we can see that there are seven tasks at least run on a dual-core CPU in the user software; task switching and preemption may occur at any time. Because of Linux is a time-sharing operating system, there is no guarantee that every task switching and preemption can be completed within a predictable time. In manned space missions, any handling or response exceed the deadline may cause loss of critical data and will have a serious impact on the entire control system, which affects the work of the entire mission and may even endanger the astronaut’s life safety or cause an incalculable loss.

In Linux kernel, task switching is mainly achieved through the context_switch() function which is composed of prapare_task_switch(), switch_mm(), switch_to(), barrier(), and finish_task_switch(). In prapare_task_switch(), Linux runs specific code about architecture to prepare task switch. In switch_mm(), Linux completes the mapping of the process in virtual address space. Data in CPU register and kernel stack are switched. The complete process switch is finalized in function switch_to(). The role of the barrier() is to ensure that the execution order of switch_to() and finish_task_switch() does not change.

In addition, task switching is the measurement point of basic efficiency in any multitasking system. It is synchronous and non-preemptive. Real-time control software implements a time slice rotation algorithm based on equal priority tasks. Factors that affect task switching include the structure of the host CPU, instruction set, and CPU characteristics. Essentially, all multiprocessing tasks can dynamically assign priorities during execution, so preemption time is also an important measure of real-time performance. We chose the task response time and the preemption time between processes in this paper.

Priority inversion is when a high-priority task accesses a shared resource through a semaphore mechanism. The semaphore has been occupied by a low-priority task, thus causing high-priority tasks to be blocked by many tasks with lower priority; real-time performance is difficult to guarantee.

For example, there are three tasks with priorities A, B, and C. Priority A > B > C. Tasks A and B are waiting for an event to occur while Task C is running. Task C enters critical area and starts using a share resource S.

Task A prepares to runs immediately because its priority is higher than Task C’s. When Task A wants to use the shared resource S, Task A is suspended because Task C is using it; Task C starts running again. Task A will not run until Task C releases shared resource S. Since Task B has a higher priority than Task C, if Task B is ready, Task B starts to run. In this case, the priority rollover problem occurs since task B preceded task A, as shown in Figure 2.

Due to priority reversal, the system will start confusing tasks and logic, task scheduling will cause time uncertainty, and the system may even crash. For example, the United States Mars Rover encountered a failure caused by a priority reversal problem, causing the system to restart in 1997.

As mentioned earlier, most payload systems are multi-tasking during the manned space phase. Priority-based multi-task scheduling will inevitably result in priority reversal in Linux std.

The priority inversion solution includes:

(1) Setting the upper limit of the priority.

Set a certain high priority for the critical zone. For processes entering the critical zone, if the priority is lower than the set online, it will get this priority.

(2) Priority inheritance.

If a higher-priority process is waiting for a shared resource that a lower-priority process is consuming, the low-priority process is raised to the same priority as the high-priority process. The lower-process’ priority can be restored to the original priority state after the shared resource is released.

(3) Interruption is prohibited so the critical area is protected by disabling interrupts.

There are only two types of priority for the system in this method: 1) preempted, which is the priority that the process has in the normal running process, and 2) disabled by the interrupt when running in the critical section.

Three patches take the method of priority inheritance protocol to solve this problem. We will also test the priority reversal to see if the priority reversal has been solved.

In the space station phase, for the different characteristics of the data processing unit and the controlling unit, a suitable patch needs to be selected as the optimization. Based on the analysis above, switching time, preemption time, and priority reversal were tested in this paper to evaluate the effect of the three patches on the Linux std. The design and test results of experiments are as follows.

We choose a stable version of Linux- Linux-4.1.18, which is supported and maintained by the Linux community. The experiments were based on the x86 platform (dual-core 2.7GHz and 1GB RAM) and the Xilinx ZYNQ7000 series ZC7020 processor (dual-core 866MHz and 1GB SDRAM). ZYNQ-7000 is also software programmable, hardware programmable, and IO programmable. In the ZYNQ7000 platform series, FPGA and ARM are integrated in one chip. The communication between the two uses AXI_HP, AXI_GP, and AXI_ACP interfaces to communicate with a bandwidth of up to gigabit.

Get Linux-4.1.18 from www.kernel.org. RTAI cannot port to ZYNQ family directly so we patch the Xenomai, and Preempt-RT patch into the standard kernel, reconfigure the kernel to remove some modules conflict to the Xenomai and add the real-time modules we need. Compile the kernel to create the kernel image and device tree -- uImage and devicetree.dts, the compilation steps are as follows:

$\cdot$ make ARCH = arm CROSS_COMPILE = arm-xilinx-linux-gnueabi- distclean

$\cdot$ make ARCH = arm CROSS_COMPILE = arm-xilinx-linux-gnueabi- zynq_myd_rt_defconfig

$\cdot$ make ARCH = arm CROSS_COMPILE = arm-xilinx-linux-gnueabi- uImage

$\cdot$ make ARCH = arm CROSS_COMPILE = arm-xilinx-linux-gnueabi- dtbs

Copy uImage, devicetree.dts, and other files such as BOOT.bin, rootfs.tar, uramdisk.image.gz and 7z020.bit to the SD card. Then, we write the Linux with patch into the flash of the ZYNQ7020.

The task response time refers to tat when the multitasking kernel decides to run another task. It saves the current state of the running task (that is, the entire contents of the CPU register) into the task’s own stack area, and then picks the next one to run. The current state of the task is reloaded into the CPU’s registers from the task’s stack and the next task is started. The design test scenario is shown in Figure 3.

The context switch time is calculated by using the read and write of pipes to complete the switch between tasks and obtaining the time stamp.

In this paper, 500 tests were performed on each of the three types of patches; that is, 1000 context switch occurred so the test results must be divided by 1000. The final results are shown in Table 1 and 2.

We can see that real-time performance in Linux with preempt-RT is better than Linux std., while real-time performance in Linux with Xenomai is the best of all. Xenomai-based Linux is a dual-core architecture. Standard Linux runs as a separate process and test run in the Linux std., so the average of switch time is about the same as the standard Linux average.

Task preemption time refers to the required time that a high-priority process immediately suspends a running process, frees up resources, and high-priority tasks preempt control to obtain. The design test scenario is shown in Figure 4. Five tasks numbered 1-5 are created; the priorities are from low to high. The task of the lowest priority gets the resources and run. High-priority tasks are ready to preempt low-priority tasks and measure the preemption time between tasks by obtaining time stamps.

Test of task preemption time runs 1000 times, as shown in Figure 3. The test results must be divided by 1000. The final results are shown in Table 3 and 4.

We can see that real-time performance in Linux with Xenomai is the best. Xenomai-based Linux is a dual-core architecture. Standard Linux kernel runs as a separate process and test run in the Linux std., so the average of preemption time is about the same as the standard Linux average and Linux with preempt-RT.

Priority inversion means that a task waits to be blocked from releasing resources than its lower-priority task. If there is a medium-priority ready task, blocking will further deteriorate. It affects the completion of real-time tasks. To test whether the three types of patches solve the problem of priority inversion, the design program captures the priority inversion problem during the operation of the operating system. The test process is shown in Figure 5. The results show that all three types of patches solve the problem of priority inversion.

Real-time problem of Linux is a real issue that the aerospace field must face. The paper researches and analyzes the Linux real-time optimization scheme, and compares it from multiple perspectives. Current optimization programs and patches have their own characteristics. The test results show that different optimization schemes can meet the special needs of different aerospace fields. For units that emphasize data processing, IPC defined in the standard kernel is heavily dependent. In real-time Linux based on Xenomai or RTAI, standard Linux is only run as a process in the entire system. Frequent kernel switching may affect the performance of system so embedded Linux system with Preempt-RT patch is more suitable. For units that emphasize controlling, the stability of real-time performance is the most important, which means an embedded Linux system with Xenomai or RTAI patch is more suitable.