Real-time Programming and Administration
This chapter describes writing and porting real-time applications to run under SunOS. This chapter is written for programmers experienced in writing real-time applications and administrators familiar with real-time processing and the Solaris system.
This chapter discusses the following topics:
Scheduling needs of real-time applications, covered in "Scheduling".
Basic Rules of Real-time Applications
Real-time response is guaranteed when certain conditions are met. This section identifies these conditions and some of the more significant design errors that can cause problems or disable a system.
Most of the potential problems described here can degrade the response time of the system. One of the potential problems can freeze a workstation. Other, more subtle, mistakes are priority inversion and system overload.
A Solaris real-time process has the following characteristics:
Runs in the RT scheduling class, as described in "Scheduling"
Locks down all the memory in its process address space, as described in "Memory Locking"
Is from a statically linked program or from a program in which all dynamic binding is completed early, as described in "Shared Libraries "
Real-time operations are described in this chapter in terms of single-threaded processes, but the description can also apply to multithreaded processes. For detailed information about multithreaded processes, see the Multithreaded Programming Guide. To guarantee real-time scheduling of a thread, the thread must be created as a bound thread, and the thread's LWP must be run in the RT scheduling class. The locking of memory and early dynamic binding is effective for all threads in a process.
When a process is the highest priority real-time process, it acquires the processor within the guaranteed dispatch latency period of becoming runnable (see "Dispatch Latency"). The process continues to run for as long as it remains the highest priority runnable process.
A real-time process can lose control of the processor or can be unable to gain control of the processor because of other events on the system. These events include external events, such as interrupts, resource starvation, waiting on external events such as synchronous I/O, and pre-emption by a higher priority process.
Real-time scheduling generally does not apply to system initialization and termination services such as open(2) and close(2).
Factors that degrading Response Time
The problems described in this section all increase the response time of the system to varying extents. The degradation can be serious enough to cause an application to miss a critical deadline.
Real-time processing can also impair the operation of aspects of other applications that are active on a system running a real-time application. Because real-time processes have higher priority, time-sharing processes can be prevented from running for significant amounts of time. This can cause interactive activities, such as displays and keyboard response time, to slow noticeably.
Synchronous I/O Calls
System response under SunOS provides no bounds to the timing of I/O events. This means that synchronous I/O calls should never be included in any program segment whose execution is time critical. Even program segments that permit very large time bounds must not perform synchronous I/O. Mass storage I/O is such a case, where causing a read or write operation hangs the system while the operation takes place.
A common application mistake is to perform I/O to get error message text from disk. This should be done from an independent process or thread that is not running in real time.
Interrupt Servicing
Interrupt priorities are independent of process priorities. Prioritizing processes does not carry through to prioritizing the services of hardware interrupts that result from the actions of the processes. This means that interrupt processing for a device controlled by a higher priority real-time process is not necessarily given priority over interrupt processing for another device controlled by a lower-priority timeshare process.
Shared Libraries
Time-sharing processes can save significant amounts of memory by using dynamically linked, shared libraries. This type of linking is implemented through a form of file mapping. Dynamically linked library routines cause implicit reads.
Real-time programs can use shared libraries and avoid dynamic binding by setting the environment variable LD_BIND_NOW to a non-NULL value when the program is invoked. This forces all dynamic linking to be bound before the program begins execution. See the Linker and Libraries Guide for more information.
Priority Inversion
A time-sharing process can block a real-time process by acquiring a resource that is required by a real-time process. Priority inversion occurs when a higher priority process is blocked by a lower priority process. The term blocking describes a situation in which a process must wait for one or more processes to relinquish control of resources. Real-time processes may miss their deadlines if this blocking is prolonged.
Consider the case depicted in the following figure, where a high-priority process requires a shared resource. A lower priority process holds the resource and is pre-empted by an intermediate priority process, blocking the high-priority process. This condition can persist for an arbitrarily long time, because the amount of time that the high-priority process must wait for the resource depends not only on the duration of the critical section being executed by the lower-priority process, but on the duration of the intermediate process blocks. Any number of intermediate processes can be involved.
Figure 9-1 Unbounded Priority Inversion
This issue and the methods of dealing with it are described in "Mutual Exclusion Lock Attributes" in Multithreaded Programming Guide.
Sticky Locks
A page is permanently locked into memory when its lock count reaches 65535 (0xFFFF). The value 0xFFFF is defined by the implementation and might change in future releases. Pages locked this way cannot be unlocked.
Runaway Real-time Processes
Runaway real-time processes can cause the system to halt or can slow the system response so much that the system appears to halt.
Note - If you have a runaway process on a SPARC system, press Stop-A. You might have to do this more than one time. If this doesn't work, or on non-SPARC systems, turn the power off, wait a moment, then turn it back on.
When a high priority real-time process does not relinquish control of the CPU, you cannot regain control of the system until the infinite loop is forced to terminate. Such a runaway process does not respond to Control-C. Attempts to use a shell set at a higher priority than that of a runaway process do not work.
Asynchronous I/O Behavior
Asynchronous I/O operations do not always execute in the sequence in which they are queued to the kernel. There is also no guarantee that asynchronous operations return to the caller in the sequence in which they were done.
If a single buffer is specified for a rapid sequence of calls to aioread(3AIO), the state of the buffer between the time that the first call is made and the time that the last result is signaled to the caller is uncertain.
An individual aio_result_t structure can be used for only one asynchronous read or write at a time.
Real-time Files
SunOS provides no facilities to ensure that files are allocated as physically contiguous.
For regular files, the read(2) and write(2) operations are always buffered. An application can use mmap(2) and msync(3C) to effect direct I/O transfers between secondary storage and process memory.
Scheduling
Real-time scheduling constraints are necessary to manage data acquisition or process control hardware. The real-time environment requires that a process be able to react to external events in a bounded amount of time. Such constraints can exceed the capabilities of a kernel designed to provide a fair distribution of the processing resources to a set of time-sharing processes.
This section describes the SunOS real-time scheduler, its priority queue, and how to use system calls and utilities that control scheduling.
Dispatch Latency
The most significant element in scheduling behavior for real-time applications is the provision of a real-time scheduling class. The standard time-sharing scheduling class is not suitable for real-time applications because this scheduling class treats every process equally and has a limited notion of priority. Real-time applications require a scheduling class in which process priorities are taken as absolute and are changed only by explicit application operations.
The term dispatch latency describes the amount of time a system takes to respond to a request for a process to begin operation. With a scheduler written specifically to honor application priorities, real-time applications can be developed with a bounded dispatch latency.
The following figure illustrates the amount of time an application takes to respond to a request from an external event.
Figure 9-2 Application Response Time.
The overall application response time consists of the interrupt response time, the dispatch latency, and the time the application takes to respond.
The interrupt response time for an application includes both the interrupt latency of the system and the device driver's own interrupt processing time. The interrupt latency is determined by the longest interval that the system must run with interrupts disabled. This time is minimized in SunOS using synchronization primitives that do not commonly require a raised processor interrupt level.
During interrupt processing, the driver's interrupt routine wakes the high-priority process and returns when finished. The system detects that a process with higher priority than the interrupted process is now ready to dispatch and dispatches the process. The time to switch context from a lower-priority process to a higher-priority process is included in the dispatch latency time.
Figure 9-3 illustrates the internal dispatch latency/application response time of a system, defined in terms of the amount of time a system takes to respond to an internal event. The dispatch latency of an internal event represents the amount of time required for one process to wake up another higher priority process, and for the system to dispatch the higher priority process.
The application response time is the amount of time a driver takes to wake up a higher-priority process, have a low-priority process release resources, reschedule the higher-priority task, calculate the response, and dispatch the task.
Interrupts can arrive and be processed during the dispatch latency interval. This processing increases the application response time, but is not attributed to the dispatch latency measurement, and so is not bounded by the dispatch latency guarantee.
Figure 9-3 Internal Dispatch Latency
With the new scheduling techniques provided with real-time SunOS, the system dispatch latency time is within specified bounds. As you can see in the table below, dispatch latency improves with a bounded number of processes.
Table 9-1 Real-time System Dispatch Latency
| Workstation | Bounded Number of Processes | Arbitrary Number of Processes |
|---|---|---|
| SPARCstation 2 | <0.5 milliseconds in a system with fewer than 16 active processes | 1.0 milliseconds |
| SPARCstation 5 | <0.3 millisecond | 0.3 millisecond |
| Ultra 1-167 | <0.15 millisecond | <0.15 millisecond |