Multithreaded STREAMS
This chapter describes how to write a multithreaded STREAMS driver or module. It covers the necessary conversion topics so that new and existing STREAMS modules and drivers run in the multithreaded kernel. It describes STREAMS-specific multithreading issues and techniques. For general information, see Writing Device Drivers.
This chapter contains the following information:
Multithreaded (MT) STREAMS Overview
The SunOS 5 operating system is fully multithreaded, which means that it can make effective use of the available parallelism of a symmetric shared-memory multiprocessor computer. All kernel subsystems are multithreaded: scheduler, virtual memory, file systems, block/character/STREAMS I/O, networking protocols, and device drivers.
MT STREAMS requires you to use some different concepts and terminology. These concepts apply not only to STREAMS drivers, but to all device drivers in the Solaris operating environment. For a more complete description of these terms, see Writing Device Drivers. Additionally, see Chapter 1, Overview of STREAMS of this guide for definitions and Chapter 8, STREAMS Kernel-Level Mechanisms for elements of MT drivers.
Some of the multithreaded terms and ideas are.
| Thread | Sequence of instructions executed within context of a process. |
| Lock | Mechanism to restrict access to data structures. |
| Single Threaded | Restricting access to a single thread. |
| Multithreaded | Allowing two or more threads access to a data element. |
| Multiprocessing | Two or more CPUs concurrently executing the Operating System. |
| Concurrency | Simultaneous execution. |
| Preemption | Suspending execution for the next thread to run. |
| Monitor | Portion of code that is single-threaded. |
| Mutual Exclusion | Exclusive access to a data element by a single thread at one time. |
| Condition Variables | Kernel event synchronization primitives. |
| Counting Semaphores | Memory-based synchronization mechanism. |
| Readers/Writer Locks | Data lock allowing one writer or many readers at one time. |
| Callback | On occurrence of a specific event, call a module function. |
| Synchronous Access | Only one thread is allowed in the perimeter. Upon return from a call the action is complete; when the thread is done, the job is done. |
| Asynchronous Access | Multiple threads are allowed in the perimeter. Upon return form a call there is no guarantee that the job is complete. |
| Perimeter Claim | A thread has synchronous access in the perimeter. The claim prevents subsequent synchronous access until the claim is released. |
| Exclusive Access | Calling a synchronous entry point in the perimeter. |
| Writer | A thread that has exclusive access to a perimeter. |
MT STREAMS Framework
The STREAMS framework consists of the stream head, documented STREAMS data structures (such as queue_t, mblk_t) and STREAMS utility routines including STREAMS facilities documented in the Device Driver Interface (DDI). The STREAMS framework enables multiple kernel threads to concurrently enter and execute code defined by each module, including the open, close, put, and service procedures of each queue within the system.
The first goal of the SunOS 5 system is to preserve the interface and flavor of STREAMS and to shield module code as much as possible from the impact of migrating to the multithreaded kernel. Most of the locking is hidden from the programmer and performed by the STREAMS kernel framework. As long as module code uses the standard, documented programmatic interfaces to shared kernel data structures (such as queue_t, mblk_t, and dblk_t), and adheres to the DDI/DKI, the user does not have to explicitly lock these framework data structures.
The second goal is to make writing MT SAFE modules simple. One of the ways that the framework accomplishes this is by using the MT STREAMS perimeter mechanisms for controlling and restricting concurrent access to a STREAMS module. STREAMS perimeters allow the module writer to select the level of concurrency that a module can tolerate.
STREAMS Framework Integrity
The STREAMS framework ensures the integrity of the STREAMS data structures, such as queue_t, mblk_t, and dblk_t as long as the module conforms to the DDI/DKI, and does not directly access global operating system data structures or facilities not described in the DDI/DKI.
The q_next fields of the queue_t structure are not modified by the framework while a thread is actively executing within a synchronous entry point. However the q_next field might change while a thread is executing within an asynchronous entry point.
The q_ptr field is considered private to the module and the framework will not manipulate its value. When making a module MT Safe, the integrity of the module-private data structures must be ensured by the module itself. This integrity can be guaranteed by creating private locks, or by the control of concurrency within the module by the use of STREAMS perimeters. Knowing what the framework supports is critical in deciding what the module writer must provide.
Note - Hardening Information. As in previous Solaris operating environment releases, a module must not call another module's put or service procedures directly. The DDI/DKI routines putnext(9F), put(9F), and other routines in Section 9F must be used to pass a message to another queue. Calling another module's routines directly circumvents the design of the MT STREAMS framework and can yield unknown results.
Note - Hardening Information. Once a message is passed using a putq, put, putnext, as well as the perimeter function qwriter, it cannot be accessed again because the use of this message has been given to the new routine. If a reference needs to be retained by the module, it should copy it by using copyb, copymsg, dupb, or dupmsg.
Message Ordering
The STREAMS framework guarantees the ordering of messages along a stream if all the modules in the stream preserve message ordering internally. This ordering guarantee only applies to messages that are sent along the same stream and produced by the same source.
The STREAMS framework does not guarantee that a message has been seen by the next put procedure when the call to putnext(9F) or qreply(9F) returns.
MT STREAMS Perimeters
Solaris STREAMS uses a facility known as perimeters for handling thread concurrency in STREAMS modules. Perimeters allow the module writer to select conditions that will result in exclusive access to a queue, a pair of queues, or all the queues in a module. This makes multithreading issues easier to work with in STREAMS.
Perimeters work somewhat like reader/writer locks, where there can be many readers, but only one writer. A synchronous access to the perimeter is similar to holding the writer lock, in that only one thread can be in the perimeter at a time. Synchronous entry will hold a perimeter exclusively until the thread eventually unwinds out of the perimeter (usually when it returns from a put/putnext or a qwriter call that initially invoked the synchronous behavior). While a thread has synchronous access to the perimeter, all other access (synchronous or asynchronous) will be deferred.
An asynchronous access is similar to the reader lock, where many threads can be in the perimeter at a time, including the possible recursive entry of a thread previously entering the perimeter. The asynchronous "claim" is not released until the thread winds out of the put/putnext. Because asynchronous access is similar to the reader lock, any synchronous access will be deferred or blocked until all asynchronous claims are released.
Caution - Hardening Information. The
perimeter data is private to the STREAMS framework, and should not be modified
by the module itself.
STREAMS enables the definition of two synchronous perimeters. One is an inner perimeter, and is used to define synchronous entry points on a queue or a queue pair (the read and write queue's for a specific module instance). It also identifies an outer perimeter, which is made up of all the inner perimeters for all the queues for a specific module.
There is also a special inner perimeter, PERMOD, that is similar to the outer perimeter, but does not have the overhead of the outer perimeter. PERMOD identifies a single synchronous entry point for all queues for this module. Because PERMOD is like a hybrid of an inner perimeter and outer perimeter, the PERMOD perimeter cannot have an outer perimeter.