Inner Perimeters
For the most part, the module writer does not need to specify an inner perimeter, as the STREAMS framework automatically creates it for the module. What needs to be specified is the type of perimeter, and the concurrency of the perimeter.
Inner perimeters come in two types:
| D_PERQ | Enables synchronous entry to be different between the read queue and the write queue. Therefore, if a synchronous putnext is occurring on the read queue, a synchronous or asynchronous putnext can occur on the write queue (or other (a)synchronous access on the write queue). |
| D_QPAIR | Protects both the read queue and the write queue, so synchronous access to one queue will prevent synchronous or asynchronous access to the other queue. |
Another perimeter, D_PERMOD is slightly different, and is discussed in "PERMOD Perimeter".
An inner perimeter becomes exclusive (writer) whenever an inner synchronous entry point is encountered. By default all the entry points are considered to be synchronous until enabled as "shared" entry points. As previously stated, synchronous entry points remain exclusive until the thread returns to the caller of the synchronous entry point. If the synchronous function calls putnext, the perimeter remains exclusive across the putnext, up till the synchronous function can return to its caller, and subsequent entries into the perimeter will be deferred.
Inner perimeters can specify additional concurrency on the STREAMS entry points for open/close, put/putnext, service, and callbacks as shown in Table 12-4.
Outer Perimeters
The module writer can also specify an outer perimeter. An outer perimeter is the linked list of all inner perimeters for all queues associated with the specified module. Entering the outer perimeter is equivalent to entering each of the inner perimeters. As this can also be an expensive operation, the outer perimeter is only entered synchronously, and upon successful completion of a qwriter(PERIM_OUTER) makes the outer perimeter exclusive. This also has the effect of making each of the inner perimeters exclusive.
Use of outer perimeters is reserved for module data that has an effect on all queue instances of the module, such as module state that might allow messages to pass between other instances of the module, information that allows a driver to configure shared hardware, or at open/close time when information is needed for all open instances for a module.
Outer perimeters, at this time, have only one concurrency modifier. This is D_MTOCEXCL, and instructs the framework to enter the outer perimeter on each open and close of queues for the module.
PERMOD Perimeter
The PERMOD perimeter is a hybrid of the inner and outer perimeter. It is implemented primarily for modules that might have a large number of queue instances, and cannot afford the latency for entering the outer perimeter. Because it is a hybrid, PERMOD perimeters cannot have an outer perimeter, and modules that have D_MTPERMOD and D_MTOUTPERIM defined will fail at open. As PERMOD perimeters are implemented as inner perimeters, they share all the concurrency states as the inner perimeter, see Table 12-2.
Hot Perimeters
All STREAMS modules and drivers in the Solaris operating environment must be D_MTSAFE, and must account for multithreading. Specifying an inner and/or outer perimeter will handle concurrency issues that the module writer may encounter while developing the module or driver. Experienced STREAMS programmers might decide that the perimeter should not have any synchronous entry points, and should run fully hot. To define a fully-hot perimeter, the module writer need only specify the D_MTSAFE flag without an inner perimeter type (D_MTPERQ, D_MTQPAIR, D_MTPERMOD) and without an outer perimeter (D_MTOUTPERIM).
Caution - Hardening Information. All STREAMS entry points run concurrently, and
in a multiprocessor environment, there can be a put procedure
running simultaneously with a service procedure or even
a close procedure. So the writer must take precautions
against kernel panics by making sure that other concurrent threads will not
reference data the current thread is trying to change or remove.
Defining Perimeter Types
To configure a module with perimeter types and concurrency types, use the f_flag field in fmodsw(9s) with D_MTSAFE or'd with the appropriate perimeter type flags. See Table 12-1.
The easiest method is to initially implement your module and configure it to be per-module single threaded, and increase the level of concurrency as needed. "Sample Multithreaded Device Driver Using a Per Module Inner Perimeter" provides a complete example of using a per-module perimeter, and "Sample Multithreaded Module With Outer Perimeter" provides a complete example with a higher level of concurrency.
To configure a driver with perimeter and concurrency types, put MT_SAFE and the appropriate perimeter flags in the cb_flag field of the cb_ops structure for the driver.
Choosing a Perimeter Type
Table 12-1 summarizes examples of when to use an inner perimeter, or both an inner and outer perimeter for a STREAMS module.
Table 12-1 Choosing a Perimeter Type
Module Description | Perimeter Type |
|---|---|
A module where the put procedure reads as well as modifies module global data. Use a per-module inner perimeter on the single-threaded module. | inner |
A module where all the module private data associated with a queue (or a read/write pair of queues) can be configured to be single-threaded. Use an inner perimeter for each corresponding queue (or queue pair) . | inner |
A module where most of the module private data is associated with a queue (or a queue pair); but has some module global data that is mostly read. Use an inner perimeter for the queue (or queue pair) plus an outer perimeter for global data. Use qwriter to protect the sections where it modifies the module's global data (see "qwriter"). | inner and outer |
A module that requires higher concurrency for certain message types while not requiring message ordering. Use an inner perimeter for shared access to the put procedures. Use an outer perimeter for put procedures that require exclusive access. Use qwriter when messages are handled in the put procedures that require exclusive access (see "qwriter"). | inner and outer |
A hardware driver can use an appropriate set of inner and outer perimeters to restrict the concurrency in the open, close, put, and service procedures. With explicit synchronization primitives (mutex, condition variables, readers/writer, semaphore), these drivers restrict the concurrency when accessing the hardware registers in interrupt handlers. When designing such drivers, you need to be aware of the issues listed in "MT SAFE Modules Using Explicit Locks". | inner and outer |
Several flags specify the inner and outer perimeters (see Table 12-2 and Table 12-3). These flags fall into three categories:
Define the presence and scope of the inner perimeter
Define the presence of the outer perimeter (which can have only one scope)
Modify the default concurrency for the different entry points
You configure the inner perimeter by choosing one of the mutually exclusive flags shown in Table 12-2.
Table 12-2 Inner Perimeter Flags
Flag | Description |
|---|---|
D_MTPERMOD | The module has an inner perimeter that encloses all the module's queues |
D_MTAPAIR | The module has an inner perimeter around each read/write pair of queues |
D_MTPERQ | The module has an inner perimeter around each queue |
none | The module has no inner perimeter |
Configure the outer perimeter using the flag shown in Table 12-3.
Table 12-3 Outer Perimeter Flag
Flag | Description |
|---|---|
D_MTOUTEPERIM | In addition to an inner perimeter (or none), the module has an outer perimeter that encloses all the module's queues. This can be combined with all the inner perimeter options except D_MTPERMOD. |
By default all synchronous entry points enter the inner perimeter exclusively and enter the outer perimeter shared. To modify this behavior use the flags shown in Table 12-4.
Table 12-4 Modify Exclusive/Shared Access Flags
Flag | Description |
|---|---|
D_MTOCEXCL | open/close entry points are synchronous. |
D_MTPUTSHARED | put/putnext entry points are asynchronous |
_D_MTOCSHARED | open/close entry points are asynchronous (experimental) |
_D_MTCBSHARED | callbacks (via qtimeout, qbufcall) are asynchronous (experimental) |
_D_MTSVCSHARED | service procedures are asynchronous (experimental) |
Caution - Hardening Information. Concurrency flags designated with a preceding underbar
"_" are experimental, and their behavior might change in the future and should
not be relied upon.
MT SAFE Modules and Drivers
A module or a driver can be either MT SAFE or MT UNSAFE. A module or driver is MT SAFE when its data values are correct regardless of the order that multiple threads access and modify the data. For MT SAFE mode, use MT STREAMS perimeters to restrict the concurrency in a module or driver to:
Per-module single threading
Per queue-pair single threading
Per queue single threading
Per queue or per queue-pair single threading of the put and service procedures with per module single threading of the open and close routines
Unrestricted concurrency in the put and service procedures with the ability to restrict the concurrency when handling messages that modify data
Completely unrestricted concurrency
MT SAFE Module
To configure a module as being MT SAFE, use the f_flag field in fmodsw(9S).
The easiest method is to initially implement your module and configure it to be per-module single threaded, and increase the level of concurrency as needed. "Sample Multithreaded Device Driver Using a Per Module Inner Perimeter" provides a complete example of using a per-module perimeter, and "Sample Multithreaded Module With Outer Perimeter" provides a complete example with a higher level of concurrency.
MT SAFE modules can use different MT STREAMS perimeters to restrict the concurrency in the module to a concurrency that is natural given the data structures that the module contains, thereby removing the need for module private locks (see "MT STREAMS Perimeters" for information on perimeters). A module that requires unrestricted concurrency can be configured to have no perimeters. Such modules have to use explicit locking primitives to protect their data structures. While such modules can exploit the maximum level of concurrency allowed by the underlying hardware platform, they are more complex to develop and support. See "MT SAFE Modules Using Explicit Locks".
Independent of the perimeters, there will be at most one thread allowed within any given queue's service procedure.
Your MT SAFE modules should use perimeters and avoid using module private locks (mutex, condition variables, readers/writer, or semaphore). Should you opt to use module private locks, you need to read "MT SAFE Modules Using Explicit Locks" along with this section.
Note - MT UNSAFE mode for STREAMS modules was temporarily supported as an aid in porting SVR4 modules; however, MT UNSAFE is not supported after SVR4. Beginning with the release of the Solaris 7 operating environment, no MT UNSAFE module or driver has been supported.
Note - Upper and lower multiplexors share the same perimeter type and concurrency level.