MT SAFE Driver

To configure a driver as being MT SAFE, initialize the cb_ops(9S) and dev_ops(9S) data structures. This code must be in the header section of your module. For more information, see Example 12-1, and dev_ops(9S).

The driver is configured to be MT SAFE by setting the cb_flag to D_MP. It also specifies any MT STREAMS perimeters by setting flags in the cb_flag field. (See mt-streams(9F).)

Routines Used Inside a Perimeter

This section describes the routines and data fields used after you enter a perimeter.

`qprocson`/`qprocsoff`

The routines qprocson(9F) and qprocsoff(9F) respectively enable and disable the put and service procedures of the queue pair. Before calling qprocson(9F) and after calling qprocsoff(9F), the module's put and service procedures are disabled; messages flow around the module as if it were not present in the stream.

Call qprocson(9F) in the first open of a module, but only after allocating and initializing any module resources on which the put and service procedures depend. Call the qprocsoff routine in the close routine of the module before deallocating any resources on which the put and service procedures depend.

Note - To avoid deadlocks, modules must not hold private locks across the calls to qprocson(9F) or qprocsoff(9F).

`qtimeout`/`qunbufcall`

The timeout(9F) and bufcall(9F) callbacks are asynchronous. For a module using MT STREAMS perimeters, the timeout(9F) and bufcall(9F) callback functions execute outside the scope of the perimeters. This makes synchronization of callbacks with the rest of the module complex.

To make timeout(9F) and bufcall(9F) functionality easier to use for modules with perimeters, there are additional interfaces that use synchronous callbacks. These routines are qtimeout(9F), quntimeout(9F), qbufcall(9F), and qunbufcall(9F). When using these routines, the callback functions are executed inside the perimeters, and hence have the same concurrency restrictions as the put and service procedures.

`qwriter`

Modules can use the qwriter(9F) function to upgrade from shared to exclusive access at a perimeter. For example, a module with an outer perimeter can use qwriter(9F) in the put procedure to upgrade to exclusive access at the outer perimeter. A module where the put procedure runs with shared access at the inner perimeter (D_MTPUTSHARED) can use qwriter(9F) in the put procedure to upgrade to exclusive access at the inner perimeter.

Returning from a qwriter call does not mean that the callback function has executed. If the framework can become exclusive in the qwriter call, it will enter the perimeter synchronously, and execute the callback. If it cannot, the callback will be deferred. It is a good idea for any caller of qwriter to immediately return to its caller as there is little that can be accomplished in this thread of execution.

Caution - Hardening Information. Do not call qwriter with another queue, as qwriter assumes that the caller has already made a claim to the perimeter that the queue is associated with (asynchronous entry), and calling another perimeter will cause problems.

Note - qwriter(9F) cannot be used in the open or close procedures. If a module needs exclusive access at the outer perimeter in the open and/or close procedures, it has to specify that the outer perimeter should always be entered exclusively for open and close (using D_MTOCEXCL).

The STREAMS framework guarantees that all deferred qwriter(9F) callbacks associated with a queue have executed before the module's close routine is called for that queue.

For an example of a driver using qwriter(9F) see Example 12-2.

`qwait`

A module that uses perimeters and must wait in its open or close procedure for a message from another STREAMS module has to wait outside the perimeters; otherwise, the message would never be allowed to enter its put and service procedures. This is accomplished by using the qwait(9F) interface. See qwriter(9F) man page for an example.

Asynchronous Callback Functions

Interrupt handlers and other asynchronous callback functions require special care by the module writer, because they can execute asynchronously to threads executing within the module open, close, put, and service procedures.

For modules using perimeters, use qtimeout(9F) and qbufcall(9F) instead of timeout(9F) and bufcall(9F). The qtimeout and qbufcall callbacks are synchronous and consequently introduce no special synchronization requirements.

Because a thread can enter the module at any time, you must ensure that the asynchronous callback function acquires the proper private locks before accessing private module data structures, and releases these locks before returning. You must cancel any outstanding registered callback routines before the data structures on which the callback routines depend are deallocated and the module closed.

For hardware device interrupts, this involves disabling the device interrupts.
Outstanding callbacks from timeout(9F) and bufcall(9F) must be canceled by calling untimeout(9F) and unbufcall(9F).

The module cannot hold certain private locks across calls to untimeout(9F) or unbufcall(9F). These locks are those that the module's timeout(9F) or bufcall(9F) callback functions acquire. See "MT SAFE Modules Using Explicit Locks".
If outstanding callbacks from esballoc(9F) are associated with a particular stream, they must be allowed to complete before the module close routine deallocates the private data structures on which they depend.

`close()` Race Conditions

Because the callback functions are by nature asynchronous, they can be executing or about to execute at the time the module close routine is called. You must cancel all outstanding callback and interrupt conditions before deallocating those data structures or returning from the close routine.

The callback functions scheduled with timeout(9F) and bufcall(9F) are guaranteed to have been canceled by the time untimeout(9F) and unbufcall(9F) return. The same is true for qtimeout(9F) and qbufcall(9F) by the time quntimeout(9F) and qunbufcall(9F) return. You must also take responsibility for other asynchronous routines, including esballoc(9F) callbacks and hardware, as well as software interrupts.

Unloading a Module that Uses `esballoc`

The STREAMS framework prevents a module or driver text from being unloaded while there are open instances of the module or driver. If a module does not cancel all callbacks in the last close routine, it should not be allowed to be unloaded.

This is an issue mainly for modules and drivers using esballoc because esballoc callbacks cannot be canceled. Thus, modules and drivers using esballoc have to be prepared to handle calls to the esballoc callback free function after the last instance of the module or driver has been closed.

Modules and drivers can maintain a semaphore count of outstanding callbacks. They can deny an unload by making the _fini(9E) routine return EBUSY if there are outstanding callbacks.

Use of the `q_next` Field

The q_next field in the queue_t structure can be referenced in open, close, put, and service procedures as well as the synchronous callback procedures (scheduled with qtimeout(9F), qbufcall(9F), and qwriter(9F)). However, the value in the q_next field should not be trusted. It is relevant to the STREAMS framework, but may not be relevant to a specific module.

All other module code, such as interrupt routines, timeout(9F) and esballoc(9F) callback routines, cannot dereference q_next. Those routines have to use the "next" version of all functions. For instance, use canputnext(9F) instead of dereferencing q_next and using canput(9F).

MT SAFE Modules Using Explicit Locks

Although the result is not reliable, you can use explicit locks either instead of perimeters or to augment the concurrency restrictions provided by the perimeters.

Caution - Explicit locks cannot be used to preserve message ordering in a module because of the risk of re-entering the module. Use MT STREAMS perimeters to preserve message ordering.

All four types of kernel synchronization primitives are available to the module writer: mutexes, readers/writer locks, semaphores, and condition variables. Because cv_wait implies a context switch, it can only be called from the module's open and close procedures, which are executed with valid process context. You must use the synchronization primitives to protect accesses and ensure the integrity of private module data structures.

Constraints When Using Locks

When adding locks in a module, observe these constraints:

Avoid holding module private locks across calls to putnext(9F). The module might be re-entered by the same thread that called putnext(9F), causing the module to try to acquire a lock that it already holds. This can cause kernel panic.
Do not hold module private locks, acquired in put or service procedures, across the calls to qprocson(9F) or qprocsoff(9F). Doing this causes deadlock, since qprocson(9F) and qprocsoff(9F) wait until all threads leave the inner perimeter.
Similarly, do not hold locks, acquired in the timeout(9F) and bufcall(9F) callback procedures, across the calls to untimeout(9F) or unbufcall(9F). Doing this causes deadlock, because untimeout(9F)and unbufcall(9F) wait until an already executing callback has completed.

The first restriction deters using module private locks to preserve message ordering. The preferred mechanism is to use MT STREAMS perimeters to preserve message ordering.

Preserving Message Ordering

Module private locks cannot be used to preserve message ordering because they cannot be held across calls to putnext(9F) and the other messages that pass routines to other modules. The alternatives for preserving message ordering are:

Use MT STREAMS perimeters.
Pass all messages through the service procedures. The service procedure can drop the locks before calling putnext(9F) or qreply(9F), without reordering messages, because the framework guarantees that at most, one thread will execute in the service procedure for a given queue.

Use perimeters to avoid the performance penalty for using service procedures.

Preparing to Port

When modifying a STREAMS driver to take advantage of the multithreaded kernel, a level of MT safety is selected according to:

The desired degree of concurrency
The natural concurrency of the underlying module
The amount of effort or complexity required

Much of the effort in conversion is simply determining the appropriate degree of data sharing and the corresponding granularity of locking (see Table 12-1). The actual time spent configuring perimeters and/or installing locks should be much smaller than the time spent in analysis.

To port your module, you must understand the data structures used within your module, as well as the accesses to those data structures. You must fully understand the relationship between all portions of the module and private data within that module, and to use the MT STREAMS perimeters (or the synchronization primitives available) to maintain the integrity of these private data structures.

You must explicitly restrict access to private module data structures as appropriate to ensure the integrity of these data structures. You must use the MT STREAMS perimeters to restrict the concurrency in the module so that the parts of the module that modify private data are single-threaded with respect to the parts of the module that read the same data. (For more information about perimeters, see "MT STREAMS Perimeters".) Besides perimeters, you can use the synchronization primitives available (mutex, condition variables, readers/writer, semaphore) to explicitly restrict access to module private data appropriate for the operations within the module on that data.

The first step in multithreading a module or driver is to analyze the module, breaking the entire module up into a list of individual operations and the private data structures referenced in each operation. Part of this first step is deciding upon a level of concurrency for the module. Ask yourself which of these operations can be multithreaded and which must be single-threaded. Try to find a level of concurrency that is "natural" for the module and matches one of the available perimeters (or, alternatively, requires the minimal number of locks) , and has a simple and straightforward implementation. Avoid additional unnecessary complexity.


12. Multithreaded STREAMS MT SAFE Modules and Drivers MT SAFE Module


	MT SAFE Driver To configure a driver as being MT SAFE, initialize the `cb_ops`(9S) and `dev_ops`(9S) data structures. This code must be in the header section of your module. For more information, see Example 12-1, and `dev_ops`(9S). The driver is configured to be MT SAFE by setting the `cb_flag` to `D_MP`. It also specifies any MT STREAMS perimeters by setting flags in the `cb_flag` field. (See `mt-streams`(9F).) Routines Used Inside a Perimeter This section describes the routines and data fields used after you enter a perimeter. `qprocson`/`qprocsoff` The routines `qprocson`(9F) and `qprocsoff(9F)` respectively enable and disable the `put` and `service` procedures of the queue pair. Before calling `qprocson`(9F) and after calling `qprocsoff(9F)`, the module's `put` and `service` procedures are disabled; messages flow around the module as if it were not present in the stream. Call `qprocson`(9F) in the first `open` of a module, but only after allocating and initializing any module resources on which the `put` and `service` procedures depend. Call the `qprocsoff` routine in the `close` routine of the module before deallocating any resources on which the `put` and `service` procedures depend. Note - To avoid deadlocks, modules must not hold private locks across the calls to `qprocson`(9F) or `qprocsoff(9F)`. `qtimeout`/`qunbufcall` The `timeout`(9F) and `bufcall`(9F) callbacks are asynchronous. For a module using MT STREAMS perimeters, the `timeout`(9F) and `bufcall`(9F) callback functions execute outside the scope of the perimeters. This makes synchronization of callbacks with the rest of the module complex. To make `timeout`(9F) and `bufcall`(9F) functionality easier to use for modules with perimeters, there are additional interfaces that use synchronous callbacks. These routines are `qtimeout`(9F), `quntimeout`(9F), `qbufcall`(9F), and `qunbufcall`(9F). When using these routines, the callback functions are executed inside the perimeters, and hence have the same concurrency restrictions as the `put` and `service` procedures. `qwriter` Modules can use the `qwriter`(9F) function to upgrade from shared to exclusive access at a perimeter. For example, a module with an outer perimeter can use `qwriter`(9F) in the `put` procedure to upgrade to exclusive access at the outer perimeter. A module where the `put` procedure runs with shared access at the inner perimeter (`D_MTPUTSHARED`) can use `qwriter`(9F) in the `put` procedure to upgrade to exclusive access at the inner perimeter. Returning from a `qwriter` call does not mean that the `callback` function has executed. If the framework can become exclusive in the `qwriter` call, it will enter the perimeter synchronously, and execute the `callback`. If it cannot, the `callback` will be deferred. It is a good idea for any caller of `qwriter` to immediately return to its caller as there is little that can be accomplished in this thread of execution. Caution - Hardening Information. Do not call `qwriter` with another queue, as `qwriter` assumes that the caller has already made a claim to the perimeter that the queue is associated with (asynchronous entry), and calling another perimeter will cause problems. Note - `qwriter`(9F) cannot be used in the `open` or `close` procedures. If a module needs exclusive access at the outer perimeter in the `open` and/or `close` procedures, it has to specify that the outer perimeter should always be entered exclusively for `open` and `close` (using `D_MTOCEXCL`). The STREAMS framework guarantees that all deferred `qwriter`(9F) callbacks associated with a queue have executed before the module's `close` routine is called for that queue. For an example of a driver using `qwriter`(9F) see Example 12-2. `qwait` A module that uses perimeters and must wait in its `open` or `close` procedure for a message from another STREAMS module has to wait outside the perimeters; otherwise, the message would never be allowed to enter its `put` and `service` procedures. This is accomplished by using the `qwait`(9F) interface. See `qwriter`(9F) man page for an example. Asynchronous Callback Functions Interrupt handlers and other asynchronous callback functions require special care by the module writer, because they can execute asynchronously to threads executing within the module `open`, `close`, `put`, and `service` procedures. For modules using perimeters, use `qtimeout`(9F) and `qbufcall`(9F) instead of `timeout`(9F) and `bufcall`(9F). The `qtimeout` and `qbufcall` callbacks are synchronous and consequently introduce no special synchronization requirements. Because a thread can enter the module at any time, you must ensure that the asynchronous callback function acquires the proper private locks before accessing private module data structures, and releases these locks before returning. You must cancel any outstanding registered callback routines before the data structures on which the callback routines depend are deallocated and the module closed. For hardware device interrupts, this involves disabling the device interrupts. Outstanding callbacks from `timeout`(9F) and `bufcall`(9F) must be canceled by calling `untimeout`(9F) and `unbufcall`(9F). The module cannot hold certain private locks across calls to `untimeout`(9F) or `unbufcall`(9F). These locks are those that the module's `timeout`(9F) or `bufcall`(9F) callback functions acquire. See "MT SAFE Modules Using Explicit Locks". If outstanding callbacks from `esballoc`(9F) are associated with a particular stream, they must be allowed to complete before the module `close` routine deallocates the private data structures on which they depend. `close()` Race Conditions Because the callback functions are by nature asynchronous, they can be executing or about to execute at the time the module `close` routine is called. You must cancel all outstanding callback and interrupt conditions before deallocating those data structures or returning from the `close` routine. The callback functions scheduled with `timeout`(9F) and `bufcall`(9F) are guaranteed to have been canceled by the time `untimeout`(9F) and `unbufcall`(9F) return. The same is true for `qtimeout`(9F) and `qbufcall`(9F) by the time `quntimeout`(9F) and `qunbufcall`(9F) return. You must also take responsibility for other asynchronous routines, including `esballoc`(9F) callbacks and hardware, as well as software interrupts. Unloading a Module that Uses `esballoc` The STREAMS framework prevents a module or driver text from being unloaded while there are `open` instances of the module or driver. If a module does not cancel all callbacks in the last `close` routine, it should not be allowed to be unloaded. This is an issue mainly for modules and drivers using `esballoc` because `esballoc` callbacks cannot be canceled. Thus, modules and drivers using `esballoc` have to be prepared to handle calls to the `esballoc` callback free function after the last instance of the module or driver has been closed. Modules and drivers can maintain a semaphore count of outstanding callbacks. They can deny an unload by making the `_fini`(9E) routine return `EBUSY` if there are outstanding callbacks. Use of the `q_next` Field The `q_next` field in the `queue_t` structure can be referenced in `open`, `close`, `put`, and `service` procedures as well as the synchronous callback procedures (scheduled with `qtimeout`(9F), `qbufcall`(9F), and `qwriter`(9F)). However, the value in the `q_next` field should not be trusted. It is relevant to the STREAMS framework, but may not be relevant to a specific module. All other module code, such as interrupt routines, `timeout`(9F) and `esballoc`(9F) callback routines, cannot dereference `q_next`. Those routines have to use the "next" version of all functions. For instance, use `canputnext`(9F) instead of dereferencing `q_next` and using `canput`(9F). MT SAFE Modules Using Explicit Locks Although the result is not reliable, you can use explicit locks either instead of perimeters or to augment the concurrency restrictions provided by the perimeters. Caution - Explicit locks cannot be used to preserve message ordering in a module because of the risk of re-entering the module. Use MT STREAMS perimeters to preserve message ordering. All four types of kernel synchronization primitives are available to the module writer: mutexes, readers/writer locks, semaphores, and condition variables. Because `cv_wait` implies a context switch, it can only be called from the module's `open` and `close` procedures, which are executed with valid process context. You must use the synchronization primitives to protect accesses and ensure the integrity of private module data structures. Constraints When Using Locks When adding locks in a module, observe these constraints: Avoid holding module private locks across calls to `putnext`(9F). The module might be re-entered by the same thread that called `putnext`(9F), causing the module to try to acquire a lock that it already holds. This can cause kernel panic. Do not hold module private locks, acquired in `put` or `service` procedures, across the calls to `qprocson`(9F) or `qprocsoff`(9F). Doing this causes deadlock, since `qprocson`(9F) and `qprocsoff`(9F) wait until all threads leave the inner perimeter. Similarly, do not hold locks, acquired in the `timeout`(9F) and `bufcall`(9F) callback procedures, across the calls to `untimeout`(9F) or `unbufcall`(9F). Doing this causes deadlock, because `untimeout`(9F)and `unbufcall`(9F) wait until an already executing callback has completed. The first restriction deters using module private locks to preserve message ordering. The preferred mechanism is to use MT STREAMS perimeters to preserve message ordering. Preserving Message Ordering Module private locks cannot be used to preserve message ordering because they cannot be held across calls to `putnext`(9F) and the other messages that pass routines to other modules. The alternatives for preserving message ordering are: Use MT STREAMS perimeters. Pass all messages through the `service` procedures. The `service` procedure can drop the locks before calling `putnext`(9F) or `qreply`(9F), without reordering messages, because the framework guarantees that at most, one thread will execute in the `service` procedure for a given queue. Use perimeters to avoid the performance penalty for using `service` procedures. Preparing to Port When modifying a STREAMS driver to take advantage of the multithreaded kernel, a level of MT safety is selected according to: The desired degree of concurrency The natural concurrency of the underlying module The amount of effort or complexity required Much of the effort in conversion is simply determining the appropriate degree of data sharing and the corresponding granularity of locking (see Table 12-1). The actual time spent configuring perimeters and/or installing locks should be much smaller than the time spent in analysis. To port your module, you must understand the data structures used within your module, as well as the accesses to those data structures. You must fully understand the relationship between all portions of the module and private data within that module, and to use the MT STREAMS perimeters (or the synchronization primitives available) to maintain the integrity of these private data structures. You must explicitly restrict access to private module data structures as appropriate to ensure the integrity of these data structures. You must use the MT STREAMS perimeters to restrict the concurrency in the module so that the parts of the module that modify private data are single-threaded with respect to the parts of the module that read the same data. (For more information about perimeters, see "MT STREAMS Perimeters".) Besides perimeters, you can use the synchronization primitives available (`mutex`, condition variables, readers/writer, semaphore) to explicitly restrict access to module private data appropriate for the operations within the module on that data. The first step in multithreading a module or driver is to analyze the module, breaking the entire module up into a list of individual operations and the private data structures referenced in each operation. Part of this first step is deciding upon a level of concurrency for the module. Ask yourself which of these operations can be multithreaded and which must be single-threaded. Try to find a level of concurrency that is "natural" for the module and matches one of the available perimeters (or, alternatively, requires the minimal number of locks) , and has a simple and straightforward implementation. Avoid additional unnecessary complexity.

MT SAFE Driver

Routines Used Inside a Perimeter

qprocson/qprocsoff

qtimeout/qunbufcall

qwriter

qwait