Chapter 8

Direct Memory Access (DMA)

Many devices can temporarily take control of the bus and perform data transfers to (and from) main memory or other devices. Because the device is doing the work without the help of the CPU, this type of data transfer is known as direct memory access (DMA). DMA transfers can be performed between two devices, between a device and memory, or between memory and memory. This chapter explains transfers between a device and memory only.

This chapter provides information on the following subjects:

DMA Model

The Solaris Device Driver Interface/Driver-Kernel Interface (DDI/DKI) provides a high-level, architecture-independent model for DMA. This allows the framework (the DMA routines) to hide such architecture-specific details as:

Setting up DMA mappings
Building scatter-gather lists
Ensuring that I/O and CPU caches are consistent

There are several abstractions that are used in the DDI/DKI to describe aspects of a DMA transaction. These include:

DMA object - Memory that is the source or destination of a DMA transfer.
DMA handle - An opaque object returned from a successful ddi_dma_alloc_handle(9F) call. The DMA handle can be used in subsequent DMA subroutine calls to refer to such DMA objects.
DMA cookie - A ddi_dma_cookie(9S) structure (ddi_dma_cookie_t) describes a contiguous portion of a DMA object that is entirely addressable by the device. It contains DMA addressing information required to program the DMA engine.

Rather than knowing that a platform needs to map an object (typically a memory buffer) into a special DMA area of the kernel address space, device drivers instead allocate DMA resources for the object. The DMA routines then perform any platform-specific operations needed to set the object up for DMA access. The driver receives a DMA handle to identify the DMA resources allocated for the object. This handle is opaque to the device driver; the driver must save the handle and pass it in subsequent calls to DMA routines, but should not interpret it in any way.

Operations are defined on a DMA handle that provide the following services:

Manipulating DMA resources
Synchronizing DMA objects
Retrieving attributes of the allocated resources

Types of Device DMA

Devices perform one of the following three types of DMA:

Bus-Master DMA

If the device is capable of acting as a true bus master (where the DMA engine resides on the device board), the driver should program the device's DMA registers directly. The transfer address and count are obtained from the DMA cookie and given to the device.

Third-party DMA

Third-party DMA utilizes a system DMA engine resident on the main system board, which has several DMA channels available for use by devices. The device relies on the system's DMA engine to perform the data transfers between the device and memory. The driver uses DMA engine routines (see ddi_dmae(9F)) to initialize and program the DMA engine. For each DMA data transfer, the driver programs the DMA engine and then gives the device a command to initiate the transfer in cooperation with that engine.

First-party DMA

Under first-party DMA, the device drives its own DMA bus cycles using a channel from the system's DMA engine. The ddi_dmae_1stparty(9F) function is used to configure this channel in a cascade mode so that the DMA engine will not interfere with the transfer.

Types of Host Platform DMA

The platform that the device operates on provides one of two types of memory access: direct memory access (DMA) or direct virtual memory access (DVMA).

On platforms that support DMA, the system provides the device with a physical address in order to perform transfers. In this case, the transfer of a DMA object can actually consist of a number of physically discontiguous transfers. An example of this occurs when an application transfers a buffer that spans several contiguous virtual pages that map to physically discontiguous pages. To deal with the discontiguous memory, devices for these platforms usually have some kind of scatter-gather DMA capability. Typically, IA systems provide physical addresses for direct memory transfers.

On platforms that support DVMA, the system provides the device with a virtual address to perform transfers. In this case, the underlying platform provides some form of memory management unit (MMU) that translates device accesses to these virtual addresses into the proper physical addresses. The device transfers to and from a contiguous virtual image that can be mapped to discontiguous physical pages. Devices that operate in these platforms don't need scatter-gather DMA capability. Typically, SPARC platforms provide virtual addresses for direct memory transfers.

DMA Software Components: Handles, Windows, and Cookies

A DMA handle is an opaque pointer representing an object (usually a memory buffer or address) where a device can perform DMA transfers. Several different calls to DMA routines use the handle to identify the DMA resources allocated for the object.

An object represented by a DMA handle is completely covered by one or more DMA cookies. A DMA cookie represents a contiguous piece of memory to or from which the DMA engine can transfer data. The system uses the information in a DMA attribute ddi_dma_attr(9S) structure provided by the driver, as well as the memory location and alignment of the target object, to decide how to divide an object into multiple cookies.

If the object is too big to fit the request within system resource limitations, it has to be broken up into multiple DMA windows. Only one window is activated at one time and has resources allocated. The ddi_dma_getwin(9F) function is used to position between windows within an object. Each DMA window consists of one or more DMA cookies. For more information, see "DMA Windows".

Some DMA engines can accept more than one cookie. Such engines perform scatter-gather I/O without the help of the system. In this case, it is most efficient if the driver uses ddi_dma_nextcookie(9F) to get as many cookies as the DMA engine can handle and program them all into the engine. The device can then be programmed to transfer the total number of bytes covered by all these DMA cookies combined.


	Chapter 8 Direct Memory Access (DMA) Many devices can temporarily take control of the bus and perform data transfers to (and from) main memory or other devices. Because the device is doing the work without the help of the CPU, this type of data transfer is known as direct memory access (DMA). DMA transfers can be performed between two devices, between a device and memory, or between memory and memory. This chapter explains transfers between a device and memory only. This chapter provides information on the following subjects: "DMA Model" "Types of Device DMA" "Managing DMA Resources" "DMA Software Components: Handles, Windows, and Cookies" "DMA Operations" "Managing DMA Resources" "DMA Windows" DMA Model The Solaris Device Driver Interface/Driver-Kernel Interface (DDI/DKI) provides a high-level, architecture-independent model for DMA. This allows the framework (the DMA routines) to hide such architecture-specific details as: Setting up DMA mappings Building scatter-gather lists Ensuring that I/O and CPU caches are consistent There are several abstractions that are used in the DDI/DKI to describe aspects of a DMA transaction. These include: DMA object - Memory that is the source or destination of a DMA transfer. DMA handle - An opaque object returned from a successful `ddi_dma_alloc_handle`(9F) call. The DMA handle can be used in subsequent DMA subroutine calls to refer to such DMA objects. DMA cookie - A `ddi_dma_cookie`(9S) structure (`ddi_dma_cookie_t`) describes a contiguous portion of a DMA object that is entirely addressable by the device. It contains DMA addressing information required to program the DMA engine. Rather than knowing that a platform needs to map an object (typically a memory buffer) into a special DMA area of the kernel address space, device drivers instead allocate DMA resources for the object. The DMA routines then perform any platform-specific operations needed to set the object up for DMA access. The driver receives a DMA handle to identify the DMA resources allocated for the object. This handle is opaque to the device driver; the driver must save the handle and pass it in subsequent calls to DMA routines, but should not interpret it in any way. Operations are defined on a DMA handle that provide the following services: Manipulating DMA resources Synchronizing DMA objects Retrieving attributes of the allocated resources Types of Device DMA Devices perform one of the following three types of DMA: "Bus-Master DMA" "Third-party DMA" "First-party DMA" Bus-Master DMA If the device is capable of acting as a true bus master (where the DMA engine resides on the device board), the driver should program the device's DMA registers directly. The transfer address and count are obtained from the DMA cookie and given to the device. Third-party DMA Third-party DMA utilizes a system DMA engine resident on the main system board, which has several DMA channels available for use by devices. The device relies on the system's DMA engine to perform the data transfers between the device and memory. The driver uses DMA engine routines (see `ddi_dmae`(9F)) to initialize and program the DMA engine. For each DMA data transfer, the driver programs the DMA engine and then gives the device a command to initiate the transfer in cooperation with that engine. First-party DMA Under first-party DMA, the device drives its own DMA bus cycles using a channel from the system's DMA engine. The `ddi_dmae_1stparty`(9F) function is used to configure this channel in a cascade mode so that the DMA engine will not interfere with the transfer. Types of Host Platform DMA The platform that the device operates on provides one of two types of memory access: direct memory access (DMA) or direct virtual memory access (DVMA). On platforms that support DMA, the system provides the device with a physical address in order to perform transfers. In this case, the transfer of a DMA object can actually consist of a number of physically discontiguous transfers. An example of this occurs when an application transfers a buffer that spans several contiguous virtual pages that map to physically discontiguous pages. To deal with the discontiguous memory, devices for these platforms usually have some kind of scatter-gather DMA capability. Typically, IA systems provide physical addresses for direct memory transfers. On platforms that support DVMA, the system provides the device with a virtual address to perform transfers. In this case, the underlying platform provides some form of memory management unit (MMU) that translates device accesses to these virtual addresses into the proper physical addresses. The device transfers to and from a contiguous virtual image that can be mapped to discontiguous physical pages. Devices that operate in these platforms don't need scatter-gather DMA capability. Typically, SPARC platforms provide virtual addresses for direct memory transfers. DMA Software Components: Handles, Windows, and Cookies A DMA handle is an opaque pointer representing an object (usually a memory buffer or address) where a device can perform DMA transfers. Several different calls to DMA routines use the handle to identify the DMA resources allocated for the object. An object represented by a DMA handle is completely covered by one or more DMA cookies. A DMA cookie represents a contiguous piece of memory to or from which the DMA engine can transfer data. The system uses the information in a DMA attribute `ddi_dma_attr(9S)` structure provided by the driver, as well as the memory location and alignment of the target object, to decide how to divide an object into multiple cookies. If the object is too big to fit the request within system resource limitations, it has to be broken up into multiple DMA windows. Only one window is activated at one time and has resources allocated. The `ddi_dma_getwin`(9F) function is used to position between windows within an object. Each DMA window consists of one or more DMA cookies. For more information, see "DMA Windows". Some DMA engines can accept more than one cookie. Such engines perform scatter-gather I/O without the help of the system. In this case, it is most efficient if the driver uses `ddi_dma_nextcookie`(9F) to get as many cookies as the DMA engine can handle and program them all into the engine. The device can then be programmed to transfer the total number of bytes covered by all these DMA cookies combined.