BUS_DMA(9) | Kernel Developer's Manual | BUS_DMA(9) |
bus_dma
,
bus_dma_tag_create
,
bus_dma_tag_destroy
,
bus_dmamap_create
,
bus_dmamap_destroy
,
bus_dmamap_load
,
bus_dmamap_load_bio
,
bus_dmamap_load_ccb
,
bus_dmamap_load_mbuf
,
bus_dmamap_load_mbuf_sg
,
bus_dmamap_load_uio
,
bus_dmamap_unload
,
bus_dmamap_sync
,
bus_dmamem_alloc
,
bus_dmamem_free
— Bus and
Machine Independent DMA Mapping Interface
#include
<machine/bus.h>
int
bus_dma_tag_create
(bus_dma_tag_t
parent, bus_size_t
alignment, bus_addr_t
boundary, bus_addr_t
lowaddr, bus_addr_t
highaddr,
bus_dma_filter_t
*filtfunc, void
*filtfuncarg, bus_size_t
maxsize, int
nsegments, bus_size_t
maxsegsz, int
flags, bus_dma_lock_t
*lockfunc, void
*lockfuncarg,
bus_dma_tag_t *dmat);
int
bus_dma_tag_destroy
(bus_dma_tag_t
dmat);
int
bus_dmamap_create
(bus_dma_tag_t
dmat, int flags,
bus_dmamap_t *mapp);
int
bus_dmamap_destroy
(bus_dma_tag_t
dmat, bus_dmamap_t
map);
int
bus_dmamap_load
(bus_dma_tag_t
dmat, bus_dmamap_t
map, void *buf,
bus_size_t buflen,
bus_dmamap_callback_t
*callback, void
*callback_arg, int
flags);
int
bus_dmamap_load_bio
(bus_dma_tag_t
dmat, bus_dmamap_t
map, struct bio
*bio,
bus_dmamap_callback_t
*callback, void
*callback_arg, int
flags);
int
bus_dmamap_load_ccb
(bus_dma_tag_t
dmat, bus_dmamap_t
map, union ccb
*ccb,
bus_dmamap_callback_t
*callback, void
*callback_arg, int
flags);
int
bus_dmamap_load_mbuf
(bus_dma_tag_t
dmat, bus_dmamap_t
map, struct mbuf
*mbuf,
bus_dmamap_callback2_t
*callback, void
*callback_arg, int
flags);
int
bus_dmamap_load_mbuf_sg
(bus_dma_tag_t
dmat, bus_dmamap_t
map, struct mbuf
*mbuf, bus_dma_segment_t
*segs, int *nsegs,
int flags);
int
bus_dmamap_load_uio
(bus_dma_tag_t
dmat, bus_dmamap_t
map, struct uio
*uio,
bus_dmamap_callback2_t
*callback, void
*callback_arg, int
flags);
void
bus_dmamap_unload
(bus_dma_tag_t
dmat, bus_dmamap_t
map);
void
bus_dmamap_sync
(bus_dma_tag_t
dmat, bus_dmamap_t
map, op);
int
bus_dmamem_alloc
(bus_dma_tag_t
dmat, void **vaddr,
int flags,
bus_dmamap_t *mapp);
void
bus_dmamem_free
(bus_dma_tag_t
dmat, void *vaddr,
bus_dmamap_t map);
Direct Memory Access (DMA) is a method of transferring data without involving the CPU, thus providing higher performance. A DMA transaction can be achieved between device to memory, device to device, or memory to memory.
The bus_dma
API is a bus, device, and
machine-independent (MI) interface to DMA mechanisms. It provides the client
with flexibility and simplicity by abstracting machine dependent issues like
setting up DMA mappings, handling cache issues, bus specific features and
limitations.
A tag structure (bus_dma_tag_t) is used to describe the properties of a group of related DMA transactions. One way to view this is that a tag describes the limitations of a DMA engine. For example, if a DMA engine in a device is limited to 32-bit addresses, that limitation is specified by a parameter when creating the tag for that device. Similarly, a tag can be marked as requiring buffers whose addresses are aligned to a specific boundary.
Some devices may require multiple tags to describe DMA transactions with differing properties. For example, a device might require 16-byte alignment of its descriptor ring while permitting arbitrary alignment of I/O buffers. In this case, the driver must create one tag for the descriptor ring and a separate tag for I/O buffers. If a device has restrictions that are common to all DMA transactions in addition to restrictions that differ between unrelated groups of transactions, the driver can first create a “parent” tag that decribes the common restrictions. The per-group tags can then inherit these restrictions from this “parent” tag rather than having to list them explicitly when creating the per-group tags.
A mapping structure (bus_dmamap_t) represents a mapping of a memory region for DMA. On systems with I/O MMUs, the mapping structure tracks any I/O MMU entries used by a request. For DMA requests that require bounce pages, the mapping tracks the bounce pages used.
To prepare for one or more DMA transactions,
a mapping must be bound to a memory region by calling one of the
bus_dmamap_load
()
functions. These functions configure the mapping which can include
programming entries in an I/O MMU and/or allocating bounce pages. An output
of these functions (either directly or indirectly by invoking a callback
routine) is the list of scatter/gather address ranges a consumer can pass to
a DMA engine to access the memory region. When a mapping is no longer
needed, the mapping must be unloaded via
bus_dmamap_unload
().
Before and after each DMA transaction,
bus_dmamap_sync
()
must be used to ensure that the correct data is used by the DMA engine and
the CPU. If a mapping uses bounce pages, the sync operations copy data
between the bounce pages and the memory region bound to the mapping. Sync
operations also handle architecture-specific details such as CPU cache
flushing and CPU memory operation ordering.
bus_dma
handles two types of DMA
transactions: static and dynamic. Static transactions are used with a
long-lived memory region that is reused for many transactions such as a
descriptor ring. Dynamic transactions are used for transfers to or from
transient buffers such as I/O buffers holding a network packet or disk
block. Each transaction type uses a different subset of the
bus_dma
API.
Static transactions use memory regions allocated by
bus_dma
. Each static memory region is allocated by
calling bus_dmamem_alloc
(). This function requires a
valid tag describing the properties of the DMA transactions to this region
such as alignment or address restrictions. Multiple regions can share a
single tag if they share the same restrictions.
bus_dmamem_alloc
()
allocates a memory region along with a mapping object. The associated tag,
memory region, and mapping object must then be passed to
bus_dmamap_load
() to bind the mapping to the
allocated region and obtain the scatter/gather list.
It is expected that
bus_dmamem_alloc
()
will attempt to allocate memory requiring less expensive sync operations
(for example, implementations should not allocate regions requiring bounce
pages), but sync operations should still be used. For example, a driver
should use bus_dmamap_sync
() in an interrupt handler
before reading descriptor ring entries written by the device prior to the
interrupt.
When a consumer is finished with a memory
region, it should unload the mapping via
bus_dmamap_unload
()
and then release the memory region and mapping object via
bus_dmamem_free
().
Dynamic transactions map memory regions provided by other parts of
the system. A tag must be created via
bus_dma_tag_create
() to describe the DMA
transactions to and from these memory regions, and a pool of mapping objects
must be allocated via bus_dmamap_create
() to track
the mappings of any in-flight transactions.
When a consumer wishes to schedule a
transaction for a memory region, the consumer must first obtain an unused
mapping object from its pool of mapping objects. The memory region must be
bound to the mapping object via one of the
bus_dmamap_load
()
functions. Before scheduling the transaction, the consumer should sync the
memory region via bus_dmamap_sync
() with one or more
of the “PRE” flags. After the transaction has completed, the
consumer should sync the memory region via
bus_dmamap_sync
() with one or more of the
“POST” flags. The mapping can then be unloaded via
bus_dmamap_unload
(), and the mapping object can be
returned to the pool of unused mapping objects.
When a consumer is no longer scheduling
DMA transactions, the mapping objects should be freed via
bus_dmamap_destroy
(),
and the tag should be freed via
bus_dma_tag_destroy
().
client_filter
(void
*filtarg, bus_addr_t testaddr)Address filters can be specified during tag creation to allow
for devices whose DMA address restrictions cannot be specified by a
single window. The filtarg argument is specified
by the client during tag creation to be passed to all invocations of the
callback. The testaddr argument contains a
potential starting address of a DMA mapping. The filter function
operates on the set of addresses from testaddr to
‘trunc_page(testaddr) + PAGE_SIZE -
1
’, inclusive. The filter function should return zero if
any mapping in this range can be accommodated by the device and non-zero
otherwise.
bus_addr_t ds_addr; bus_size_t ds_len;
The ds_addr field contains the device visible address of the DMA segment, and ds_len contains the length of the DMA segment. Although the DMA segments returned by a mapping call will adhere to all restrictions necessary for a successful DMA operation, some conversion (e.g. a conversion from host byte order to the device's byte order) is almost always required when presenting segment information to the device.
NULL
on some platforms under certain conditions,
it should never be assumed that it will be NULL
in
all cases.bus_dmamap_load
(),
bus_dmamap_load_bio
() or
bus_dmamap_load_ccb
(). Callbacks are of the
format:
client_callback
(void
*callback_arg, bus_dma_segment_t *segs,
int nseg, int error)The callback_arg is the callback argument passed to dmamap load functions. The segs and nseg arguments describe an array of bus_dma_segment_t structures that represent the mapping. This array is only valid within the scope of the callback function. The success or failure of the mapping is indicated by the error argument. More information on the use of callbacks can be found in the description of the individual dmamap load functions.
bus_dmamap_load_uio
()
or bus_dmamap_load_mbuf
().
Callback2s are of the format:
client_callback2
(void
*callback_arg, bus_dma_segment_t *segs,
int nseg, bus_size_t
mapsize, int error)Callback2's behavior is the same as bus_dmamap_callback_t with the addition that the length of the data mapped is provided via mapsize.
bus_dmamap_sync
()
description below for more details on how to use these operations.
All operations specified below are performed from the host
memory point of view, where a read implies data coming from the device
to the host memory, and a write implies data going from the host memory
to the device. Alternatively, the operations can be thought of in terms
of driver operations, where reading a network packet or storage sector
corresponds to a read operation in bus_dma
.
BUS_DMASYNC_PREREAD
BUS_DMASYNC_PREWRITE
BUS_DMASYNC_POSTREAD
BUS_DMASYNC_POSTWRITE
BUS_DMA_LOCK
and immediately after with
BUS_DMA_UNLOCK
. If the load operation does not
need to be deferred, then it will not be called since the function loading
the map should be holding the appropriate locks. This method is of the
format:
lockfunc
(void
*lockfunc_arg, bus_dma_lock_op_t op)The lockfuncarg argument is specified by the client during tag creation to be passed to all invocations of the callback. The op argument specifies the lock operation to perform.
Two lockfunc
implementations are provided for convenience.
busdma_lock_mutex
()
performs standard mutex operations on the sleep mutex provided via
lockfuncarg.
dflt_lock
()
will generate a system panic if it is called. It is substituted into the
tag when lockfunc is passed as
NULL
to
bus_dma_tag_create
() and is useful for tags that
should not be used with deferred load operations.
lockfunc
().
BUS_DMA_LOCK
BUS_DMA_UNLOCK
bus_dma_tag_create
(parent,
alignment, boundary,
lowaddr, highaddr,
*filtfunc, *filtfuncarg,
maxsize, nsegments,
maxsegsz, flags,
lockfunc, lockfuncarg,
*dmat)All tags created by a device driver
must inherit from the tag returned by
bus_get_dma_tag
()
to honor restrictions between the parent bridge, CPU memory, and the
device.
0
’ indicates that there are no
boundary restrictions.BUS_SPACE_MAXADDR
and a
lowaddr of
BUS_SPACE_MAXADDR_32BIT
. Similarly a device
that can only perform DMA to addresses below 16MB would specify a
highaddr of
BUS_SPACE_MAXADDR
and a
lowaddr of
BUS_SPACE_MAXADDR_24BIT
. Some implementations
require that some region of device visible address space, overlapping
available host memory, be outside the window. This area of
‘safe memory
’ is used to bounce
requests that would otherwise conflict with the exclusion window.NULL
) to be
called for any attempt to map memory into the window described by
lowaddr and highaddr. A
filter function is only required when the single window described by
lowaddr and highaddr
cannot adequately describe the constraints of the device. The filter
function will be called for every machine page that overlaps the
exclusion window.NULL
.BUS_SPACE_UNRESTRICTED
may be specified.BUS_DMA_ALLOCNOW
ENOMEM
is returned. This should not be
used for tags that only describe buffers that will be allocated
with bus_dmamem_alloc
(). Also, due to
resource sharing with other tags, this flag does not guarantee
that resources will be allocated or reserved exclusively for this
tag. It should be treated only as a minor optimization.BUS_DMA_COHERENT
bus_dmamem_alloc
(). For
bus_dma_tag_create
(), the
BUS_DMA_COHERENT
flag is currently
implemented on arm64.NULL
) to be called when busdma needs to
manipulate a lock on behalf of the client. If
NULL
is specified,
dflt_lock
() is used.Returns ENOMEM
if sufficient memory is
not available for tag creation or allocating mapping resources.
bus_dma_tag_destroy
(dmat)bus_dma_tag_create
().
Returns EBUSY
if any DMA maps remain
associated with dmat or
‘0
’ on success.
bus_dmamap_create
(dmat,
flags, *mapp)BUS_DMA_COHERENT
bus_dmamap_sync
(), but it may reduce the
cost of performing these operations. For
bus_dmamap_create
(), the
BUS_DMA_COHERENT
flag is currently
implemented on sparc64.Returns ENOMEM
if sufficient memory is
not available for creating the map or allocating mapping resources.
bus_dmamap_destroy
(dmat,
map)Returns EBUSY
if a mapping is still
active for map.
bus_dmamap_load
(dmat,
map, buf,
buflen, *callback,
callback_arg, flags)BUS_DMA_NOWAIT
BUS_DMA_NOCACHE
bus_dmamap_load
(), the
BUS_DMA_NOCACHE
flag is currently
implemented on sparc64.Return values to the caller are as follows:
EINPROGRESS
Note that subsequent load operations for the same tag that do not require extra resources will still succeed. This may result in out-of-order processing of requests. If the caller requires the order of requests to be preserved, then the caller is required to stall subsequent requests until a pending request's callback is invoked.
ENOMEM
BUS_DMA_NOWAIT
flag.EINVAL
When the callback is called, it is presented with an error value indicating the disposition of the mapping. Error may be one of the following:
EFBIG
bus_dmamap_load_bio
(dmat,
map, bio,
callback, callback_arg,
flags)bus_dmamap_load
() which
maps buffers pointed to by bio for DMA transfers.
bio may point to either a mapped or unmapped
buffer.bus_dmamap_load_ccb
(dmat,
map, ccb,
callback, callback_arg,
flags)bus_dmamap_load
() which
maps data pointed to by ccb for DMA transfers. The
data for ccb may be any of the following types:
bus_dmamap_load_ccb
()
supports the following CCB XPT function codes:
bus_dmamap_load_mbuf
(dmat,
map, mbuf,
callback2, callback_arg,
flags)bus_dmamap_load
() which
maps mbuf chains for DMA transfers. A bus_size_t
argument is also passed to the callback routine, which contains the mbuf
chain's packet header length. The BUS_DMA_NOWAIT
flag is implied, thus no callback deferral will happen.
Mbuf chains are assumed to be in kernel virtual address space.
Beside the error values listed for
bus_dmamap_load
(),
EINVAL
will be returned if the size of the mbuf
chain exceeds the maximum limit of the DMA tag.
bus_dmamap_load_mbuf_sg
(dmat,
map, mbuf,
segs, nsegs,
flags)bus_dmamap_load_mbuf
() except
that it returns immediately without calling a callback function. It is
provided for efficiency. The scatter/gather segment array
segs is provided by the caller and filled in
directly by the function. The nsegs argument is
returned with the number of segments filled in. Returns the same errors as
bus_dmamap_load_mbuf
().bus_dmamap_load_uio
(dmat,
map, uio,
callback2, callback_arg,
flags)bus_dmamap_load
() which
maps buffers pointed to by uio for DMA transfers. A
bus_size_t argument is also passed to the callback
routine, which contains the size of uio, i.e.
uio->uio_resid. The
BUS_DMA_NOWAIT
flag is implied, thus no callback
deferral will happen. Returns the same errors as
bus_dmamap_load
().
If uio->uio_segflg is
UIO_USERSPACE
, then it is assumed that the
buffer, uio is in
uio->uio_td->td_proc's address space. User
space memory must be in-core and wired prior to attempting a map load
operation. Pages may be locked using vslock(9).
bus_dmamap_unload
(dmat,
map)bus_dmamap_unload
()
will not perform any implicit synchronization of DMA buffers. This must
be done explicitly by a call to
bus_dmamap_sync
() prior to unloading the
map.
bus_dmamap_sync
(dmat,
map, op)The
bus_dmamap_sync
()
function is the method used to ensure that CPU's and device's direct
memory access (DMA) to shared memory is coherent. For example, the CPU
might be used to set up the contents of a buffer that is to be made
available to a device. To ensure that the data are visible via the
device's mapping of that memory, the buffer must be loaded and a DMA
sync operation of BUS_DMASYNC_PREWRITE
must be
performed after the CPU has updated the buffer and before the device
access is initiated. If the CPU modifies this buffer again later,
another BUS_DMASYNC_PREWRITE
sync operation must
be performed before an additional device access. Conversely, suppose a
device updates memory that is to be read by a CPU. In this case, the
buffer must be loaded, and a DMA sync operation of
BUS_DMASYNC_PREREAD
must be performed before the
device access is initiated. The CPU will only be able to see the results
of this memory update once the DMA operation has completed and a
BUS_DMASYNC_POSTREAD
sync operation has been
performed.
If read and write operations are not preceded and followed by the appropriate synchronization operations, behavior is undefined.
bus_dmamem_alloc
(dmat,
**vaddr, flags,
*mapp)BUS_DMA_WAITOK
BUS_DMA_NOWAIT
ENOMEM
is returned.BUS_DMA_COHERENT
bus_dmamap_create
() above for a
description of this flag. For
bus_dmamem_alloc
(), the
BUS_DMA_COHERENT
flag is currently
implemented on arm, arm64 and sparc64.BUS_DMA_ZERO
BUS_DMA_NOCACHE
bus_dmamem_alloc
(), the
BUS_DMA_NOCACHE
flag is currently
implemented on amd64 and i386 where it results in the Strong
Uncacheable PAT to be set for the allocated virtual address
range.The size of memory to be allocated is
maxsize as specified in the call to
bus_dma_tag_create
()
for dmat.
The current implementation of
bus_dmamem_alloc
()
will allocate all requests as a single segment.
An initial load operation is required to
obtain the bus address of the allocated memory, and an unload operation
is required before freeing the memory, as described below in
bus_dmamem_free
().
Maps are automatically handled by this function and should not be
explicitly allocated or destroyed.
Although an explicit load is not
required for each access to the memory referenced by the returned map,
the synchronization requirements as described in the
bus_dmamap_sync
()
section still apply and should be used to achieve portability on
architectures without coherent buses.
Returns ENOMEM
if sufficient memory is
not available for completing the operation.
bus_dmamem_free
(dmat,
*vaddr, map)bus_dmamem_alloc
(). Any mappings will be
invalidated. Arguments are as follows:
Behavior is undefined if invalid arguments are passed to any of
the above functions. If sufficient resources cannot be allocated for a given
transaction, ENOMEM
is returned. All routines that
are not of type void will return 0 on success or an
error code on failure as discussed above.
All void routines will succeed if provided with valid arguments.
Two locking protocols are used by bus_dma
.
The first is a private global lock that is used to synchronize access to the
bounce buffer pool on the architectures that make use of them. This lock is
strictly a leaf lock that is only used internally to
bus_dma
and is not exposed to clients of the
API.
The second protocol involves protecting
various resources stored in the tag. Since almost all
bus_dma
operations are done through requests from
the driver that created the tag, the most efficient way to protect the tag
resources is through the lock that the driver uses. In cases where
bus_dma
acts on its own without being called by the
driver, the lock primitive specified in the tag is acquired and released
automatically. An example of this is when the
bus_dmamap_load
()
callback function is called from a deferred context instead of the driver
context. This means that certain bus_dma
functions
must always be called with the same lock held that is specified in the tag.
These functions include:
bus_dmamap_load
()bus_dmamap_load_bio
()bus_dmamap_load_ccb
()bus_dmamap_load_mbuf
()bus_dmamap_load_mbuf_sg
()bus_dmamap_load_uio
()bus_dmamap_unload
()bus_dmamap_sync
()There is one exception to this rule. It is common practice to call some of these functions during driver start-up without any locks held. So long as there is a guarantee of no possible concurrent use of the tag by different threads during this operation, it is safe to not hold a lock for these functions.
Certain bus_dma
operations should not be
called with the driver lock held, either because they are already protected
by an internal lock, or because they might sleep due to memory or resource
allocation. The following functions must not be called with any
non-sleepable locks held:
bus_dma_tag_create
()bus_dmamap_create
()bus_dmamem_alloc
()All other functions do not have a locking protocol and can thus be called with or without any system or driver locks held.
devclass(9), device(9), driver(9), rman(9), vslock(9)
Jason R. Thorpe, A Machine-Independent DMA Framework for NetBSD, Proceedings of the Summer 1998 USENIX Technical Conference, USENIX Association, June 1998.
The bus_dma
interface first appeared in
NetBSD 1.3.
The bus_dma
API was adopted from
NetBSD for use in the CAM SCSI subsystem. The
alterations to the original API were aimed to remove the need for a
bus_dma_segment_t array stored in each
bus_dmamap_t while allowing callers to queue up on
scarce resources.
The bus_dma
interface was designed and
implemented by Jason R. Thorpe of the Numerical
Aerospace Simulation Facility, NASA Ames Research Center. Additional input
on the bus_dma
design was provided by
Chris Demetriou, Charles
Hannum, Ross Harvey, Matthew
Jacob, Jonathan Stone, and
Matt Thomas.
The bus_dma
interface in
FreeBSD benefits from the contributions of
Justin T. Gibbs, Peter Wemm,
Doug Rabson, Matthew N.
Dodd, Sam Leffler, Maxime
Henrion, Jake Burkholder,
Takahashi Yoshihiro, Scott
Long and many others.
This manual page was written by Hiten M. Pandya and Justin T. Gibbs.
August 11, 2018 | Debian |