| io_uring_enter(2) | Linux Programmer's Manual | io_uring_enter(2) |
io_uring_enter - initiate and/or complete asynchronous I/O
#include <liburing.h>
int io_uring_enter(unsigned int fd, unsigned int to_submit,
unsigned int min_complete, unsigned int flags,
sigset_t *sig);
int io_uring_enter2(unsigned int fd, unsigned int to_submit,
unsigned int min_complete, unsigned int flags,
sigset_t *sig, size_t sz);
io_uring_enter(2) is used to initiate and complete I/O using the shared submission and completion queues setup by a call to io_uring_setup(2). A single call can both submit new I/O and wait for completions of I/O initiated by this call or previous calls to io_uring_enter(2).
fd is the file descriptor returned by io_uring_setup(2). to_submit specifies the number of I/Os to submit from the submission queue. flags is a bitmask of the following values:
int io_uring_enter2(unsigned int fd, unsigned int to_submit,
unsigned int min_complete, unsigned int flags,
const void *arg, size_t argsz);
which behaves just like the original definition by default. However, if IORING_ENTER_EXT_ARG is set, then instead of a sigset_t being passed in, a pointer to a struct io_uring_getevents_arg is used instead and argsz must be set to the size of this structure. The definition is as follows:
struct io_uring_getevents_args {
__u64 sigmask;
__u32 sigmask_sz;
__u32 pad;
__u64 ts;
};
which allows passing in both a signal mask as well as pointer to a struct __kernel_timespec timeout value. If ts is set to a valid pointer, then this time value indicates the timeout for waiting on events. If an application is waiting on events and wishes to stop waiting after a specified amount of time, then this can be accomplished directly in version 5.11 and newer by using this feature.
If the io_uring instance was configured for polling, by specifying IORING_SETUP_IOPOLL in the call to io_uring_setup(2), then min_complete has a slightly different meaning. Passing a value of 0 instructs the kernel to return any events which are already complete, without blocking. If min_complete is a non-zero value, the kernel will still return immediately if any completion events are available. If no event completions are available, then the call will poll either until one or more completions become available, or until the process has exceeded its scheduler time slice.
Note that, for interrupt driven I/O (where IORING_SETUP_IOPOLL was not specified in the call to io_uring_setup(2)), an application may check the completion queue for event completions without entering the kernel at all.
When the system call returns that a certain amount of SQEs have been consumed and submitted, it's safe to reuse SQE entries in the ring. This is true even if the actual IO submission had to be punted to async context, which means that the SQE may in fact not have been submitted yet. If the kernel requires later use of a particular SQE entry, it will have made a private copy of it.
sig is a pointer to a signal mask (see sigprocmask(2)); if sig is not NULL, io_uring_enter(2) first replaces the current signal mask by the one pointed to by sig, then waits for events to become available in the completion queue, and then restores the original signal mask. The following io_uring_enter(2) call:
ret = io_uring_enter(fd, 0, 1, IORING_ENTER_GETEVENTS, &sig);
is equivalent to atomically executing the following calls:
pthread_sigmask(SIG_SETMASK, &sig, &orig); ret = io_uring_enter(fd, 0, 1, IORING_ENTER_GETEVENTS, NULL); pthread_sigmask(SIG_SETMASK, &orig, NULL);
See the description of pselect(2) for an explanation of why the sig parameter is necessary.
Submission queue entries are represented using the following data structure:
/*
* IO submission data structure (Submission Queue Entry)
*/ struct io_uring_sqe {
__u8 opcode; /* type of operation for this sqe */
__u8 flags; /* IOSQE_ flags */
__u16 ioprio; /* ioprio for the request */
__s32 fd; /* file descriptor to do IO on */
union {
__u64 off; /* offset into file */
__u64 addr2;
};
union {
__u64 addr; /* pointer to buffer or iovecs */
__u64 splice_off_in;
}
__u32 len; /* buffer size or number of iovecs */
union {
__kernel_rwf_t rw_flags;
__u32 fsync_flags;
__u16 poll_events; /* compatibility */
__u32 poll32_events; /* word-reversed for BE */
__u32 sync_range_flags;
__u32 msg_flags;
__u32 timeout_flags;
__u32 accept_flags;
__u32 cancel_flags;
__u32 open_flags;
__u32 statx_flags;
__u32 fadvise_advice;
__u32 splice_flags;
__u32 rename_flags;
__u32 unlink_flags;
__u32 hardlink_flags;
};
__u64 user_data; /* data to be passed back at completion time */
union {
struct {
/* index into fixed buffers, if used */
union {
/* index into fixed buffers, if used */
__u16 buf_index;
/* for grouped buffer selection */
__u16 buf_group;
}
/* personality to use, if used */
__u16 personality;
union {
__s32 splice_fd_in;
__u32 file_index; };
};
__u64 __pad2[3];
}; };
The opcode describes the operation to be performed. It can be one of:
If IORING_POLL_ADD_MULTI is set in the SQE len field, then the poll will work in multi shot mode instead. That means it'll repatedly trigger when the requested event becomes true, and hence multiple CQEs can be generated from this single SQE. The CQE flags field will have IORING_CQE_F_MORE set on completion if the application should expect further CQE entries from the original request. If this flag isn't set on completion, then the poll request has been terminated and no further events will be generated. This mode is available since 5.13.
If IORING_POLL_UPDATE_EVENTS is set in the SQE len field, then the request will update an existing poll request with the mask of events passed in with this request. The lookup is based on the user_data field of the original SQE submitted, and this values is passed in the addr field of the SQE. This mode is available since 5.13.
If IORING_POLL_UPDATE_USER_DATA is set in the SQE len field, then the request will update the user_data of an existing poll request based on the value passed in the off field. This mode is available since 5.13.
This command works like an async poll(2) and the completion event result is the returned mask of events. For the variants that update user_data or events , the completion result will be similar to IORING_OP_POLL_REMOVE.
This command also supports the following modifiers in ioprio:
IORING_RECVSEND_POLL_FIRST If set, io_uring will assume the socket is
currently full and attempting to send data will be unsuccessful. For this
case, io_uring will arm internal poll and trigger a send of the data when
there is enough space available. This initial send attempt can be wasteful
for the case where the socket is expected to be full, setting this flag will
bypass the initial send attempt and go straight to arming poll. If poll does
indicate that data can be sent, the operation will proceed.
This command also supports the following modifiers in ioprio:
IORING_RECVSEND_POLL_FIRST If set, io_uring will assume the socket is
currently empty and attempting to receive data will be unsuccessful. For
this case, io_uring will arm internal poll and trigger a receive of the data
when the socket has data to be read. This initial receive attempt can be
wasteful for the case where the socket is expected to be empty, setting this
flag will bypass the initial receive attempt and go straight to arming poll.
If poll does indicate that data is ready to be received, the operation will
proceed.
This command also supports the following modifiers in ioprio:
IORING_RECVSEND_POLL_FIRST If set, io_uring will assume the socket is
currently full and attempting to send data will be unsuccessful. For this
case, io_uring will arm internal poll and trigger a send of the data when
there is enough space available. This initial send attempt can be wasteful
for the case where the socket is expected to be full, setting this flag will
bypass the initial send attempt and go straight to arming poll. If poll does
indicate that data can be sent, the operation will proceed.
This command also supports the following modifiers in ioprio:
IORING_RECVSEND_POLL_FIRST If set, io_uring will assume the socket is
currently empty and attempting to receive data will be unsuccessful. For
this case, io_uring will arm internal poll and trigger a receive of the data
when the socket has data to be read. This initial receive attempt can be
wasteful for the case where the socket is expected to be empty, setting this
flag will bypass the initial receive attempt and go straight to arming poll.
If poll does indicate that data is ready to be received, the operation will
proceed.
Since 5.15, this command also supports the following modifiers in timeout_flags:
IORING_TIMEOUT_BOOTTIME If set, then the clocksource used is
CLOCK_BOOTTIME instead of CLOCK_MONOTONIC. This clocksource
differs in that it includes time elapsed if the system was suspend while
having a timeout request in-flight.
IORING_TIMEOUT_REALTIME If set, then the clocksource used
is CLOCK_REALTIME instead of CLOCK_MONOTONIC.
If timeout_flags contain IORING_TIMEOUT_UPDATE, instead of removing an existing operation, it updates it. addr and return values are same as before. addr2 field must contain a pointer to a struct timespec64 structure. timeout_flags may also contain IORING_TIMEOUT_ABS, in which case the value given is an absolute one, not a relative one. Available since 5.11.
If the file_index field is set to a positive number, the file won't be installed into the normal file table as usual but will be placed into the fixed file table at index file_index - 1. In this case, instead of returning a file descriptor, the result will contain either 0 on success or an error. If the index points to a valid empty slot, the installation is guaranteed to not fail. If there is already a file in the slot, it will be replaced, similar to IORING_OP_FILES_UPDATE. Please note that only io_uring has access to such files and no other syscall can use them. See IOSQE_FIXED_FILE and IORING_REGISTER_FILES.
Available since 5.5.
If the file_index field is set to a positive number, the file won't be installed into the normal file table as usual but will be placed into the fixed file table at index file_index - 1. In this case, instead of returning a file descriptor, the result will contain either 0 on success or an error. If the index points to a valid empty slot, the installation is guaranteed to not fail. If there is already a file in the slot, it will be replaced, similar to IORING_OP_FILES_UPDATE. Please note that only io_uring has access to such files and no other syscall can use them. See IOSQE_FIXED_FILE and IORING_REGISTER_FILES.
Available since 5.15.
If the file_index field is set to a positive number, the file won't be installed into the normal file table as usual but will be placed into the fixed file table at index file_index - 1. In this case, instead of returning a file descriptor, the result will contain either 0 on success or an error. If the index points to a valid empty slot, the installation is guaranteed to not fail. If there is already a file in the slot, it will be replaced, similar to IORING_OP_FILES_UPDATE. Please note that only io_uring has access to such files and no other syscall can use them. See IOSQE_FIXED_FILE and IORING_REGISTER_FILES.
Available since 5.15.
If the file_index field is set to a positive number, the file won't be installed into the normal file table as usual but will be placed into the fixed file table at index file_index - 1. In this case, instead of returning a file descriptor, the result will contain either 0 on success or an error. If the index points to a valid empty slot, the installation is guaranteed to not fail. If there is already a file in the slot, it will be replaced, similar to IORING_OP_FILES_UPDATE. Please note that only io_uring has access to such files and no other syscall can use them. See IOSQE_FIXED_FILE and IORING_REGISTER_FILES.
Available since 5.19.
The flags field of the first struct io_uring_cqe may likely contain IORING_CQE_F_MORE , which means that there will be a second completion event / notification for the request, with the user_data field set to the same value. The user must not modify the data buffer until the notification is posted. The first cqe follows the usual rules and so its res field will contain the number of bytes sent or a negative error code. The notification's res field will be set to zero and the flags field will contain IORING_CQE_F_NOTIF . The two step model is needed because the kernel may hold on to buffers for a long time, e.g. waiting for a TCP ACK, and having a separate cqe for request completions allows userspace to push more data without extra delays. Note, notifications are only responsible for controlling the lifetime of the buffers, and as such don't mean anything about whether the data has atually been sent out or received by the other end. Even errored requests may generate a notification, and the user must check for IORING_CQE_F_MORE rather than relying on the result.
fd must be set to the socket file descriptor, addr must contain a pointer to the buffer, len denotes the length of the buffer to send, and msg_flags holds the flags associated with the system call. When addr2 is non-zero it points to the address of the target with addr_len specifying its size, turning the request into a sendto(2) system call equivalent.
Available since 6.0.
This command also supports the following modifiers in ioprio:
IORING_RECVSEND_POLL_FIRST If set, io_uring will assume the socket is
currently full and attempting to send data will be unsuccessful. For this
case, io_uring will arm internal poll and trigger a send of the data when
there is enough space available. This initial send attempt can be wasteful
for the case where the socket is expected to be full, setting this flag will
bypass the initial send attempt and go straight to arming poll. If poll does
indicate that data can be sent, the operation will proceed.
IORING_RECVSEND_FIXED_BUF If set, instructs io_uring to use
a pre-mapped buffer. The buf_index field should contain an index into
an array of fixed buffers. See io_uring_register(2) for details on
how to setup a context for fixed buffer I/O.
The flags field is a bit mask. The supported flags are:
The semantics are chosen to accommodate several use cases. First, when all but the last request of a normal link without linked timeouts are marked with the flag, only one CQE per lin is posted. Additionally, it enables suppression of CQEs in cases where the side effects of a successfully executed operation is enough for userspace to know the state of the system. One such example would be writing to a synchronisation file.
This flag is incompatible with IOSQE_IO_DRAIN. Using both of them in a single ring is undefined behavior, even when they are not used together in a single request. Currently, after the first request with IOSQE_CQE_SKIP_SUCCESS, all subsequent requests marked with drain will be failed at submission time. Note that the error reporting is best effort only, and restrictions may change in the future.
Available since 5.17.
ioprio specifies the I/O priority. See ioprio_get(2) for a description of Linux I/O priorities.
fd specifies the file descriptor against which the operation will be performed, with the exception noted above.
If the operation is one of IORING_OP_READ_FIXED or IORING_OP_WRITE_FIXED, addr and len must fall within the buffer located at buf_index in the fixed buffer array. If the operation is either IORING_OP_READV or IORING_OP_WRITEV, then addr points to an iovec array of len entries.
rw_flags, specified for read and write operations, contains a bitwise OR of per-I/O flags, as described in the preadv2(2) man page.
The fsync_flags bit mask may contain either 0, for a normal file integrity sync, or IORING_FSYNC_DATASYNC to provide data sync only semantics. See the descriptions of O_SYNC and O_DSYNC in the open(2) manual page for more information.
The bits that may be set in poll_events are defined in <poll.h>, and documented in poll(2).
user_data is an application-supplied value that will be copied into the completion queue entry (see below). buf_index is an index into an array of fixed buffers, and is only valid if fixed buffers were registered. personality is the credentials id to use for this operation. See io_uring_register(2) for how to register personalities with io_uring. If set to 0, the current personality of the submitting task is used.
Once the submission queue entry is initialized, I/O is submitted by placing the index of the submission queue entry into the tail of the submission queue. After one or more indexes are added to the queue, and the queue tail is advanced, the io_uring_enter(2) system call can be invoked to initiate the I/O.
Completions use the following data structure:
/*
* IO completion data structure (Completion Queue Entry)
*/ struct io_uring_cqe {
__u64 user_data; /* sqe->data submission passed back */
__s32 res; /* result code for this event */
__u32 flags; };
user_data is copied from the field of the same name in the submission queue entry. The primary use case is to store data that the application will need to access upon completion of this particular I/O. The flags is used for certain commands, like IORING_OP_POLL_ADD or in conjunction with IOSQE_BUFFER_SELECT or IORING_OP_MSG_RING, , see those entries for details. res is the operation-specific result, but io_uring-specific errors (e.g. flags or opcode invalid) are returned through this field. They are described in section CQE ERRORS.
For read and write opcodes, the return values match errno values documented in the preadv2(2) and pwritev2(2) man pages, with res holding the equivalent of -errno for error cases, or the transferred number of bytes in case the operation is successful. Hence both error and success return can be found in that field in the CQE. For other request types, the return values are documented in the matching man page for that type, or in the opcodes section above for io_uring-specific opcodes.
io_uring_enter(2) returns the number of I/Os successfully consumed. This can be zero if to_submit was zero or if the submission queue was empty. Note that if the ring was created with IORING_SETUP_SQPOLL specified, then the return value will generally be the same as to_submit as submission happens outside the context of the system call.
The errors related to a submission queue entry will be returned through a completion queue entry (see section CQE ERRORS), rather than through the system call itself.
Errors that occur not on behalf of a submission queue entry are returned via the system call directly. On such an error, a negative error code is returned. The caller should not rely on errno variable.
These are the errors returned by io_uring_enter(2) system call.
Without IORING_FEAT_NODROP the application is attempting to overcommit the number of requests it can have pending. The application should wait for some completions and try again. May occur if the application tries to queue more requests than we have room for in the CQ ring, or if the application attempts to wait for more events without having reaped the ones already present in the CQ ring.
These io_uring-specific errors are returned as a negative value in the res field of the completion queue entry.
| 2019-01-22 | Linux |