rbd - manage rados block device (RBD) images
rbd [ -c ceph.conf ] [ -m monaddr ] [--cluster cluster-name]
[ -p | --pool pool ] [ command ... ]
rbd is a utility for manipulating rados block device (RBD)
images, used by the Linux rbd driver and the rbd storage driver for
QEMU/KVM. RBD images are simple block devices that are striped over objects
and stored in a RADOS object store. The size of the objects the image is
striped over must be a power of two.
- -c ceph.conf, --conf
ceph.conf
- Use ceph.conf configuration file instead of the default
/etc/ceph/ceph.conf to determine monitor addresses during startup.
- --no-progress
- Do not output progress information (goes to standard error by default for
some commands).
- --image-format
format-id
- Specifies which object layout to use. The default is 2.
- format 1 - (deprecated) Use the original format for a new rbd image. This
format is understood by all versions of librbd and the kernel rbd module,
but does not support newer features like cloning.
- format 2 - Use the second rbd format, which is supported by librbd and
kernel since version 3.11 (except for striping). This adds support for
cloning and is more easily extensible to allow more features in the
future.
- --object-size
size-in-B/K/M
- Specifies the object size in B/K/M. Object size will be rounded up the
nearest power of two; if no suffix is given, unit B is assumed. The
default object size is 4M, smallest is 4K and maximum is 32M.
- --stripe-unit
size-in-B/K/M
- Specifies the stripe unit size in B/K/M. If no suffix is given, unit B is
assumed. See striping section (below) for more details.
- --stripe-count
num
- Specifies the number of objects to stripe over before looping back to the
first object. See striping section (below) for more details.
- --snap snap
- Specifies the snapshot name for the specific operation.
- --id username
- Specifies the username (without the client. prefix) to use with the
map command.
- --keyring
filename
- Specifies a keyring file containing a secret for the specified user to use
with the map command. If not specified, the default keyring locations will
be searched.
- --keyfile
filename
- Specifies a file containing the secret key of --id user to use with
the map command. This option is overridden by --keyring if the
latter is also specified.
- --shared
lock-tag
- Option for lock add that allows multiple clients to lock the same
image if they use the same tag. The tag is an arbitrary string. This is
useful for situations where an image must be open from more than one
client at once, like during live migration of a virtual machine, or for
use underneath a clustered filesystem.
- -o krbd-options, --options
krbd-options
- Specifies which options to use when mapping or unmapping an image via the
rbd kernel driver. krbd-options is a comma-separated list of options
(similar to mount(8) mount options). See kernel rbd (krbd) options section
below for more details.
- --read-only
- Map the image read-only. Equivalent to -o ro.
- --image-feature
feature-name
- Specifies which RBD format 2 feature should be enabled when creating an
image. Multiple features can be enabled by repeating this option multiple
times. The following features are supported:
- layering: layering support
- striping: striping v2 support
- exclusive-lock: exclusive locking support
- object-map: object map support (requires exclusive-lock)
- fast-diff: fast diff calculations (requires object-map)
- deep-flatten: snapshot flatten support
- journaling: journaled IO support (requires exclusive-lock)
- --image-shared
- Specifies that the image will be used concurrently by multiple clients.
This will disable features that are dependent upon exclusive ownership of
the image.
- --whole-object
- Specifies that the diff should be limited to the extents of a full object
instead of showing intra-object deltas. When the object map feature is
enabled on an image, limiting the diff to the object extents will
dramatically improve performance since the differences can be computed by
examining the in-memory object map instead of querying RADOS for each
object within the image.
- --limit
- Specifies the limit for the number of snapshots permitted.
- ls [-l | --long]
[pool-name]
- Will list all rbd images listed in the rbd_directory object. With -l, also
show snapshots, and use longer-format output including size, parent (if
clone), format, etc.
- du [-p | --pool
pool-name] [image-spec | snap-spec]
- Will calculate the provisioned and actual disk usage of all images and
associated snapshots within the specified pool. It can also be used
against individual images and snapshots.
If the RBD fast-diff feature is not enabled on images, this
operation will require querying the OSDs for every potential object
within the image.
- info image-spec
| snap-spec
- Will dump information (such as size and object size) about a specific rbd
image. If image is a clone, information about its parent is also
displayed. If a snapshot is specified, whether it is protected is shown as
well.
- create (-s | --size
size-in-M/G/T) [--image-format format-id] [--object-size
size-in-B/K/M] [--stripe-unit size-in-B/K/M --stripe-count
num] [--image-feature feature-name]... [--image-shared]
image-spec
- Will create a new rbd image. You must also specify the size via --size.
The --stripe-unit and --stripe-count arguments are optional, but must be
used together.
- clone [--object-size
size-in-B/K/M] [--stripe-unit size-in-B/K/M --stripe-count
num] [--image-feature feature-name] [--image-shared]
parent-snap-spec child-image-spec
- Will create a clone (copy-on-write child) of the parent snapshot. Object
size will be identical to that of the parent image unless specified. Size
will be the same as the parent snapshot. The --stripe-unit and
--stripe-count arguments are optional, but must be used together.
The parent snapshot must be protected (see rbd snap
protect). This requires image format 2.
- flatten
image-spec
- If image is a clone, copy all shared blocks from the parent snapshot and
make the child independent of the parent, severing the link between parent
snap and child. The parent snapshot can be unprotected and deleted if it
has no further dependent clones.
This requires image format 2.
- children
snap-spec
- List the clones of the image at the given snapshot. This checks every
pool, and outputs the resulting poolname/imagename.
This requires image format 2.
- resize (-s | --size
size-in-M/G/T) [--allow-shrink] image-spec
- Resize rbd image. The size parameter also needs to be specified. The
--allow-shrink option lets the size be reduced.
- rm
image-spec
- Delete an rbd image (including all data blocks). If the image has
snapshots, this fails and nothing is deleted.
- export
[--export-format format (1 or 2)] (image-spec |
snap-spec) [dest-path]
- Export image to dest path (use - for stdout). The --export-format accepts
'1' or '2' currently. Format 2 allow us to export not only the content of
image, but also the snapshots and other properties, such as image_order,
features.
- import
[--export-format format (1 or 2)] [--image-format format-id]
[--object-size size-in-B/K/M] [--stripe-unit size-in-B/K/M
--stripe-count num] [--image-feature feature-name]...
[--image-shared] src-path [image-spec]
- Create a new image and imports its data from path (use - for stdin). The
import operation will try to create sparse rbd images if possible. For
import from stdin, the sparsification unit is the data block size of the
destination image (object size).
The --stripe-unit and --stripe-count arguments are optional,
but must be used together.
The --export-format accepts '1' or '2' currently. Format 2
allow us to import not only the content of image, but also the snapshots
and other properties, such as image_order, features.
- export-diff
[--from-snap snap-name] [--whole-object] (image-spec |
snap-spec) dest-path
- Export an incremental diff for an image to dest path (use - for stdout).
If an initial snapshot is specified, only changes since that snapshot are
included; otherwise, any regions of the image that contain data are
included. The end snapshot is specified using the standard --snap option
or @snap syntax (see below). The image diff format includes metadata about
image size changes, and the start and end snapshots. It efficiently
represents discarded or 'zero' regions of the image.
- merge-diff
first-diff-path second-diff-path
merged-diff-path
- Merge two continuous incremental diffs of an image into one single diff.
The first diff's end snapshot must be equal with the second diff's start
snapshot. The first diff could be - for stdin, and merged diff could be -
for stdout, which enables multiple diff files to be merged using something
like 'rbd merge-diff first second - | rbd merge-diff - third result'. Note
this command currently only support the source incremental diff with
stripe_count == 1
- import-diff
src-path image-spec
- Import an incremental diff of an image and applies it to the current
image. If the diff was generated relative to a start snapshot, we verify
that snapshot already exists before continuing. If there was an end
snapshot we verify it does not already exist before applying the changes,
and create the snapshot when we are done.
- diff [--from-snap
snap-name] [--whole-object] image-spec |
snap-spec
- Dump a list of byte extents in the image that have changed since the
specified start snapshot, or since the image was created. Each output line
includes the starting offset (in bytes), the length of the region (in
bytes), and either 'zero' or 'data' to indicate whether the region is
known to be zeros or may contain other data.
- cp (src-image-spec |
src-snap-spec) dest-image-spec
- Copy the content of a src-image into the newly created dest-image.
dest-image will have the same size, object size, and image format as
src-image.
- mv src-image-spec
dest-image-spec
- Rename an image. Note: rename across pools is not supported.
- image-meta list
image-spec
- Show metadata held on the image. The first column is the key and the
second column is the value.
- image-meta get
image-spec key
- Get metadata value with the key.
- image-meta set
image-spec key value
- Set metadata key with the value. They will displayed in image-meta
list.
- image-meta remove
image-spec key
- Remove metadata key with the value.
- object-map
rebuild image-spec | snap-spec
- Rebuild an invalid object map for the specified image. An image snapshot
can be specified to rebuild an invalid object map for a snapshot.
- snap ls
image-spec
- Dump the list of snapshots inside a specific image.
- snap create
snap-spec
- Create a new snapshot. Requires the snapshot name parameter
specified.
- snap rollback
snap-spec
- Rollback image content to snapshot. This will iterate through the entire
blocks array and update the data head content to the snapshotted
version.
- snap rm [--force]
snap-spec
- Remove the specified snapshot.
- snap purge
image-spec
- Remove all snapshots from an image.
- snap protect
snap-spec
- Protect a snapshot from deletion, so that clones can be made of it (see
rbd clone). Snapshots must be protected before clones are made;
protection implies that there exist dependent cloned children that refer
to this snapshot. rbd clone will fail on a nonprotected snapshot.
This requires image format 2.
- snap unprotect
snap-spec
- Unprotect a snapshot from deletion (undo snap protect). If cloned
children remain, snap unprotect fails. (Note that clones may exist
in different pools than the parent snapshot.)
This requires image format 2.
- snap limit set
[--limit] limit image-spec
- Set a limit for the number of snapshots allowed on an image.
- snap limit clear
image-spec
- Remove any previously set limit on the number of snapshots allowed on an
image.
- map [-o | --options
krbd-options ] [--read-only] image-spec |
snap-spec
- Map the specified image to a block device via the rbd kernel module.
- unmap [-o | --options
krbd-options ] image-spec | snap-spec |
device-path
- Unmap the block device that was mapped via the rbd kernel module.
- showmapped
- Show the rbd images that are mapped via the rbd kernel module.
- nbd map [--device
device-path] [--read-only] image-spec |
snap-spec
- Map the specified image to a block device via the rbd-nbd tool.
- nbd unmap
device-path
- Unmap the block device that was mapped via the rbd-nbd tool.
- nbd list
- Show the list of used nbd devices via the rbd-nbd tool.
- status
image-spec
- Show the status of the image, including which clients have it open.
- feature disable
image-spec feature-name...
- Disable the specified feature on the specified image. Multiple features
can be specified.
- feature
enable image-spec feature-name...
- Enable the specified feature on the specified image. Multiple features can
be specified.
- lock list
image-spec
- Show locks held on the image. The first column is the locker to use with
the lock remove command.
- lock add [--shared
lock-tag] image-spec lock-id
- Lock an image. The lock-id is an arbitrary name for the user's
convenience. By default, this is an exclusive lock, meaning it will fail
if the image is already locked. The --shared option changes this behavior.
Note that locking does not affect any operation other than adding a lock.
It does not protect an image from being deleted.
- lock remove
image-spec lock-id locker
- Release a lock on an image. The lock id and locker are as output by lock
ls.
- bench --io-type
<read | write> [--io-size size-in-B/K/M/G/T] [--io-threads
num-ios-in-flight] [--io-total size-in-B/K/M/G/T]
[--io-pattern seq | rand] image-spec
- Generate a series of IOs to the image and measure the IO throughput and
latency. If no suffix is given, unit B is assumed for both --io-size and
--io-total. Defaults are: --io-size 4096, --io-threads 16, --io-total 1G,
--io-pattern seq.
- trash ls
[pool-name]
- List all entries from trash.
- trash mv
image-spec
- Move an image to the trash. Images, even ones actively in-use by clones,
can be moved to the trash and deleted at a later time.
- trash rm
image-id
- Delete an image from trash. If image deferment time has not expired you
can not removed it unless use force. But an actively in-use by clones or
has snapshots can not be removed.
- trash restore
image-id
- Restore an image from trash.
image-spec is [pool-name]/image-name
snap-spec is [pool-name]/image-name@snap-name
The default for pool-name is "rbd". If an image
name contains a slash character ('/'), pool-name is required.
You may specify each name individually, using --pool, --image and
--snap options, but this is discouraged in favor of the above spec
syntax.
RBD images are striped over many objects, which are then stored by
the Ceph distributed object store (RADOS). As a result, read and write
requests for the image are distributed across many nodes in the cluster,
generally preventing any single node from becoming a bottleneck when
individual images get large or busy.
The striping is controlled by three parameters:
- object-size
- The size of objects we stripe over is a power of two. It will be rounded
up the nearest power of two. The default object size is 4 MB, smallest is
4K and maximum is 32M.
- stripe_unit
- Each [stripe_unit] contiguous bytes are stored adjacently in the
same object, before we move on to the next object.
- stripe_count
- After we write [stripe_unit] bytes to [stripe_count]
objects, we loop back to the initial object and write another stripe,
until the object reaches its maximum size. At that point, we move on to
the next [stripe_count] objects.
By default, [stripe_unit] is the same as the object size
and [stripe_count] is 1. Specifying a different [stripe_unit]
requires that the STRIPINGV2 feature be supported (added in Ceph v0.53) and
format 2 images be used.
Most of these options are useful mainly for debugging and
benchmarking. The default values are set in the kernel and may therefore
depend on the version of the running kernel.
Per client instance rbd map options:
- fsid=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee - FSID that should be assumed by
the client.
- ip=a.b.c.d[:p] - IP and, optionally, port the client should use.
- share - Enable sharing of client instances with other mappings
(default).
- noshare - Disable sharing of client instances with other mappings.
- crc - Enable CRC32C checksumming for data writes (default).
- nocrc - Disable CRC32C checksumming for data writes.
- cephx_require_signatures - Require cephx message signing (since 3.19,
default).
- nocephx_require_signatures - Don't require cephx message signing (since
3.19).
- tcp_nodelay - Disable Nagle's algorithm on client sockets (since 4.0,
default).
- notcp_nodelay - Enable Nagle's algorithm on client sockets (since
4.0).
- cephx_sign_messages - Enable message signing (since 4.4, default).
- nocephx_sign_messages - Disable message signing (since 4.4).
- mount_timeout=x - A timeout on various steps in rbd map and rbd
unmap sequences (default is 60 seconds). In particular, since 4.2 this
can be used to ensure that rbd unmap eventually times out when
there is no network connection to a cluster.
- osdkeepalive=x - OSD keepalive timeout (default is 5 seconds).
- osd_idle_ttl=x - OSD idle TTL (default is 60 seconds).
Per mapping (block device) rbd map options:
- rw - Map the image read-write (default).
- ro - Map the image read-only. Equivalent to --read-only.
- queue_depth=x - queue depth (since 4.2, default is 128 requests).
- lock_on_read - Acquire exclusive lock on reads, in addition to writes and
discards (since 4.9).
- exclusive - Disable automatic exclusive lock transitions (since
4.12).
rbd unmap options:
- •
- force - Force the unmapping of a block device that is open (since 4.9).
The driver will wait for running requests to complete and then unmap;
requests sent to the driver after initiating the unmap will be
failed.
To create a new rbd image that is 100 GB:
rbd create mypool/myimage --size 102400
To use a non-default object size (8 MB):
rbd create mypool/myimage --size 102400 --object-size 8M
To delete an rbd image (be careful!):
To create a new snapshot:
rbd snap create mypool/myimage@mysnap
To create a copy-on-write clone of a protected snapshot:
rbd clone mypool/myimage@mysnap otherpool/cloneimage
To see which clones of a snapshot exist:
rbd children mypool/myimage@mysnap
To delete a snapshot:
rbd snap rm mypool/myimage@mysnap
To map an image via the kernel with cephx enabled:
rbd map mypool/myimage --id admin --keyfile secretfile
To map an image via the kernel with different cluster name other
than default ceph:
rbd map mypool/myimage --cluster cluster-name
To unmap an image:
To create an image and a clone from it:
rbd import --image-format 2 image mypool/parent
rbd snap create mypool/parent@snap
rbd snap protect mypool/parent@snap
rbd clone mypool/parent@snap otherpool/child
To create an image with a smaller stripe_unit (to better
distribute small writes in some workloads):
rbd create mypool/myimage --size 102400 --stripe-unit 65536B --stripe-count 16
To change an image from one image format to another, export it and
then import it as the desired image format:
rbd export mypool/myimage@snap /tmp/img
rbd import --image-format 2 /tmp/img mypool/myimage2
To lock an image for exclusive use:
rbd lock add mypool/myimage mylockid
To release a lock:
rbd lock remove mypool/myimage mylockid client.2485
To list images from trash:
To defer delete an image (use --delay to set delay-time,
default is 0):
rbd trash mv mypool/myimage
To delete an image from trash (be careful!):
rbd trash rm mypool/myimage-id
To force delete an image from trash (be careful!):
rbd trash rm mypool/myimage-id --force
To restore an image from trash:
rbd trash restore mypool/myimage-id
To restore an image from trash and rename it:
rbd trash restore mypool/myimage-id --image mynewimage
rbd is part of Ceph, a massively scalable, open-source,
distributed storage system. Please refer to the Ceph documentation at
http://ceph.com/docs for more information.
2010-2019, Inktank Storage, Inc. and contributors. Licensed under
Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0)