nbdcopy(1) | LIBNBD | nbdcopy(1) |
nbdcopy - copy to and from an NBD server
nbdcopy [--allocated] [-C N|--connections=N] [--destination-is-zero|--target-is-zero] [--flush] [--no-extents] [-p|--progress|--progress=FD] [-R N|--requests=N] [-S N|--sparse=N] [--synchronous] [-T N|--threads=N] SOURCE DESTINATION SOURCE, DESTINATION := - | FILE | DEVICE | NBD-URI | [ CMD ARGS ... ] nbdcopy --help nbdcopy -V|--version
This copies everything from the NBD server at "example.com" to a local file called local.img.
This streams the first part of the disk on the NBD server at "example.com" into the file(1) command. Note here that "-" means to stream to stdout (and therefore into the pipe to the file command).
nbdinfo(1) is another way to detect the content on an NBD server.
Copy the full local hard disk "/dev/sdX" to the NBD server listening on the Unix domain socket /tmp/unixsock. Because of the -p option this will print a progress bar.
Copy a full disk from one NBD server to another.
Run qemu-nbd(8) as a subprocess to open disk.qcow2, which is then copied to disk.raw. Note "--" to prevent qemu-nbd flags from being interpreted as nbdcopy flags.
Concatenate two raw-format disk images into the qcow2 file output.qcow2. The output file has to be precreated.
nbdcopy copies to and from an NBD server. It can upload a local file to an NBD server, or download the contents of an NBD server to a local file, device or stdin/stdout. It can also copy between NBD servers.
The local file can be a file, a block device (eg. "/dev/cdrom"), or "-" which means stream in from stdin or stream out to stdout.
The NBD server can be specified using an NBD URI (like "nbd://localhost"). The NBD server can be local or remote, and encryption can be used if libnbd was built with encryption support. Alternately you can use square brackets around a qemu-nbd(8) or nbdkit(1) command to run the NBD server as a subprocess of nbdcopy.
For more complex copying operations including converting between disk formats use "qemu-img convert", see qemu-img(1).
To get nbdcopy to write the progress bar to a file you can use the following shell commands:
exec 3>/tmp/progress nbdcopy --progress=3 ... exec 3>&-
Note --threads=0 means autodetect and --threads=1 means use a single thread.
The three options --connections, --threads and --requests are related and control the amount of parallelism available. The defaults should ensure a reasonable amount of parallelism if possible and you don’t need to adjust them, but this section tries to describe what is going on.
Firstly if either side of the copy is streaming to or from stdio, a pipe, or a socket, or if you use the --synchronous option, then nbdcopy works in synchronous mode with no parallelism, and nothing else in this section applies.
The --connections=N option controls NBD multi-conn (see "Multi-conn" in libnbd(3)), opening up to N connections to the NBD server (or to both NBD servers if copying between NBD servers). This defaults to 4. The NBD servers must support and advertise multi-conn. For nbdkit(1) availability of multi-conn can depend on the plugin. You can use nbdinfo(1) to find out if a particular NBD server is advertising multi-conn. If the NBD server doesn’t advertise multi-conn then only one connection will be opened regardless of the --connections flag.
When copying between two NBD servers, the number of connections is limited to the minimum multi-conn supported on both sides. For the purposes of this calculation, you can consider local files and block devices as supporting infinite multi-conn.
When you run an NBD server as a subprocess (using the "[ ... ]" syntax) multi-conn cannot be used.
The --threads=N option allows nbdcopy to start up to N threads (defaulting to the number of cores). However nbdcopy cannot use more threads than the number of NBD connections.
The --requests=N option controls the maximum number of requests in flight on each NBD connection. This enables the NBD server to process requests in parallel even when multi-conn isn’t available or when using a single thread. The default is chosen to allow a reasonable amount of parallelism without using too much memory.
Because of this parallelism, nbdcopy does not read or write blocks in order. If for some reason you require that blocks are copied in strict order then you must use --synchronous.
Instead of connecting to an already running server using an NBD URI, you can run an NBD server as a subprocess using:
nbdcopy -- [ CMD ARGS ... ] ...
This requires the server to support systemd socket activation, which both qemu-nbd(8) and nbdkit(1) support (see also nbd_connect_systemd_socket_activation(3)).
"[" and "]" must be separate command line parameters. You will usually need to use "--" to stop nbdcopy from misinterpreting NBD server flags as nbdcopy flags. Both the source and destination may be subprocesses. nbdcopy cleans up the subprocess on exit.
Some examples follow.
In this example, qemu-nbd(8) is run as a subprocess. The subprocess opens disk.qcow2 and exposes it as NBD to nbdcopy. nbdcopy streams this to stdout ("-") into the pipe which is read by hexdump(1).
Two subprocesses are created, qemu-nbd(8) as the source and nbdkit(1) as the destination. The qcow2 file is converted to raw and stored temporarily in the RAM disk (nbdkit-memory-plugin(1)).
When nbdcopy exits both servers are killed and the RAM disk goes away, so this command has no overall effect, but is useful for testing.
You can use nbdcopy, cmp(1) and bash(1) process substitution to compare the content of two NBD servers for equality:
cmp <( nbdcopy nbd://server1 - ) <( nbdcopy nbd://server2 - )
Note this tests that the content is logically equal. It does not compare the NBD metadata such as sparseness (see nbdinfo(1) --map option). Thus for example a run of allocated zeroes in one server will match a hole in the other.
libnbd(3), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdkit(1), qemu-img(1).
Richard W.M. Jones
Copyright (C) 2020 Red Hat Inc.
This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
2021-02-09 | libnbd-1.6.1 |