| nbdkit-curl-plugin(1) | NBDKIT | nbdkit-curl-plugin(1) |
nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
nbdkit -r curl [url=]http://example.com/disk.img
"nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content served over HTTP, FTP, and more, into a Network Block Device. It uses a library called libcurl (also known as cURL) to read data from URLs. The exact list of protocols that libcurl can handle depends on how it was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and more (see: "curl -V").
Note: This plugin supports writes. However for HTTP, you may not want nbdkit to issue PUT requests to the remote server (which probably doesn't understand them). To force nbdkit to use a readonly connection, pass the -r flag.
Although this plugin can access SFTP (ie. SSH) servers, it is much better to use nbdkit-ssh-plugin(1). This plugin can be used to access "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
nbdkit -r curl http://example.com/disk.img
serves the remote disk image as NBD on TCP port 10809 (to control ports and protocols used to serve NBD see nbdkit(1)).
Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
Pass empty string in order to not use the default certificate store that libcurl is compiled with.
Set CA certificates directory location for libcurl. See CURLOPT_CAPATH(3) for more information.
Set a cookie in the request header when connecting to the remote server.
A typical example is:
cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
This option can be used at most once. It only works for HTTP and HTTPS transports. To set multiple cookies you must concatenate them yourself, eg:
cookie='name1=content1; name2=content2'
See CURLOPT_COOKIE(3) for more information about this. The format is quite strict and must consist of "key=value", each cookie separated by exactly "; " (semicolon and space).
If the cookie is used for authentication then passing it on the command line is not secure on shared machines. Use the alternate "+FILENAME" syntax to pass it in a file, "-" to read the cookie interactively, or "-FD" to read it from a file descriptor.
Enable cookie processing (eg. when following redirects), starting with an empty list of cookies. This is equivalent to calling CURLOPT_COOKIEFILE(3) with an empty string.
Enable cookie processing (eg. when following redirects), prepopulating cookies from the given file. The file can contain cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3). Cookies sent by the server are not saved when nbdkit exits.
Enable cookie processing (eg. when following redirects), prepopulating cookies from the given file, and writing server cookies back to the file when the NBD handle is closed. The file can contain cookies in any format supported by curl, see CURLOPT_COOKIEJAR(3).
There is some advice on the internet telling you to set this to /dev/null, but you should not do this because it can corrupt /dev/null. If you don't want a cookiejar, omit this option. If you want to enable cookie processing without updating a permanent cookiejar, use the "cookiefile=" option instead.
Run "SCRIPT" (a command or shell script fragment) to generate the HTTP/HTTPS cookies. "cookie-script" cannot be used with "cookie". See "HEADER AND COOKIE SCRIPTS" below.
Follow redirects from the server. The default is true (to follow redirects), but you can set this to false to prevent this.
For HTTP/HTTPS, send a custom header, or remove a header that curl has added. To add a custom header:
header='X-My-Name: John Doe'
To remove a header that curl has added, add the header followed by a colon and no value:
header='User-Agent:'
To add a custom header that has no value, you have to use a semicolon instead of colon. This adds an "X-Empty:" header with no value:
header='X-Empty;'
See CURLOPT_HTTPHEADER(3). You can use this option multiple times in order to add several headers. Note this sends the header in all requests, even when following a redirect, which can cause headers (eg. containing sensitive authorization information) to be sent to hosts other than the one originally requested.
Run "SCRIPT" (a command or shell script fragment) to generate the HTTP/HTTPS headers. "header-script" cannot be used with "header". See "HEADER AND COOKIE SCRIPTS" below.
Note that passing this on the command line is not secure on shared machines.
Limit the protocols that are allowed in the URL. Use this option for extra security if the URL comes from an untrusted source and you want to avoid security isues in the more obscure protocols that curl supports. (See qemu CVE-2013-0249 for an example of a security bug introduced by allowing unrestricted protocols).
For example if you only intend HTTP and HTTPS URLs to be used, then add this parameter: "protocols=http,https"
This restriction also applies if the plugin follows a redirect to another protocol (eg. you start with an "https://" URL which the server redirects to "ftp://"). To prevent redirects altogether see the "followlocation" parameter.
The value of this parameter is a comma-separated list of protocols. The following protocols are known: dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp, rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs, smtp, smtps, telnet, tftp.
The default is to allow any protocol.
Set the proxy. See CURLOPT_PROXY(3).
Set the proxy username and password.
Enable TCP keepalives.
Enable Nagle’s algorithm. Small writes on the network socket will not be sent immediately but will be held in a local buffer and coalesced if possible. This is more efficient for the network but can cause increased latency.
The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is disabled for better latency at the expense of network efficiency.
Instead of using a TCP connection, connect to the server over the named Unix domain socket. See CURLOPT_UNIX_SOCKET_PATH(3).
This parameter is required.
"url=" is a magic config key and may be omitted in most cases. See "Magic parameters" in nbdkit(1).
Send user-agent header when using HTTP or HTTPS. The default is no user-agent header.
While the "header" and "cookie" parameters can be used to specify static headers and cookies which are used in every HTTP/HTTPS request, the alternate "header-script" and "cookie-script" parameters can be used to run an external script or program to generate headers and/or cookies. This is particularly useful to access services which require an authorization token. In addition the "header-script-renew" and "cookie-script-renew" parameters allow you to renew the authorization token by rerunning the script periodically.
"header-script" is incompatible with "header", and "cookie-script" is incompatible with "cookie".
The header script should print zero or more HTTP headers, each line of output in the same format as the "header" parameter. The headers printed by the script are passed to CURLOPT_HTTPHEADER(3).
In the following example, an imaginary web service requires authentication using a token fetched from a separate login server. The token expires after 60 seconds, so we also tell the plugin that it must renew the token (by re-running the script) if more than 50 seconds have elapsed since the last request:
nbdkit curl https://service.example.com/disk.img \
header-script='
printf "Authorization: Bearer "
curl -s -X POST https://auth.example.com/login |
jq -r .token
' \
header-script-renew=50
The cookie script should print a single line in the same format as the "cookie" parameter. This is passed to CURLOPT_COOKIE(3).
Within the "header-script" and "cookie-script" the following shell variables are available:
VMware ESXi’s web server can expose both VMDK and raw format disk images, but requires you to log in using HTTP Basic Authentication. While you can use the "user" and "password" parameters to send HTTP Basic Authentication headers in every request, tests have shown that it is faster to accept the cookie which the server returns and send that instead. (It is not clear why it is faster, but one theory is that VMware has to do a more expensive username and password check each time.)
The web server can be accessed as below. Since the cookie expires after a certain period of time, we use "cookie-script-renew", and because the server uses a self-signed certificate we must use --insecure and "sslverify=false".
SERVER=esx.example.com
DCPATH=data
DS=datastore1
GUEST=guest-name
URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
nbdkit curl "$URL" \
cookie-script='
curl --head -s --insecure -u root:password "$url" |
sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
' \
cookie-script-renew=500 \
sslverify=false
Accessing objects like container layers from Docker Hub requires that you first fetch an authorization token, even for anonymous access. These tokens expire after about 5 minutes (300 seconds) so must be periodically renewed.
You will need this authorization script (/tmp/auth.sh):
#!/bin/sh -
IMAGE=library/fedora
curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
jq -r .token
You will also need this script to get the blobSum of the layer (/tmp/blobsum.sh):
#!/bin/sh -
TOKEN=`/tmp/auth.sh`
IMAGE=library/fedora
curl -s -X GET -H "Authorization: Bearer $TOKEN" \
"https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
jq -r '.fsLayers[0].blobSum'
Both scripts must be executable, and both can be run on their own to check they are working. To run nbdkit:
IMAGE=library/fedora
BLOBSUM=`/tmp/blobsum.sh`
URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
nbdkit curl "$URL" \
header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
header-script-renew=200 \
--filter=gzip
Note that this exposes a tar file over NBD. See also nbdkit-tar-filter(1).
Use "nbdkit --dump-config" to find the location of $plugindir.
"nbdkit-curl-plugin" first appeared in nbdkit 1.2.
curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3), CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3), CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3), CURLOPT_SSL_CIPHER_LIST(3), CURLOPT_SSLVERSION(3), CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3), CURLOPT_TLS13_CIPHERS(3), CURLOPT_URL(3), CURLOPT_UNIX_SOCKET_PATH(3), CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3), nbdkit(1), nbdkit-extentlist-filter(1), nbdkit-file-plugin(1), nbdkit-retry-filter(1), nbdkit-retry-request-filter(1), nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3), http://curl.haxx.se, https://curl.se/docs/ssl-ciphers.html
Richard W.M. Jones
Parts derived from Alexander Graf's "QEMU Block driver for CURL images".
Copyright (C) 2014-2020 Red Hat Inc.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
| 2023-01-04 | nbdkit-1.32.5 |