| CH-IMAGE(1) | Charliecloud | CH-IMAGE(1) |
ch-image - Build and manage images; completely unprivileged
$ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT
$ ch-image [...] build-cache [...]
$ ch-image [...] delete IMAGE_REF
$ ch-image [...] gestalt [SELECTOR]
$ ch-image [...] import PATH IMAGE_REF
$ ch-image [...] list [-l] [IMAGE_REF]
$ ch-image [...] pull [...] IMAGE_REF [DEST_REF]
$ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF]
$ ch-image [...] reset
$ ch-image [...] undelete IMAGE_REF
$ ch-image { --help | --version | --dependencies }
ch-image is a tool for building and manipulating container images, but not running them (for that you want ch-run). It is completely unprivileged, with no setuid/setgid/setcap helpers. Many operations can use caching for speed. The action to take is specified by a sub-command.
Options that print brief information and then exit:
Common options placed before or after the sub-command:
Charliecloud provides the option --arch ARCH to specify the architecture for architecture-aware registry operations. The argument ARCH can be: (1) yolo, to bypass architecture-aware code and use the registry’s default architecture; (2) host, to use the host’s architecture, obtained with the equivalent of uname -m (default if --arch not specified); or (3) an architecture name. If the specified architecture is not available, the error message will list which ones are.
Notes:
Charliecloud does not have configuration files; thus, it has no separate login subcommand to store secrets. Instead, Charliecloud will prompt for a username and password when authentication is needed. Note that some repositories refer to the secret as something other than a “password”; e.g., GitLab calls it a “personal access token (PAT)”, Quay calls it an “application token”, and nVidia NGC calls it an “API token”.
For non-interactive authentication, you can use environment variables CH_IMAGE_USERNAME and CH_IMAGE_PASSWORD. Only do this if you fully understand the implications for your specific use case, because it is difficult to securely store secrets in environment variables.
By default for most subcommands, all registry access is anonymous. To instead use authenticated access for everything, specify --auth or set the environment variable $CH_IMAGE_AUTH=yes. The exception is push, which always runs in authenticated mode. Even for pulling public images, it can be useful to authenticate for registries that have per-user rate limits, such as Docker Hub. (Older versions of Charliecloud started with anonymous access, then tried to upgrade to authenticated if it seemed necessary. However, this turned out to be brittle; see issue #1318.)
The username and password are remembered for the life of the process and silently re-offered to the registry if needed. One case when this happens is on push to a private registry: many registries will first offer a read-only token when ch-image checks if something exists, then re-authenticate when upgrading the token to read-write for upload. If your site uses one-time passwords such as provided by a security device, you can specify --password-many to provide a new secret each time.
These values are not saved persistently, e.g. in a file. Note that we do use normal Python variables for this information, without pinning them into physical RAM with mlock(2) or any other special treatment, so we cannot guarantee they will never reach non-volatile storage.
Most registries use something called Bearer authentication, where the client (e.g., Charliecloud) includes a token in the headers of every HTTP request.
The authorization dance is different from the typical UNIX approach, where there is a separate login sequence before any content requests are made. The client starts by simply making the HTTP request it wants (e.g., to GET an image manifest), and if the registry doesn’t like the client’s token (or if there is no token because the client doesn’t have one yet), it replies with HTTP 401 Unauthorized, but crucially it also provides instructions in the response header on how to get a token. The client then follows those instructions, obtains a token, re-tries the request, and (hopefully) all is well. This approach also allows a client to upgrade a token if needed, e.g. when transitioning from asking if a layer exists to uploading its content.
The distinction between Charliecloud’s anonymous mode and authenticated modes is that it will only ask for anonymous tokens in anonymous mode and authenticated tokens in authenticated mode. That is, anonymous mode does involve an authentication procedure to obtain a token, but this “authentication” is done anonymously. (Yes, it’s confusing.)
Registries also often reply HTTP 401 when an image does not exist, rather than the seemingly more correct HTTP 404 Not Found. This is to avoid information leakage about the existence of images the client is not allowed to pull, and it’s why Charliecloud never says an image simply does not exist.
ch-image maintains state using normal files and directories located in its storage directory; contents include various caches and temporary images used for building.
In descending order of priority, this directory is located at:
Unlike many container implementations, there is no notion of storage drivers, graph drivers, etc., to select and/or configure.
The storage directory can reside on any single filesystem (i.e., it cannot be split across multiple filesystems). However, it contains lots of small files and metadata traffic can be intense. For example, the Charliecloud test suite uses approximately 400,000 files and directories in the storage directory as of this writing. Place it on a filesystem appropriate for this; tmpfs’es such as /var/tmp are a good choice if you have enough RAM (/tmp is not recommended because ch-run bind-mounts it into containers by default).
While you can currently poke around in the storage directory and find unpacked images runnable with ch-run, this is not a supported use case. The supported workflow uses ch-convert to obtain a packed image; see the tutorial for details.
The storage directory format changes on no particular schedule. ch-image is normally able to upgrade directories produced by a given Charliecloud version up to one year after that version’s release. Upgrades outside this window and downgrades are not supported. In these cases, ch-image will refuse to run until you delete and re-initialize the storage directory with ch-image reset.
WARNING:
Subcommands that create images, such as build and pull, can use a build cache to speed repeated operations. That is, an image is created by starting from the empty image and executing a sequence of instructions, largely Dockerfile instructions but also some others like “pull” and “import”. Some instructions are expensive to execute (e.g., RUN wget http://slow.example.com/bigfile or transferring data billed by the byte), so it’s often cheaper to retrieve their results from cache instead.
The build cache uses a relatively new Git under the hood; see the installation instructions for version requirements. Charliecloud implements workarounds for Git’s various storage limitations, so things like file metadata and Git repositories within the image should work. Important exception: No files named .git* or other Git metadata are permitted in the image’s root directory.
The cache has three modes, enabled, disabled, and a hybrid mode called rebuild where the cache is fully enabled for FROM instructions, but all other operations re-execute and re-cache their results. The purpose of rebuild is to do a clean rebuild of a Dockerfile atop a known-good base image.
Enabled mode is selected with --cache or setting $CH_IMAGE_CACHE to enabled, disabled mode with --no-cache or disabled, and rebuild mode with --rebuild or rebuild. The default mode is enabled if an appropriate Git is installed, otherwise disabled.
Other container implementations typically use build caches based on overlayfs, or fuse-overlayfs in unprivileged situations (configured via a “storage driver”). This works by creating a new tmpfs for each instruction, layered atop the previous instruction’s tmpfs using overlayfs. Each layer can then be tarred up separately to form a tar-based diff.
The Git-based cache has two advantages over the overlayfs approach. First, kernel-mode overlayfs is only available unprivileged in Linux 5.11 and higher, forcing the use of fuse-overlayfs and its accompanying FUSE overhead for unprivileged use cases. Second, Git de-duplicates and compresses files in a fairly sophisticated way across the entire build cache, not just between image states with an ancestry relationship (detailed in the next section).
A disadvantage is lowered performance in some cases. Preliminary experiments suggest this performance penalty is relatively modest, and sometimes Charliecloud is actually faster than alternatives. We have ongoing experiments to answer this performance question in more detail.
Charliecloud’s build cache takes advantage of Git’s file de-duplication features. This operates across the entire build cache, i.e., files are de-duplicated no matter where in the cache they are found or the relationship between their container images. Files are de-duplicated at different times depending on whether they are identical or merely similar.
Identical files are de-duplicated at git add time; in ch-image build terms, that’s upon committing a successful instruction. That is, it’s impossible to store two files with the same content in the build cache. If you try — say with RUN yum install -y foo in one Dockerfile and RUN yum install -y foo bar in another, which are different instructions but both install RPM foo’s files — the content is stored once and each copy gets its own metadata and a pointer to the content, much like filesystem hard links.
Similar files, however, are only de-duplicated during Git’s garbage collection process. When files are initially added to a Git repository (with git add), they are stored inside the repository as (possibly compressed) individual files, called objects in Git jargon. Upon garbage collection, which happens both automatically when certain parameters are met and explicitly with git gc, these files are archived and (re-)compressed together into a single file called a packfile. Also, existing packfiles may be re-written into the new one.
During this process, similar files are identified, and each set of similar files is stored as one base file plus diffs to recover the others. (Similarity detection seems to be based primarily on file size.) This delta process is agnostic to alignment, which is an advantage over alignment-sensitive block-level de-duplicating filesystems. Exception: “Large” files are not compressed or de-duplicated. We use the Git default threshold of 512 MiB (as of this writing).
Charliecloud runs Git garbage collection at two different times. First, a lighter-weight garbage pass runs automatically when the number of loose files (objects) grows beyond a limit. This limit is in flux as we learn more about build cache performance, but it’s quite a bit higher than the Git default. This garbage runs in the background and can continue after the build completes; you may see Git processes using a lot of CPU.
An important limitation of the automatic garbage is that large packfiles (again, this is in flux, but it’s several GiB) will not be re-packed, limiting the scope of similar file detection. To address this, a heavier garbage collection can be run manually with ch-image build-cache --gc. This will re-pack (and re-write) the entire build cache, de-duplicating all similar files. In both cases, garbage uses all available cores.
git build-cache prints the specific garbage collection parameters in use, and -v can be added for more detail.
Because Git uses content-addressed storage, upon commit, it must read in full all files modified by an instruction. This I/O cost can be a significant fraction of build time for some large images. Regular files larger than the experimental large file threshold are stored outside the Git repository, somewhat like Git Large File Storage. ch-image uses hard links to bring large files in and out of images as needed, which is a fast metadata operation that ignores file content.
Option --cache-large sets the threshold in MiB; if not set, environment variable CH_IMAGE_CACHE_LARGE is used; if that is not set either, the default value 0 indicates that no files are considered large.
There are two trade-offs. First, large files in any image with the same path, mode, size, and mtime (to nanosecond precision if possible) are considered identical, even if their content is not actually identical; e.g., touch(1) shenanigans can corrupt an image. Second, every version of a large file is stored verbatim and uncompressed (e.g., a large file with a one-byte change will be stored in full twice), and large files do not participate in the build cache’s de-duplication, so more storage space will likely be used. Unused versions are deleted by ch-image build-cache --gc.
(Note that Git has an unrelated setting called core.bigFileThreshold.)
Suppose we have this Dockerfile:
$ cat a.df FROM alpine:3.9 RUN echo foo RUN echo bar
On our first build, we get:
$ ch-image build -t foo -f a.df .
1. FROM alpine:3.9 [ ... pull chatter omitted ... ]
2. RUN echo foo copying image ... foo
3. RUN echo bar bar grown in 3 instructions: foo
Note the dot after each instruction’s line number. This means that the instruction was executed. You can also see this by the output of the two echo commands.
But on our second build, we get:
$ ch-image build -t foo -f a.df .
1* FROM alpine:3.9
2* RUN echo foo
3* RUN echo bar copying image ... grown in 3 instructions: foo
Here, instead of being executed, each instruction’s results were retrieved from cache. (Charliecloud uses lazy retrieval; nothing is actually retrieved until the end, as seen by the “copying image” message.) Cache hit for each instruction is indicated by an asterisk (*) after the line number. Even for such a small and short Dockerfile, this build is noticeably faster than the first.
We can also try a second, slightly different Dockerfile. Note that the first three instructions are the same, but the third is different:
$ cat c.df FROM alpine:3.9 RUN echo foo RUN echo qux $ ch-image build -t c -f c.df .
1* FROM alpine:3.9
2* RUN echo foo
3. RUN echo qux copying image ... qux grown in 3 instructions: c
Here, the first two instructions are hits from the first Dockerfile, but the third is a miss, so Charliecloud retrieves that state and continues building.
We can also inspect the cache:
$ ch-image build-cache --tree * (c) RUN echo qux | * (a) RUN echo bar |/ * RUN echo foo * (alpine+3.9) PULL alpine:3.9 * (HEAD -> root) ROOT named images: 4 state IDs: 5 commits: 5 files: 317 disk used: 3 MiB
Here there are four named images: a and c that we built, the base image alpine:3.9 (written as alpine+3.9 because colon is not allowed in Git branch names), and the empty base of everything root. Also note how a and c diverge after the last common instruction RUN echo foo.
Build an image from a Dockerfile and put it in the storage directory.
$ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT
Uses ch-run -w -u0 -g0 --no-passwd --unsafe to execute RUN instructions. Note that FROM implicitly pulls the base image if needed, so you may want to read about the pull subcommand below as well.
Required argument:
Options:
Note: See documentation for ch-run --bind for important caveats and gotchas.
Note: Other instructions that modify the image filesystem, e.g. COPY, can only access host files from the context directory, regardless of this option.
If no colon present in the name, append :latest.
ch-image is a fully unprivileged image builder. It does not use any setuid or setcap helper programs, and it does not use configuration files /etc/subuid or /etc/subgid. This contrasts with the “rootless” or “fakeroot” modes of some competing builders, which do require privileged supporting code or utilities.
This approach does yield some quirks. We provide built-in workarounds that should mostly work (i.e., --force), but it can be helpful to understand what is going on.
ch-image executes all instructions as the normal user who invokes it. For RUN, this is accomplished with ch-run -w --uid=0 --gid=0 (and some other arguments), i.e., your host EUID and EGID both mapped to zero inside the container, and only one UID (zero) and GID (zero) are available inside the container. Under this arrangement, processes running in the container for each RUN appear to be running as root, but many privileged system calls will fail without the workarounds described below. This affects any fully unprivileged container build, not just Charliecloud.
The most common time to see this is installing packages. For example, here is RPM failing to chown(2) a file, which makes the package update fail:
Updating : 1:dbus-1.10.24-13.el7_6.x86_64 2/4 Error unpacking rpm package 1:dbus-1.10.24-13.el7_6.x86_64 error: unpacking of archive failed on file /usr/libexec/dbus-1/dbus-daemon-launch-helper;5cffd726: cpio: chown
Cleanup : 1:dbus-libs-1.10.24-12.el7.x86_64 3/4 error: dbus-1:1.10.24-13.el7_6.x86_64: install failed
This one is (ironically) apt-get failing to drop privileges:
E: setgroups 65534 failed - setgroups (1: Operation not permitted) E: setegid 65534 failed - setegid (22: Invalid argument) E: seteuid 100 failed - seteuid (22: Invalid argument) E: setgroups 0 failed - setgroups (1: Operation not permitted)
By default, nothing is done to avoid these problems, though ch-image does try to detect if the workarounds could help. --force activates the workarounds: ch-image injects extra commands to intercept these system calls and fake a successful result, using fakeroot(1). There are three basic steps:
The details are specific to each distribution. ch-image analyzes image content (e.g., grepping /etc/debian_version) to select a configuration; see lib/fakeroot.py for details. ch-image prints exactly what it is doing.
ch-image is an independent implementation and shares no code with other Dockerfile interpreters. It uses a formal Dockerfile parsing grammar developed from the Dockerfile reference documentation and miscellaneous other sources, which you can examine in the source code.
We believe this independence is valuable for several reasons. First, it helps the community examine Dockerfile syntax and semantics critically, think rigorously about what is really needed, and build a more robust standard. Second, it yields disjoint sets of bugs (note that Podman, Buildah, and Docker all share the same Dockerfile parser). Third, because it is a much smaller code base, it illustrates how Dockerfiles work more clearly. Finally, it allows straightforward extensions if needed to support scientific computing.
ch-image tries hard to be compatible with Docker and other interpreters, though as an independent implementation, it is not bug-compatible.
The following subsections describe differences from the Dockerfile reference that we expect to be approximately permanent. For not-yet-implemented features and bugs in this area, see related issues on GitHub.
None of these are set in stone. We are very interested in feedback on our assessments and open questions. This helps us prioritize new features and revise our thinking about what is needed for HPC containers.
The context directory is bind-mounted into the build, rather than copied like Docker. Thus, the size of the context is immaterial, and the build reads directly from storage like any other local process would. However, you still can’t access anything outside the context directory.
Variable substitution happens for all instructions, not just the ones listed in the Dockerfile reference.
ARG and ENV cause cache misses upon definition, in contrast with Docker where these variables miss upon use, except for certain cache-excluded variables that never cause misses, listed below.
Note that ARG and ENV have different syntax despite very similar semantics.
ch-image passes the following proxy environment variables in to the build. Changes to these variables do not cause a cache miss. They do not require an ARG instruction, as documented in the Dockerfile reference. Unlike Docker, they are available if the same-named environment variable is defined; --build-arg is not required.
HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy NO_PROXY no_proxy
In addition to those listed in the Dockerfile reference, these environment variables are passed through in the same way:
SSH_AUTH_SOCK USER
Finally, these variables are also pre-defined but are unrelated to the host environment:
PATH=/ch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TAR_OPTIONS=--no-same-owner
Variables set with ARG are available anywhere in the Dockerfile, unlike Docker, where they only work in FROM instructions, and possibly in other ARG before the first FROM.
The FROM instruction accepts option --arg=NAME=VALUE, which serves the same purpose as the ARG instruction. It can be repeated.
Especially for people used to UNIX cp(1), the semantics of the Dockerfile COPY instruction can be confusing.
Most notably, when a source of the copy is a directory, the contents of that directory, not the directory itself, are copied. This is documented, but it’s a real gotcha because that’s not what cp(1) does, and it means that many things you can do in one cp(1) command require multiple COPY instructions.
Also, the reference documentation is incomplete. In our experience, Docker also behaves as follows; ch-image does the same in an attempt to be bug-compatible.
RUN mkdir -p /foo/bar && touch /foo/bar/baz COPY foo /foo
We expect the following differences to be permanent:
Build image bar using ./foo/bar/Dockerfile and context directory ./foo/bar:
$ ch-image build -t bar -f ./foo/bar/Dockerfile ./foo/bar [...] grown in 4 instructions: bar
Same, but infer the image name and Dockerfile from the context directory path:
$ ch-image build ./foo/bar [...] grown in 4 instructions: bar
Build using humongous vendor compilers you want to bind-mount instead of installing into the image:
$ ch-image build --bind /opt/bigvendor:/opt . $ cat Dockerfile FROM centos:7 RUN /opt/bin/cc hello.c #COPY /opt/lib/*.so /usr/local/lib # fail: COPY doesn't bind mount RUN cp /opt/lib/*.so /usr/local/lib # possible workaround RUN ldconfig
$ ch-image [...] build-cache [...]
Print basic information about the cache. If -v is given, also print some Git statistics and the Git repository configuration.
If any of the following options are given, do the corresponding operation before printing. Multiple options can be given, in which case they happen in this order.
$ ch-image [...] delete IMAGE_GLOB
Delete the image(s) described by IMAGE_GLOB from the storage directory (including all build stages).
IMAGE_GLOB can be either a plain image reference or an image reference with glob characters to match multiple images. For example, ch-image delete 'foo*' will delete all images whose names start with foo.
Importantly, this sub-command does not also remove the image from the build cache. Therefore, it can be used to reduce the size of the storage directory, trading off the time needed to retrieve an image from cache.
WARNING:
$ ch-image [...] gestalt [SELECTOR]
Provide information about the configuration and available features of ch-image. End users generally will not need this; it is intended for testing and debugging.
SELECTOR is one of:
Print information about images. If no argument given, list the images in builder storage.
$ ch-image [...] list [-l] [IMAGE_REF]
Optional argument:
List images in builder storage:
$ ch-image list alpine:3.9 (amd64) alpine:latest (amd64) debian:buster (amd64)
Print details about Debian Buster image:
$ ch-image list debian:buster details of image: debian:buster in local storage: no full remote ref: registry-1.docker.io:443/library/debian:buster available remotely: yes remote arch-aware: yes host architecture: amd64 archs available: 386 bae2738ed83
amd64 98285d32477
arm/v7 97247fd4822
arm64/v8 122a0342878
For remotely available images like Debian Buster, the associated digest is listed beside each available architecture. Importantly, this feature does not provide the hash of the local image, which is only calculated on push.
$ ch-image [...] import PATH IMAGE_REF
Copy the image at PATH into builder storage with name IMAGE_REF. PATH can be:
If the imported image contains Charliecloud metadata, that will be imported unchanged, i.e., images exported from ch-image builder storage will be functionally identical when re-imported.
NOTE:
Pull the image described by the image reference IMAGE_REF from a repository to the local filesystem.
$ ch-image [...] pull [...] IMAGE_REF [DEST_REF]
See the FAQ for the gory details on specifying image references.
Destination:
Options:
This script does a fair amount of validation and fixing of the layer tarballs before flattening in order to support unprivileged use despite image problems we frequently see in the wild. For example, device files are ignored, and file and directory permissions are increased to a minimum of rwx------ and rw------- respectively. Note, however, that symlinks pointing outside the image are permitted, because they are not resolved until runtime within a container.
The following metadata in the pulled image is retained; all other metadata is currently ignored. (If you have a need for additional metadata, please let us know!)
Note that some images (e.g., those with a “version 1 manifest”) do not contain metadata. A warning is printed in this case.
Download the Debian Buster image matching the host’s architecture and place it in the storage directory:
$ uname -m aarch32 pulling image: debian:buster requesting arch: arm64/v8 manifest list: downloading manifest: downloading config: downloading layer 1/1: c54d940: downloading flattening image layer 1/1: c54d940: listing validating tarball members resolving whiteouts layer 1/1: c54d940: extracting image arch: arm64 done
Same, specifying the architecture explicitly:
$ ch-image --arch=arm/v7 pull debian:buster pulling image: debian:buster requesting arch: arm/v7 manifest list: downloading manifest: downloading config: downloading layer 1/1: 8947560: downloading flattening image layer 1/1: 8947560: listing validating tarball members resolving whiteouts layer 1/1: 8947560: extracting image arch: arm (may not match host arm64/v8)
Push the image described by the image reference IMAGE_REF from the local filesystem to a repository.
$ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF]
See the FAQ for the gory details on specifying image references.
Destination:
Options:
Because Charliecloud is fully unprivileged, the owner and group of files in its images are not meaningful in the broader ecosystem. Thus, when pushed, everything in the image is flattened to user:group root:root. Also, setuid/setgid bits are removed, to avoid surprises if the image is pulled by a privileged container implementation.
Push a local image to the registry example.com:5000 at path /foo/bar with tag latest. Note that in this form, the local image must be named to match that remote reference.
$ ch-image push example.com:5000/foo/bar:latest pushing image: example.com:5000/foo/bar:latest layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: a1664c4: checking if already in repository layer 1/1: a1664c4: not present, uploading config: 89315a2: checking if already in repository config: 89315a2: not present, uploading manifest: uploading cleaning up done
Same, except use local image alpine:3.9. In this form, the local image name does not have to match the destination reference.
$ ch-image push alpine:3.9 example.com:5000/foo/bar:latest pushing image: alpine:3.9 destination: example.com:5000/foo/bar:latest layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: a1664c4: checking if already in repository layer 1/1: a1664c4: not present, uploading config: 89315a2: checking if already in repository config: 89315a2: not present, uploading manifest: uploading cleaning up done
Same, except use unpacked image located at /var/tmp/image rather than an image in ch-image storage. (Also, the sole layer is already present in the remote registry, so we don’t upload it again.)
$ ch-image push --image /var/tmp/image example.com:5000/foo/bar:latest pushing image: example.com:5000/foo/bar:latest image path: /var/tmp/image layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: 892e38d: checking if already in repository layer 1/1: 892e38d: already present config: 546f447: checking if already in repository config: 546f447: not present, uploading manifest: uploading cleaning up done
$ ch-image [...] reset
Delete all images and cache from ch-image builder storage.
$ ch-image [...] undelete IMAGE_REF
If IMAGE_REF has been deleted but is in the build cache, recover it from the cache. Only available when the cache is enabled, and will not overwrite IMAGE_REF if it exists.
Also sets verbose mode if not already set (equivalent to --verbose).
If Charliecloud was obtained from your Linux distribution, use your distribution’s bug reporting procedures.
Otherwise, report bugs to: https://github.com/hpc/charliecloud/issues
Full documentation at: <https://hpc.github.io/charliecloud>
2014–2022, Triad National Security, LLC and others
| 2023-01-29 12:36 UTC | 0.31 |