| CH-IMAGE(1) | Charliecloud | CH-IMAGE(1) |
ch-image - Build and manage images; completely unprivileged
$ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT
$ ch-image [...] list
$ ch-image [...] pull [...] IMAGE_REF [IMAGE_DIR]
$ ch-image [...] storage-path
$ ch-image { --help | --version | --dependencies }
ch-image is a tool for building and manipulating container images, but not running them (for that you want ch-run). It is completely unprivileged, with no setuid/setgid/setcap helpers.
Options that print brief information and then exit:
Common options placed before the sub-command:
ch-image maintains state using normal files and directories, including unpacked container images, located in its storage directory. There is no notion of storage drivers, graph drivers, etc., to select and/or configure. In descending order of priority, this directory is located at:
The storage directory can reside on any filesystem. However, it contains lots of small files and metadata traffic can be intense. For example, the Charliecloud test suite uses approximately 400,000 files and directories in the storage directory as of this writing. Place it on a filesystem appropriate for this; tmpfs’es such as /var/tmp are a good choice if you have enough RAM (/tmp is not recommended because ch-run bind-mounts it into containers by default).
While you can currently poke around in the storage directory and find unpacked images runnable with ch-run, this is not a supported use case. The supported workflow uses ch-builder2tar or ch-builder2squash to obtain a packed image; see the tutorial for details.
WARNING:
Build an image from a Dockerfile and put it in the storage directory. Use ch-run(1) to execute RUN instructions.
Required argument:
Options:
Note: This applies only to RUN instructions. Other instructions that modify the image filesystem, e.g. COPY, can only access host files from the context directory.
ch-image is a fully unprivileged image builder. It does not use any setuid or setcap helper programs, and it does not use configuration files /etc/subuid or /etc/subgid. This contrasts with the “rootless” or “fakeroot” modes of some competing builders, which do require privileged supporting code or utilities.
This approach does yield some quirks. We provide built-in workarounds that should mostly work (i.e., --force), but it can be helpful to understand what is going on.
ch-image executes all instructions as the normal user who invokes it. For RUN, this is accomplished with ch-run -w --uid=0 --gid=0 (and some other arguments), i.e., your host EUID and EGID both mapped to zero inside the container, and only one UID (zero) and GID (zero) are available inside the container. Under this arrangement, processes running in the container for each RUN appear to be running as root, but many privileged system calls will fail without the workarounds described below. This affects any fully unprivileged container build, not just Charliecloud.
The most common time to see this is installing packages. For example, here is RPM failing to chown(2) a file, which makes the package update fail:
Updating : 1:dbus-1.10.24-13.el7_6.x86_64 2/4 Error unpacking rpm package 1:dbus-1.10.24-13.el7_6.x86_64 error: unpacking of archive failed on file /usr/libexec/dbus-1/dbus-daemon-launch-helper;5cffd726: cpio: chown
Cleanup : 1:dbus-libs-1.10.24-12.el7.x86_64 3/4 error: dbus-1:1.10.24-13.el7_6.x86_64: install failed
This one is (ironically) apt-get failing to drop privileges:
E: setgroups 65534 failed - setgroups (1: Operation not permitted) E: setegid 65534 failed - setegid (22: Invalid argument) E: seteuid 100 failed - seteuid (22: Invalid argument) E: setgroups 0 failed - setgroups (1: Operation not permitted)
By default, nothing is done to avoid these problems, though ch-image does try to detect if the workarounds could help. --force activates the workarounds: ch-image injects extra commands to intercept these system calls and fake a successful result, using fakeroot(1). There are three basic steps:
The details are specific to each distribution. ch-image analyzes image content (e.g., grepping /etc/debian_version) to select a configuration; see lib/fakeroot.py for details. ch-image prints exactly what it is doing.
Print the storage directory path and exit.
Pull the image described by the image reference IMAGE_REF from a repository by HTTPS. See the FAQ for the gory details on specifying image references.
This script does a fair amount of validation and fixing of the layer tarballs before flattening in order to support unprivileged use despite image problems we frequently see in the wild. For example, device files are ignored, and file and directory permissions are increased to a minimum of rwx------ and rw------- respectively. Note, however, that symlinks pointing outside the image are permitted, because they are not resolved until runtime within a container.
Destination argument:
Options:
ch-image is an independent implementation and shares no code with other Dockerfile interpreters. It uses a formal Dockerfile parsing grammar developed from the Dockerfile reference documentation and miscellaneous other sources, which you can examine in the source code.
We believe this independence is valuable for several reasons. First, it helps the community examine Dockerfile syntax and semantics critically, think rigorously about what is really needed, and build a more robust standard. Second, it yields disjoint sets of bugs (note that Podman, Buildah, and Docker all share the same Dockerfile parser). Third, because it is a much smaller code base, it illustrates how Dockerfiles work more clearly. Finally, it allows straightforward extensions if needed to support scientific computing.
ch-image tries hard to be compatible with Docker and other interpreters, though as an independent implementation, it is not bug-compatible.
This section describes differences from the Dockerfile reference that we expect to be approximately permanent. For an overview of features we have not yet implemented and our plans, see our road map on GitHub. Plain old bugs are in our GitHub issues.
None of these are set in stone. We are very interested in feedback on our assessments and open questions. This helps us prioritize new features and revise our thinking about what is needed for HPC containers.
The context directory is bind-mounted into the build, rather than copied like Docker. Thus, the size of the context is immaterial, and the build reads directly from storage like any other local process would. However, you still can’t access anything outside the context directory.
ch-image can authenticate using one-time passwords, e.g. those provided by a security token. Unlike docker login, it does not assume passwords are persistent.
Variable substitution happens for all instructions, not just the ones listed in the Dockerfile reference.
ARG and ENV cause cache misses upon definition, in contrast with Docker where these variables miss upon use, except for certain cache-excluded variables that never cause misses, listed below.
Like Docker, ch-image pre-defines the following proxy variables, which do not require an ARG instruction. However, they are available if the same-named environment variable is defined; --build-arg is not required. Changes to these variables do not cause a cache miss.
HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy NO_PROXY no_proxy
The following variables are also pre-defined:
PATH=/ch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TAR_OPTIONS=--no-same-owner
Note that ARG and ENV have different syntax despite very similar semantics.
Especially for people used to UNIX cp(1), the semantics of the Dockerfile COPY instruction can be confusing.
Most notably, when a source of the copy is a directory, the contents of that directory, not the directory itself, are copied. This is documented, but it’s a real gotcha because that’s not what cp(1) does, and it means that many things you can do in one cp(1) command require multiple COPY instructions.
Also, the reference documentation is incomplete. In our experience, Docker also behaves as follows; ch-image does the same in an attempt to be bug-compatible.
RUN mkdir -p /foo/bar && touch /foo/bar/baz COPY foo /foo
We expect the following differences to be permanent:
CH_LOG_FILE
Also sets verbose mode if not already set (equivalent to --verbose).
CH_LOG_FESTOON
Build image bar using ./foo/bar/Dockerfile and context directory ./foo/bar:
$ ch-image build -t bar -f ./foo/bar/Dockerfile ./foo/bar [...] grown in 4 instructions: bar
Same, but infer the image name and Dockerfile from the context directory path:
$ ch-image build ./foo/bar [...] grown in 4 instructions: bar
Build using humongous vendor compilers you want to bind-mount instead of installing into a layer:
$ ch-image build --bind /opt/bigvendor:/opt . $ cat Dockerfile FROM centos:7 RUN /opt/bin/cc hello.c #COPY /opt/lib/*.so /usr/local/lib # fail: COPY doesn't bind mount RUN cp /opt/lib/*.so /usr/local/lib # possible workaround RUN ldconfig
Download the Debian Buster image and place it in the storage directory:
$ ch-image pull debian:buster pulling image: debian:buster manifest: downloading layer 1/1: d6ff36c: downloading layer 1/1: d6ff36c: listing validating tarball members resolving whiteouts flattening image layer 1/1: d6ff36c: extracting done
Same, except place the image in /tmp/buster:
$ ch-image pull debian:buster /tmp/buster [...] $ ls /tmp/buster bin dev home lib64 mnt proc run srv tmp var boot etc lib media opt root sbin sys usr
If Charliecloud was obtained from your Linux distribution, use your distribution’s bug reporting procedures.
Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
Full documentation at: <https://hpc.github.io/charliecloud>
2014–2020, Triad National Security, LLC
| 2020-12-20 14:44 Coordinated Universal Time | 0.21 |