Manfest-driven Image Building

The imaginator may be used to build complete images using image manifsts which describe the contents of images.

Image Building Concepts

Image streams are a progression of images (i.e. v0, v1, v2 and so on) over time. Every image stream has an associated image manifest which describes the components desired. In general, image manifests are stored in a Git repository.

Images are built by taking a SourceImage, adding files and packages and running scripts at various stages during the process. Any image that is built by the imaginator can be used as the SourceImage for another image, leading to a tree of images. The root of the tree is a bootstrap image which is generated by the imaginator. These bootstrap images are built using vendor-supplied scripts such as debootstrap and yumbootstrap.

If a SourceImage is missing (or too old (default: 1 hour)), it is automatically built. In a typical configuration, some commonly used images are automatically and periodically rebuilt. This supports fast building of application images which automatically have the latest security patches from the OS vendor.

Built images are stored in an imageserver which de-duplicates common files and efficiently replicates the data to multiple regions if desired. Machines may be live-patched with these images using the dominator, or tools such as the ami-publisher may be used to publish image artefacts across multiple AWS accounts and regions in a couple of minutes to create new virtual machines.

Manifest Components

The image manifest has multiple components. These are processed and combined in the following order to build the image contents.

manifest file

This is a required file, containing a JSON encoded object with the following fields:

Other fields may be present and they will be ignored by the imaginator. This is typically used to store other image metadata such as the tags to apply when creating an AMI (Amazon Machine Image).

files directory tree

If present, any files and symbolic links in this directory tree will be copied verbatim into the image (prior to installing packages), preserving the directory structure.

pre-install-scripts directory

An optional directory containing scripts to run prior to installing packages. These are processed in lexical order. The scripts are run in a contained environment where the root directory is the root directory of the image being built.

package-list file

An optional file containing a newline-separated list of packages to install. The contents of the files directory tree should contain any package repositories.

post-install-files directory tree

If present, any files and symbolic links in this directory tree will be copied verbatim into the image, preserving the directory structure.

scripts directory

An optional directory containing scripts to run. These are processed in lexical order. The scripts are run in a contained environment where the root directory is the root directory of the image being built.

post-scripts-files directory tree

If present, any files and symbolic links in this directory tree will be copied verbatim into the image (after the scripts are run), preserving the directory structure.

post-cleanup-scripts directory

An optional directory containing scripts to run after cleanup operations (such as after cleaning up downloaded packages and bind mounts). These are processed in lexical order. The scripts are run in a contained environment where the root directory is the root directory of the image being built.

computed-files file

An optional file listing the computed files. This is relevant only for images which will be lived patched onto machines with the dominator. Each line of the file must contain the following fields:

An alternative format file computed-files.json is supported, where the format is JSON data which must contain an array of objects with the above fields.

computed-files.add file

This is similar to the computed-files file, except that the list of computed files are added to the list of computed files in the SourceImage, thus inheriting and (if not empty) extending the list of computed files. This must not be present if the computed-files file is present.

An alternative format file computed-files.add.json is supported, which must contain JSON data.

filter file

An optional file containing a newline-separated list of regular expressions matching files which should not be updated on target machines. This is relevant only for images which will be lived patched onto machines with the dominator. This must not be present if the filter.add file is present.

filter.add file

This is similar to the filter file, except that the filter expressions are added to the filter of the SourceImage, thus inheriting and (if not empty) extending the filter. This must not be present if the filter file is present.

tags.json file

An optional JSON encoded file containing key:value tags to add to the image. Variable expansion is performed on the values.

triggers file

An optional JSON encoded file listing the triggers (services which should be restarted when certain files change when live patching machines). This is relevant only for images which will be lived patched onto machines with the dominator. The JSON data must contain an array of objects with the following fields:

This must not be present if the triggers.add file is present.

triggers.add file

This is similar to the triggers file, except that the triggers are added (merged) to the triggers of the SourceImage, thus inheriting and (if not []) extending the triggers. This must not be present if the triggers file is present.

tests directory

An optional directory containing test scripts to run. These are copied into the /tests directory tree in the image, merging with tests from the SourceImage. The tests are run concurrently after the image content is built. If any test fails or exceeds the 10 second timeout, the image is not uploaded and the build fails. The scripts are run in a contained environment where the root directory is the root directory of the image that was built.