Image management

The images located in /trinity/images (default of trix_images) are file/directory based and represent how the node would look like after installation. The files and directories can be edited or chrooted into (using lchroot, recommended method).

Image contents

The image directory is a simple file/directory based structure which is easily modified. All the files (including symlinks) placed here end up in the image and ultimately on the node.

# ls -l /trinity/images/compute
total 72
lrwxrwxrwx.   1 root root     7 Oct 11  2021 bin -> usr/bin
dr-xr-xr-x.   5 root root  4096 Oct 18 13:39 boot
drwxr-xr-x.   2 root root  4096 Oct 18 13:38 dev
drwxr-xr-x. 120 root root 12288 Oct 18 13:48 etc
drwxr-xr-x.   4 root root  4096 Oct 18 13:42 home
lrwxrwxrwx.   1 root root     7 Oct 11  2021 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Oct 11  2021 lib64 -> usr/lib64
drwxr-xr-x.   2 root root  4096 Oct 11  2021 media
drwxr-xr-x.   2 root root  4096 Oct 11  2021 mnt
drwxr-xr-x.   5 root root  4096 Oct 18 13:48 opt
dr-xr-xr-x.   2 root root  4096 Oct 18 13:36 proc
dr-xr-x---.   5 root root  4096 Oct 18 13:43 root
drwxr-xr-x.  25 root root  4096 Oct 18 13:48 run
lrwxrwxrwx.   1 root root     8 Oct 11  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root root  4096 Oct 11  2021 srv
dr-xr-xr-x.   2 root root  4096 Oct 18 13:36 sys
drwxrwxrwt.   2 root root  4096 Oct 18 13:49 tmp
drwxr-xr-x.   5 root root  4096 Oct 18 13:40 trinity
drwxr-xr-x.  12 root root  4096 Oct 18 13:36 usr
drwxr-xr-x.  20 root root  4096 Oct 18 13:38 var

Creating an image

To create a new image, run the compute-redhat.yml playbook. This will create (default name compute) an image in /trinity/images.

Modifying and updating an image

An image which has already been created can be changed in three ways. They differ mainly in where the record of the change ends up, rather than in what they are able to do: lchroot is the quickest route to a change, while a playbook can be reviewed, kept under version control and replayed, which is usually what a production image needs.

Whichever is used, the image has to be packed afterwards before the nodes will see the change.

Image manipulation is in general done on the active controller. This matters in a high-availability setup, where that is the primary: the image trees live there, and the TrinityX image playbooks refuse to run on a secondary.

Directly in the image with lchroot

The image can be easily modified, but files may end up accidentally on the controller node itself. A safer way is to do a chroot using lchroot {name}.

To get the available images, please refer to luna osimage list.

# lchroot compute
IMAGE PATH: /trinity/images/compute
chroot [root@compute /]$

In the image, commands such as dnf update can be applied, or software can be installed. Note that many software can be installed using the modules environment, which does not require modifying the images and is in general a better approach.

Once the modifications have been completed, the image needs to be packed.

An isolated playbook

Nothing about an image requires the TrinityX inventory. An image is a directory tree on the controller, so any playbook which can reach it over a chroot connection can manage it — including one kept in your own repository, with its own inventory, entirely separate from TrinityX.

The connection plugin is community.general.chroot, from the community.general collection. It is already present on a controller, as TrinityX itself uses it. An inventory of your own then needs nothing more than the image path:

[images]
my-node-image ansible_connection=community.general.chroot ansible_host=/trinity/images/compute

The path is better derived than written down, so that renaming or rebuilding an image does not leave a stale inventory behind. Luna will report it:

# luna osimage list --raw | jq -r '.[] | select(.name=="compute") | .path'
/trinity/images/compute

A playbook then looks like any other. This one installs a set of site packages, writes a limits file, and enables a service so that it runs once a node boots the image:

---
- name: Site customisations for the node image
  hosts: images
  tasks:
    - name: Install site packages
      package:
        name:
          - htop
          - sysstat
          - tcpdump
        state: present

    - name: Raise the open file limit for jobs
      copy:
        dest: /etc/security/limits.d/90-site.conf
        content: |
          * soft nofile 65536
          * hard nofile 65536
        owner: root
        group: root
        mode: '0644'

    - name: Enable the metrics collector at boot
      systemd:
        name: sysstat
        enabled: true

# ansible-playbook -i inventory.ini site-image.yml
# luna osimage pack compute

Kept in a repository of your own and versioned there, a playbook like this is the whole record of what an image carries beyond the TrinityX defaults, and it can be replayed against a rebuilt image or a second cluster.

A playbook in this form needs no preparation on the controller, because nothing in it is a TrinityX role: installing packages and writing configuration need nothing from TrinityX. The trade-off is the mirror image — a playbook outside the repository cannot reuse the TrinityX roles either.

It does still have to run on the controller which holds the image, since the image is reached by local path, and the image still has to be packed before the nodes will see the change.

A playbook alongside TrinityX (advanced)

TrinityX builds its own images with playbooks run from its repository, and the same mechanism is available to you. Every osimage known to Luna is available to Ansible as a host. The inventory script site/dynamic_hosts calls luna osimage list and presents each image as {name}.osimages.luna in the group osimages_luna, with the connection type set to chroot and the image path as the target. A play addressed at that host therefore runs inside the image, not on the controller.

Playbooks must be run from the site directory of the TrinityX repository, as the inventory script is wired in through site/ansible.cfg and is not picked up from anywhere else. Run from any other directory, the image is simply not in the inventory: Ansible reports that it could not match the host pattern, skips the play and still exits successfully. Nothing is changed and nothing fails, so it is worth checking that a play reports the tasks you expected rather than trusting its exit code.

A complete playbook which installs packages into the compute image and packs it:

---
- hosts: localhost
  tasks:
    - name: Set the image to work on
      set_fact:
        image_name: compute
      tags: always

- hosts: controllers
  vars:
    image_name: "{{ hostvars['localhost']['image_name'] }}"
  pre_tasks:
    - name: Prepare the facts the TrinityX roles expect
      include_role:
        name: trinity/init
      tags: always
  roles:
    - role: ansible/write_facts

- hosts: "{{ hostvars['localhost']['image_name'] }}.osimages.luna"
  roles:
    - role: ansible/read_facts
  tasks:
    - name: Install site monitoring tools
      package:
        name:
          - htop
          - sysstat
        state: present

- hosts: controllers
  roles:
    - role: luna/osimage-pack
      vars:
        params:
          image_name: "{{ hostvars['localhost']['image_name'] }}"

# ansible-playbook site-monitoring.yml

The first play sets the image name, the second prepares the facts the TrinityX roles expect, the third runs inside the image, and the fourth packs it. Your own tasks go in the third play, as do any TrinityX roles you want to reuse; ansible/read_facts has to remain its first role, because those facts are handed over in a file which it consumes.

The name is set once and referred to through hostvars afterwards, which is how the TrinityX playbooks do it. The name appears both in a host pattern and in the roles, and a name changed in one place but not the other addresses an image which does not exist — which, as above, is not an error.

Addressing the group osimages_luna rather than a single image applies a play to every image on the controller. That is convenient for a change which genuinely belongs everywhere, and worth avoiding otherwise: images serving different node types rarely want the same packages.

Points to be aware of

• The image has to exist in Luna before it can be addressed, as the inventory is generated from luna osimage list.
• If a play mounts anything inside the image, unmount it in the same play. A bind mount left inside an image tree is a live view of the controller's own filesystems, under a path that looks disposable.
• If the change installed a new kernel, set it in Luna with luna osimage change --kernelversion {version} {name} before packing — see Updating the kernel in an image.
• An image only reaches the nodes once it has been packed and the nodes rebooted.
• For a large change, clone the image first and work on the clone. The original then remains a bootable rollback.
• Modules run under the image's own Python interpreter, not the controller's. A task that works against the controller can fail inside an image whose Python or library set differs, and any Python library a module needs has to be present inside the image.
• Downloads are the most common case of the above. Fetching a file on the controller with delegate_to: localhost and copying it into the image is more predictable than fetching it from within the image.
• A service in an image can be enabled, but not started. There is no running systemd inside an image, so systemctl refuses runtime commands there — and a task asking for state: started is reported as successful without anything having been started. Use enabled alone, and let the node start the service when it boots.

The TrinityX playbooks define the default state of an image: re-running compute-*.yml restores whatever those roles manage and overwrites conflicting manual changes. Keeping your own changes in your own playbooks means the two can be replayed in order — TrinityX first, yours on top — and the image reproduced from scratch whenever it is needed.

Packing an image

The image which resides on the filesystem needs to be packed into a tarball and made ready to be distributed. This is done via luna osimage pack {name}

# luna osimage pack compute
osimage pack for compute queued
2023-10-19 14:44:12 :: queued pack osimage compute with queue_id 11
2023-10-19 14:44:12 :: packing osimage compute
2023-10-19 14:44:50 :: finished packing osimage compute
2023-10-19 14:44:50 :: building osimage compute
2023-10-19 14:46:20 :: finished building osimage compute
2023-10-19 14:46:20 :: creating provisioning for osimage compute
2023-10-19 14:46:25 :: created torrent provisioning for osimage compute
2023-10-19 14:46:26 :: created http provisioning for osimage compute
[========] Image compute Packed.

Cloning an image

There may be a need to branch off an image to install new software in or test updates.

In the following example a gpu is created based off the already existing compute image.

# luna osimage  clone compute gpu
2023-10-19 15:40:54 :: queued clone osimage compute->gpu with queue_id 20
2023-10-19 15:40:54 :: copying osimage compute->gpu
2023-10-19 15:41:06 :: finished copying osimage
2023-10-19 15:41:06 :: packing osimage gpu
2023-10-19 15:41:50 :: finished packing osimage gpu
2023-10-19 15:41:50 :: building osimage gpu
2023-10-19 15:43:11 :: finished building osimage gpu
2023-10-19 15:43:11 :: creating provisioning for osimage gpu
2023-10-19 15:43:15 :: created torrent provisioning for osimage gpu
2023-10-19 15:43:16 :: created http provisioning for osimage gpu
[========] OS Image gpu Cloned.

Now the image is created and it can be modified and assigned to a group or node.

# luna osimage list
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                      << Osimage >>                                                                                     |
+---+---------+------------------------------+---------------------------------------------------------+----------------------------+-------------------------+--------------+-----------+
| # |   name  |        kernelversion         |                        kernelfile                       |         imagefile          |           path          | distribution | osrelease |
+---+---------+------------------------------+---------------------------------------------------------+----------------------------+-------------------------+--------------+-----------+
| 1 | compute | 4.18.0-477.27.1.el8_8.x86_64 | compute-1697726652-vmlinuz-4.18.0-477.27.1.el8_8.x86_64 | compute-1697726690.tar.bz2 | /trinity/images/compute |    redhat    |    None   |
| 2 |   gpu   | 4.18.0-477.27.1.el8_8.x86_64 |   gpu-1697730066-vmlinuz-4.18.0-477.27.1.el8_8.x86_64   |   gpu-1697730110.tar.bz2   |   /trinity/images/gpu   |    redhat    |    None   |
+---+---------+------------------------------+---------------------------------------------------------+----------------------------+-------------------------+--------------+-----------+

Changing image for a group or node

The osimage configured with a group (default) or node (by override). It is possible to assign a different image than is configured in the group, making it very easy to quickly test one image without accidentally assigning the image to nodes who are coincidentally booting at the same time.

luna group show {name}:

# luna group show compute
+-------------------------------------------------------------------------------+
|                                Group => compute                               |
+---------------------+---------------------------------------------------------+
| name                | compute                                                 |
| domain              | cluster                                                 |
| osimage             | compute                                                 |
| osimagetag          | default                                                 |
[...]
[...]
| comment             | None                                                    |
+---------------------+---------------------------------------------------------+

To view a node, use luna node show {name}. When the osimage name is followed by another name in parentheses, this means the osimage is configured by the group, in this example the group is also named compute.

# luna node show node001
+----------------------------------------------------------------------------------------+
|                                    Node => node001                                     |
+---------------------+------------------------------------------------------------------+
| name                | node001                                                          |
| hostname            | node001.cluster                                                  |
| group               | compute                                                          |
| osimage             | compute (compute)                                                |
[...]
[...]
+---------------------+------------------------------------------------------------------+

To change the image compute for gpu for the group compute:

# luna group change --osimage gpu compute
Group, compute updated.

To change the image compute for gpu for node001 (override)

# luna node change --osimage gpu node001
Node, node001 updated.

The lack of parenthesis points out that it is an override

# luna node show node001
+----------------------------------------------------------------------------------------+
|                                    Node => node001                                     |
+---------------------+------------------------------------------------------------------+
| name                | node001                                                          |
| hostname            | node001.cluster                                                  |
| group               | compute                                                          |
| osimage             | gpu                                                              |
| osimagetag          | default                                                          |
[...]
[...]
+---------------------+------------------------------------------------------------------+

To get the node to default back to the group configuration, change it to empty:

# luna node change --osimage ""  node001
Node, node001 updated.
# luna node show node001
+----------------------------------------------------------------------------------------+
|                                    Node => node001                                     |
+---------------------+------------------------------------------------------------------+
| name                | node001                                                          |
| hostname            | node001.cluster                                                  |
| group               | compute                                                          |
| osimage             | compute (compute)                                                |
| osimagetag          | default (compute)                                                |
[...]
[...]
+---------------------+------------------------------------------------------------------+

Tailoring an image post-installation

By default, the hostname and network interface configuration is tailored by the Luna node installer. There may be a need to do this for additional configuration files (i.e. hostnames in files), the postscript can be used for this. Note the tailoring is done on the group and node level and this sticks with that, regardless of the image. Any files the postscript is modifying need to be present in those images as well. See also node / group configuration