Release Notes

Release numbering

Major releases, such as 16 and 17, can change the base of TrinityX: the supported operating systems and the core components.

Dot releases, such as 15.3 and 16.1, add features on the same base.

Updates, marked u and a number, such as 16u1, 15.3u1 and 14.4u4, contain fixes for the release they follow and no new features.

Current 16.1 Release

TrinityX 16.1 was released on November 5 2026. It is a release centred on Luna 2.2, with many fixes across the stack.

Below a brief list of the main changes, enhancements and features for TrinityX 16.1:

Luna 2.2

  • Access control: users and user groups, authentication against local accounts, PAM, or LDAP and Active Directory, and permissions on every object
  • Redfish support: BIOS settings, firmware updates, power management and BMC configuration during node installation
  • Zero-touch provisioning (ZTP) of switches, including NVLink switches
  • Mixed-architecture support: OS images for ARM and x86 packed and served from the same controller
  • Declarative node provisioning with a new node installer: install modes, disk layouts with mirrors and RAID, UEFI and legacy BIOS, secure erase and refresh of installed nodes
  • Network mounts and profiles managed in Luna
  • Numerous fixes and hardening across the Luna daemon and CLI

Generic

  • Monitoring and AlertX enhancements, including InfiniBand and power usage panels
  • Open OnDemand portal: a Nodeboot app that shows each node's boot progress, and a rewritten rack view that handles large clusters
  • An updated all.yml migration tool that keeps the cluster's own settings

Details

Luna 2.2 — access control

Covering TRIX-1993

  • Luna has users and user groups, with roles on group membership, managed from the Luna CLI.
  • Authentication runs through a chain of plugins: local accounts, PAM, and LDAP or Active Directory.
  • Every object carries an owner, user groups and access rights, enforced on every request. Lists and views show only what a user may see, and objects can be handed over between users and user groups.
  • An audit trail records who changed which object, and the Open OnDemand portal acts on behalf of the logged-in user.
  • Hardened the credentials used during node provisioning; node images and client configuration no longer carry controller credentials. Improved enforcement of access permissions, and restricted cluster export and import to administrators.

Luna 2.2 — Redfish and BMC management

Covering TRIX-1173, TRIX-1803, TRIX-1842, TRIX-1954, TRIX-1955, TRIX-2049, TRIX-2162

  • BIOS settings are read and changed through Redfish, shown with readable names. A golden BIOS configuration can be cloned, adjusted, assigned to nodes or groups and pushed.
  • BIOS and BMC firmware updates from a firmware catalogue, preserving the BMC configuration.
  • Power management of nodes through Redfish, including the next boot into the BIOS.
  • The BMC is configured for Redfish during node installation, and out-of-band inventory complements the node inventory.
  • IPMI: Cipher 17 support, BMCs configured through DHCP, and BMC plugins selected by manufacturer. An unreachable BMC no longer slows down the Luna daemon.

Luna 2.2 — zero-touch provisioning

Covering TRIX-1880, TRIX-1908, TRIX-1975

  • Switches, including NVLink switches, are provisioned through ZTP and managed with the Luna CLI.

Luna 2.2 — OS images and architectures

Covering TRIX-1549, TRIX-1867, TRIX-2172, TRIX-2210, TRIX-2247, TRIX-2253

  • OS images for ARM and x86 can be packed and served from the same controller.
  • OS images are worked on and packed in an isolated environment, locked while in use, with mounts cleaned up afterwards.

Luna 2.2 — node installer, disks and mounts

Covering TRIX-1920, TRIX-1326, TRIX-1628, TRIX-1674, TRIX-2125, TRIX-2159, TRIX-2164, TRIX-2166, TRIX-2169, TRIX-2196, TRIX-2197, TRIX-2198, TRIX-2200, TRIX-2202, TRIX-2203, TRIX-2216, TRIX-2217, TRIX-2218, TRIX-2224, TRIX-2225, TRIX-2228, TRIX-2231, TRIX-2232, TRIX-2234, TRIX-2251, TRIX-2263, TRIX-2264, TRIX-2265, TRIX-2266, TRIX-2267, TRIX-2268, TRIX-2269, TRIX-2272, TRIX-2277, TRIX-2282, TRIX-2287

  • A declarative model describes how a node is provisioned, from the install mode to its disks and mounts, and the node installer in the Luna client carries it out.
  • The node installer is driven by two settings: the install mode (auto, sync, full, local, memboot, sanitize, legacy) and a disk layout. A new cluster starts in memboot.
  • Disk layouts describe an OS set and data sets, with mirrors, stripes and RAID, persistent data sets and a choice of bootloader. Layouts the node cannot build are refused when they are stored.
  • UEFI and legacy BIOS nodes are installed alike; when GRUB is chosen, the node boots from disk first.
  • auto and sync refresh a healthy installation in place, with an OS image filter for the paths that must be kept. sanitize securely erases a node's disks and verifies the result.
  • Install failures are shown on the node console with their reason; a failure that cannot resolve itself stops the install instead of retrying.
  • Network mounts (NFS and others) are managed per node and group, and Luna renders the exports and fstab.

Luna 2.2 — profiles and secrets

Covering TRIX-1968, TRIX-1969, TRIX-1209, TRIX-1713, TRIX-2115

  • A profile is a named set of files with an optional service action (start, restart, reload or stop), assigned to groups and nodes; a group profile and a node profile stack on the same node.
  • Profiles are delivered live to running nodes, with the owner and mode of each file set. luna profile status shows which nodes are in sync, and unassigning a profile removes its files and restores any file it replaced.
  • Secrets carry file ownership and permissions, and can apply to all groups.

Luna 2.2 — networking

Covering TRIX-2012, TRIX-2097, TRIX-2127, TRIX-2128, TRIX-2238, TRIX-2249, TRIX-2299, TRIX-2301

  • Kea DHCP supports relay link selection (option 82.5, RFC 3527), and other devices can have round-robin DNS names.
  • Fixes for bonding on Ubuntu, a bond as VLAN parent, and IPv6 addressing and DNS records. DNS and DHCP no longer miss a change made while a reload is running.

Luna 2.2 — high availability

Covering TRIX-1959, TRIX-1686, TRIX-2018, TRIX-2150, TRIX-2178, TRIX-2186, TRIX-2193

  • The standby controller pulls new OS image files by itself after every pack or kernel change, including a kernel change made on the master, so both controllers boot nodes from the same image.
  • Image files are verified with a checksum when the standby pulls them and fetched again if they do not match, and a controller notices an image file it is missing.
  • An image operation refused on the standby is reported as failed, and nothing is left queued on either controller.
  • Replication starts reliably when a controller's network address comes up late after boot, and a controller only reports itself in sync once it has caught up.
  • Secrets compare correctly between the controllers, so they are no longer synchronised again without need, and nodes are configured with both controllers of the pair.

Luna 2.2 — daemon and CLI

Covering TRIX-1854, TRIX-1868, TRIX-1930, TRIX-1959, TRIX-1988, TRIX-2058, TRIX-2062, TRIX-2063, TRIX-2163, TRIX-2212, TRIX-2229

  • Numerous security and robustness improvements in the Luna daemon, and an error from the daemon now carries its reason to the CLI.
  • The Luna CLI lists objects that deviate from their parent and shows nodes, groups and the cluster in clear blocks.
  • Improved OS image packing and image synchronisation.
  • A network type is either ethernet or infiniband; other values are refused. Pre, part and post scripts and comments with non-ASCII characters, such as curly quotes pasted from a word processor, are refused instead of breaking luna show.

Distribution support

Covering TRIX-1916

  • Support for further Ubuntu releases.

Monitoring and AlertX

Covering TRIX-1653, TRIX-1172, TRIX-1288, TRIX-1753, TRIX-1888, TRIX-2383

  • Grafana panels for InfiniBand statistics and power usage per node and group.
  • AlertX rules for InfiniBand nodes and groups, nodes marked when a health check triggers, a view of historical incidents, and a faster AlertX.
  • The AlertX drainer drains nodes only when enable_alertx_draining: true is set; otherwise it logs which nodes it would have drained and why (see Draining nodes). Clusters that rely on automatic draining set it in group_vars/all.yml before re-running the playbooks. Alertmanager no longer sends alerts to the drainer's former webhook or to the Sensu bridge webhook on port 5555. Where nothing listened on them, they filled its log with failed deliveries.

Open OnDemand portal

Covering TRIX-1885, TRIX-1026, TRIX-2040, TRIX-2048, TRIX-2100, TRIX-2273, TRIX-2309, TRIX-751, TRIX-1952

  • The Nodeboot app shows each node's boot and install progress with its console; luna boot status shows the same progress on the command line. Serial console redirection for nodes is available, off by default.
  • The rack view is rewritten and handles large clusters, and the file permissions of the portal configuration are tightened.
  • The TrinityX apps come as one package with their frontends built in, and Open OnDemand installs with the Node.js and Ruby versions that match the operating system's minor release.

Installer and TrinityX Slurm

Covering TRIX-1905, TRIX-2130, TRIX-2095, TRIX-2102, TRIX-2213, TRIX-1956, TRIX-1974, TRIX-2052, TRIX-2334

  • The updated all.yml migration tool keeps the cluster's own settings, including the network settings, credentials and LDAP backend, and takes the release and its yml check from all.yml.example.
  • TrinityX Slurm: token-based access to the Slurm REST API, optional packages for further Slurm features, and corrected timeouts that could drain complete clusters.
  • prepare.sh asks before upgrading all packages, the kernel included, and remembers the answer.
  • obol takes the user and group OU from obol.conf, with --user-ou, --group-ou and --config on the command line; contributed on GitHub by fabio1299.
  • The EasyBuild account is configurable with easybuild_user, so an existing site account can be used.
  • External shares in shared_fs_disks, such as NFS, are mounted on the controllers again, not only on the nodes.

Re-running the playbooks

Covering TRIX-1983, TRIX-2327, TRIX-1977

  • A re-run leaves the configuration the site has made as it is: sshd_config, sssd.conf, corosync.conf, ood_portal.yml, the pacemaker resources and cluster settings, and a DRBD shared disk once it is set up. Each kept item is reported in the run, and -e override=yes rewrites it; see the installation notes.
  • -e override=no has the same effect as leaving the option out, except for a DRBD shared disk.
  • --tags admin-change only sets the admin group: on the files that carry it and in the sssd.conf login filter; see tags.
  • Prometheus keeps at most 33 GB of data by default, set with prometheus_storage_retention_size, and a re-run on the primary controller no longer disables stonith. sshd reads the drop-in files in /etc/ssh/sshd_config.d.

Bug fixes & hardening

  • Numerous fixes across the controller playbook, the 389 Directory Server installation, K3s, image building, package downloads, the Luna CLI and the Open OnDemand apps.

Previous 16u1 release

TrinityX 16u1 is a maintenance update for TrinityX 16, focused on operational fixes, improved provisioning and networking behaviour, HA reliability, and portal usability.

  • Improved high-availability reliability for 389 Directory Server after a standby controller takes over.
  • Fixed node cloning in the TrinityX 16 portal.
  • Expanded installer, image-builder and TrinityX Slurm packaging support.
  • Improved AlertX and InfiniBand monitoring usability.

Details

High availability and directory services

Covering TRIX-1915

  • Fixed an issue where 389 Directory Server could fail to start when a rebooted standby controller became active.

Open OnDemand and Luna GUI

Covering TRIX-1913

  • Fixed node cloning in the Luna node GUI App.

Configuration defaults and compatibility

Covering TRIX-580, TRIX-1630, TRIX-1900

  • Node Health Check (NHC) is disabled by default and requires the user to enable the functionality.
  • Fixed handling for unsupported characters in Ansible inventory group names.
  • Fixed Docker image handling when image references contain slashes.

Monitoring and AlertX

Covering TRIX-1631, TRIX-1654

  • AlertX CLI now supports node-name expansion.
  • Improved InfiniBand metrics association with nodes and reviewed the related exporter behaviour.

16 release

TrinityX 16 was released on June 18 2026. Below a brief list of the main changes, enhancements and features that's included:

  • Open OnDemand portal & Luna GUI applications redesign
  • Support for airgapped installations
  • Monitoring Enhancements with emphasis on TSDB size reduction and scaling optimization
  • Luna Improvements and Enhancements:
    • Introduction of automatic plugin and template reloading and synchronisation
    • Diskfull and Raid1 plugin support for Ubuntu
    • Network config enhancements, including improved Ubuntu bonding
    • Introduction of easy toggle between iPXE kernels
    • Core Luna improvement on startup thread spawning
  • Introduction of TrinityX Slurm to offer cross-distribution up-to-date Slurm support
  • Deprecation of OpenHPC stack
  • Option to enable default closed TCP ports for Grafana (3000), Open OnDemand (8080), Prometheus-server (9090) and Prometheus-Alertmanager (9093)
  • DS389 set as the default ldap backend for EL9, e.g. Rocky 9, Alma Linux 9 and RedHat Enterprise 9

Details

Open OnDemand portal & Luna GUI redesign

Covering TRIX-1576, TRIX-1600, TRIX-1670, TRIX-1704, TRIX-1705, TRIX-1710, TRIX-1742, TRIX-1743, TRIX-1744, TRIX-1755, TRIX-1765, TRIX-1777, TRIX-1780, TRIX-1805, TRIX-838, TRIX-1642, TRIX-1875, TRIX-1879

  • New dashboard layout and branding overhaul, with the legacy "monitor" and "service" tiles removed.
  • A suite of VueJS Open OnDemand applications rebuilt or added — Cluster, Node, Group, OS Image, Switch, BMC Setup, DNS and Other Device — alongside a refactor of the ood-apps role that stays output-compatible across OSes and airgap.
  • A new Rack GUI application (aligned with the AlertX GUI) and SSL certificate-key support for OOD.

Airgapped installations

Covering TRIX-1625, TRIX-1772, TRIX-1773, TRIX-1877, TRIX-838

  • End-to-end airgap support: per-distribution local repository handling (ELRepo, RHEL, EPEL, pacemaker, pip), and airgap-ready OOD core, OOD apps and Rack.
  • Luna and DHCP coexistence under airgap (letting luna2-daemon own DHCP to avoid race conditions) and a dedicated cv-slurm repository configuration for airgapped installs.

TrinityX Slurm & OpenHPC deprecation

Covering TRIX-1609, TRIX-1608, TRIX-1644, TRIX-1613, TRIX-1645

  • Introduction of TrinityX Slurm (cv-slurm / trx-slurm) with its own repositories, slimmed package sets for images, relocated pdsh (cv-pdsh) role, and a systemd drop-in for the slurmd override.
  • Slurm resource constraints by default: pam_slurm_adopt (cgroup/ssh-session confinement) and memory constraints. OpenHPC is deprecated in general.

Monitoring & AlertX

Covering TRIX-1287, TRIX-1284, TRIX-1721, TRIX-1692, TRIX-1889

  • New alertx_history daemon persisting alert state changes to MariaDB, surfaced through a dedicated Grafana "Alert History" dashboard, with HA-aware history handling.
  • New ECC alert rules and optimised scrape intervals. Prometheus, Alertmanager, OOD and Grafana service ports are now closed by default (manual override to open).

Distribution support: EL10, DS389 & OOD 4.1

Covering TRIX-1518, TRIX-1561, TRIX-1822, TRIX-1861, TRIX-983, TRIX-1859, TRIX-1604

  • OOD bumped to 4.1.1 (reverted to 4.0 on EL8, which 4.1 does not support), EasyBuild EL10 workarounds, and oddjob-mkhomedir for EL10.
  • 389-DS (DS389) set as the default LDAP backend with hardened certificate handling and EL9 stale-shm cleanup; SSSD responder sockets masked on Ubuntu images; intermediate resolv.conf and corrected BIND paths.

Install hardening, migration & credential safety

Covering TRIX-1681, TRIX-1580, TRIX-1581, TRIX-1623, TRIX-1657, TRIX-1658, TRIX-1663, TRIX-1809, TRIX-1865, TRIX-1839, TRIX-1872, TRIX-1884, TRIX-1890, TRIX-1870

  • Migration utility for all.yml, configurable temp directory, newer ansible-core compatibility, faster repeat package installs, and a controller auto-upgrade block.
  • Safety gates that halt installs on default Luna or BMC passwords, an unusual umask, or unexpected controller hostnames (all overridable). A Luna secrets-at-rest encryption flag and the broad Luna security audit (TRIX-1872) landed here, along with capability-based selection of the MySQL Ansible module.

Luna 2.x — networking, DHCP & iPXE

Covering TRIX-1769, TRIX-1841, TRIX-1587, TRIX-1678, TRIX-1802, TRIX-1832, TRIX-1859, TRIX-1768

  • Kea DHCP(v4+v6) maturity: a dhcp_nodes_only mode, fixed/shared-network and pool-in-zone handling, and boolean-normalised network updates, plus group/node interface-name change support.
  • Configurable iPXE kernel selection (TRIX-1841): an easy per-target toggle between iPXE kernels, backed by a dedicated Kea iPXE-kernel boot class.
  • DNSSEC and modern-vs-legacy BIND handling, and improved Ubuntu bonding so a bootif interface can safely be used as a bond member (TRIX-1832).

Luna 2.x — plugins, images, BMC & queue

Covering TRIX-1784, TRIX-1003, TRIX-1695, TRIX-1252, TRIX-1813, TRIX-1882, TRIX-1814, TRIX-1864, TRIX-1603, TRIX-1872, TRIX-1030

  • Automatic plugin and template reloading and synchronisation: plugins auto-invalidate on file changes and boot plugins sync between HA controllers. Startup overhaul, with the work queue draining and requeuing on restart and workers scaled to available cores.
  • Diskfull and Raid1 boot plugins extended beyond EL to Ubuntu and openSUSE (TRIX-1252): proper GRUB/shim and EFI bootloader installation (including for non-Ubuntu images), an arm64 removable-boot fallback, a fallback second EFI disk, and clean handling of pre-existing/foreign RAID members.
  • BMC improvements including Dell racadm plugins and controller renumbering on HA network changes; safer remote image clone/rollback; and the TRIX-1872 daemon security audit (JWT expiry, credential redaction, optional secret encryption). luna2-cli gains lrack/discover bash completions (TRIX-1030).

Bug fixes & hardening

  • Numerous fixes across image builds (Ubuntu osgrab/gpg, arch-dependent luna2-client packages), CI pipeline stabilisation and template pinning, OOD asset deployment, DRBD startup retries, and assorted Luna logging and shutdown robustness improvements.

15.3u1 release

TrinityX 15.3u1 bug-fix release was published on February 20 2026. In principle no features were added but the following bugs were resolved:

  • Compute and K3s image building fix for HA setups
  • 389-DS installation time mismatched package for EL9
  • Enhancements for RHEL based installations image creation and subscriptions
  • Enhancements for playbook runs for workload manager based settings

Details

Image building

Covering TRIX-1592, TRIX-1340, TRIX-1460

  • Reworked the Kubernetes/K3s playbooks so the image role is driven from the main image-create role, fixing compute and K3s image building on HA setups, with more robust openSUSE bootstrap and kernel-version detection.

RHEL / EL installations

Covering TRIX-1591, TRIX-1611

  • Improvements to the RHEL "extend" tasks for image creation and subscriptions, and a fix so the 389-DS install completes on EL9 (mismatched package).

Playbook runs & workload manager

Covering TRIX-1593

  • Logrotate handling is now delegated to each individual role rather than centralized, and a when constraint was added so the Slurm parts only run where applicable.

Luna 2.x

Covering TRIX-1104

  • luna2-daemon: database calls now correctly handle non-epoch date strings, and lexport data import was aligned with the current database tables.

Bug fixes & hardening

  • Ubuntu package-mirror changes: switched the ARM mirror to ports.ubuntu.com, and replaced the NLUUG mirror with the U-Twente mirror for more reliable package retrieval.

15.3 release

TrinityX 15.3 was released on February 4 2026. A Brief list of included enhancements and features:

  • BETA EL10 support, including RedHat Enterprise 10, Rocky Linux 10 and Alma Linux 10
  • Adding Kea DHCP(v4+v6) support, introducing a slow transition from depcrecated ISC dhcpd
  • Adding 389-DS support to replace deprecated openldap
  • Open OnDemand version 4 for images/nodes
  • OpenHPC version 4 for EL10
  • Any-Image to support luna osgrabbed images
  • Override to restore components to defaults
  • Bug fixes for HA Easybuild paths, bcrypt and others

Details

EL10 beta & distribution convergence

Covering TRIX-1453, TRIX-1340, TRIX-1338, TRIX-1009, TRIX-1155

  • Introduced BETA support for the EL10 family (RHEL/Rocky/Alma 10), with dedicated EL10 variable files, OpenHPC 4, EL10-aware DRBD/shared-FS handling and an EL10-aware OOD stack.
  • Converged the per-distribution logic so images build the same way across EL, Ubuntu and openSUSE: image creation was made controller-independent, and the package layer (dnf/apt/zypper) was merged behind a single package abstraction with shared, cross-distribution roles. The module environment switched to Lmod, and password hashing moved off bcrypt to a passlib-based approach.

Kea DHCP (v4 + v6)

Covering TRIX-1394, TRIX-1424

  • Added a Kea role as a Luna dependency, selectable via a single luna2_use_kea flag, preferred on RL10+. Luna gained Kea DHCPv4 and DHCPv6 templates, TSIG-key support (for both Kea and bind), DHCP-DDNS logging and interface discovery for Kea DHCPv6, paving the road for alternative DHCP back-ends alongside the existing bind/named path.

389 Directory Server backend

Covering TRIX-1435

  • Added a 389-DS role with full backup and HA support, a directory back-end selector, SELinux contexts, and obol scripting for DS389. The sssd, ssl-cert/CA-certificate and authselect paths were overhauled so authentication works consistently across all supported distributions, and images were taught which directory back-end is in use.

Open OnDemand 4 & native image import

Covering TRIX-1281, TRIX-1453, TRIX-1338, TRIX-1249

  • Open OnDemand updated to version 4 for images and nodes, with configurable Shell-app inactivity and maximum-duration timeouts and DS389 awareness. Added native TrinityX import of osgrabbed ("Any-Image") images, with a distribution fallback (EL only for now) when the image distribution is not supplied.

Override / restore-to-defaults

Covering TRIX-1109

  • Added an override mechanism to reinitialise components back to a known-good default state, covering Slurm, openldap/Luna re-init and SSL-cert regeneration.

Luna 2.x

Covering TRIX-1424, TRIX-1426, TRIX-1523, TRIX-1524, TRIX-1425, TRIX-1467, TRIX-905, TRIX-1419, TRIX-1478, TRIX-1477

  • luna2-daemon: Kea DHCPv4/v6 templating and TSIG support (TRIX-1424); full by-UUID fstab support for the diskfull and raid1 boot scripts (TRIX-1426); reduced startup overhead by trimming unnecessary named/DHCP config rewrites and network-plugin loads (TRIX-1523, TRIX-1524); correct IPv4/IPv6 handling when switching node-interface networks and VLAN-on-bond templates (TRIX-1425, TRIX-1467); input validation for provisioning method, scripts and roles (TRIX-905); a bulk hook plugin that reconfigures Slurm (TRIX-1419); and a fix for clearing VLAN/bond settings (TRIX-1478).
  • luna2-cli: fixes to correctly clear bond mode/slaves and VLAN/parent on nodes and groups (TRIX-1477).

Bug fixes & hardening

Covering TRIX-1448, TRIX-1513, TRIX-1401, TRIX-1259, TRIX-680, TRIX-1358, TRIX-1357, TRIX-1337

  • Fixed the EasyBuild mountpoint/install on passive HA controllers and the OHPC/EasyBuild mount order (TRIX-1448, TRIX-1513); renamed slurm-gres.conf to gres.conf (TRIX-1401); scoped the InfiniBand exporter to controllers/IB-connected servers only and added drain/undrain sudoers for the IB analyzer (TRIX-1259, TRIX-680); fixed trix_root and slurm.conf parameters (TRIX-1358, TRIX-1357) and a PJS GPU-usage panel bug (TRIX-1337). Numerous fixes were folded in across iPXE downloads (now using a CV-hosted location), elrepo base URLs, MOTD/logo refreshes and Ansible boolean-compatibility cleanups.

15.2 release

TrinityX 15.2 was released on October 29 2025. A brief list of enhancements and features:

  • Enhanced Slurm Graphical Configurator
  • Introduction of AlextX rule Wizard
  • MTU support in Luna as well DNS zone delegation
  • Luna daemon exporter and AlertX inclusion for monitoring functionality, node and osimage state.
  • mountoptions for shared filesystems used in HA.
  • EAR readiness
  • Native Easybuild support
  • Quite a handful of bug fixes and enhancements for e.g. Luna

Details

Slurm & workload management

Covering TRIX-1232, TRIX-771, TRIX-971, TRIX-1119, TRIX-1235, TRIX-1268, TRIX-700, TRIX-1017, TRIX-959

  • A restructured TUI/menu layout for the Slurm graphical configurator, and an AlertX NHC drainer that is now toggleable based on the selected workload manager.

Monitoring, AlertX & GPU

Covering TRIX-978, TRIX-1137, TRIX-1138, TRIX-977, TRIX-997, TRIX-622, TRIX-998, TRIX-1002

  • A new Prometheus luna2-daemon exporter with generic section-based monitoring endpoints, and a broadened set of node/osimage/HA monitoring states.
  • Extended GPU coverage: an AMD GPU exporter and recording rules supporting NVIDIA, AMD and mixed-mode environments.

EasyBuild & OpenHPC

Covering TRIX-853, TRIX-1142, TRIX-954, TRIX-984

  • Native EasyBuild support via a dedicated role, with EasyBuild and OpenHPC relocated under /opt and /trinity/shared and their NFS exports/mounts reworked so controllers and images stay consistent. OpenHPC bumped to the latest release.

HA & shared filesystems

Covering TRIX-1050, TRIX-1250, TRIX-937, TRIX-1103, TRIX-1101, TRIX-1271

  • Configurable mount options for the shared filesystem (with format verification), RHEL repositories for HA packages, and MariaDB fixes.

Luna 2.x (daemon & CLI)

Covering TRIX-1081, TRIX-1091, TRIX-1261, TRIX-1120, TRIX-709, TRIX-888, TRIX-1074, TRIX-1015, TRIX-1205, TRIX-1025, TRIX-1092, TRIX-990, TRIX-989, TRIX-686, TRIX-1079

  • Networking: native interface and group MTU support, non-authoritative networks, per-network reverse-PTR zones, a controller_network flag for controller IPs outside the default network, and IP-range validation during bootstrap.
  • Image/node handling: default grab-exclude paths, bulk node/group plugin calls via a master-only queue, by-path/by-id disk support for the diskfull and raid1 scripts, and IB-card serial tracking.
  • luna2-cli: dynamic argcomplete tab completion (TRIX-686) and a node interface-change fix.

Bug fixes & hardening

  • Numerous fixes across Grafana CLI handling, openSUSE family detection, GPU dashboard units, SSL toggles for OOD/Grafana, smarter logrotation, and floating-IP DNS behaviour.

15.1 release

TrinityX 15.1 was released on June 5 2025. A brief summary what's included:

  • Login image support. The login image will provide for Open OnDemand functionality and Shell access for users
  • OpenSuse image support. A playbook is now provided to build an opensuse image
  • External Floating IP support for HA setups where access to the active controller is needed
  • Improved AlertX functionality:
    • Silencing Alerts is now supported
    • A more scalable approach in Overview for larger clusters
    • Rules for HA related degradations
  • Introduction of python3 libraries to render configurations for slurm, genders and more:
    • Luna based configurations are now extendable by pre-configured defaults
    • Support for GRES resources, based on a dedicated gres.conf file

Details

Login image & Open OnDemand

Covering TRIX-749, TRIX-862

  • New login-image build playbook bundling Open OnDemand and ood-apps, reworked to function on login nodes without trix_login_fqdn, plus an OOD "system status" application.

openSUSE & multi-distro images

Covering TRIX-174

  • New openSUSE base-image playbook (openSUSE Leap 15.6, with SLE-specific sssd paths and symlinks) and a dedicated syslog-ng role for the SUSE family.
  • ARM and Docker-based image support, an openSUSE k3s playbook, and luna2-client RPM packaging.

ARM support

Covering TRIX-893, TRIX-896

  • Alternative iPXE binaries for ARM CPUs and all-architecture iPXE download, with a corresponding netchannel fix in Luna.

HA & AlertX

Covering TRIX-745, TRIX-702, TRIX-786

  • External Floating IP so external clients can always reach the active controller, Prometheus HA-degraded alerts, and an external-FQDN fix for Alertmanager emails.

Configuration rendering

Covering TRIX-858, TRIX-909, TRIX-939, TRIX-949

  • Introduced config-slurm / config-blocks / config-genders rendering with extendable defaults and GRES support, and a full rewrite of the NHC drainer.

Luna 2.x

Covering TRIX-861, TRIX-857, TRIX-866, TRIX-938, TRIX-906

  • luna2-daemon: node/group interface routes updated for current networking, /etc/genders generation, a Luna startup hook, remote-image sync with numeric uid/gid preservation, and bonding roles marked deprecated.
  • luna2-cli: DHCP flag corrected to boolean (TRIX-938) and a base64 option-decoding fix (TRIX-906).

Bug fixes & hardening

  • Numerous fixes across the openSUSE/Ubuntu image builds, openldap version handling, Alertmanager credential-file permissions, and iPXE cloud-boot behaviour.

15 release

TrinityX 15 was released on February 27 2025. A brief summary what's included:

  • AlertX - commandline and graphical application to manage prometheus alerts, rules and manage Node Health Checking (NHC)
  • NHC drainer - nodes triggered by the NHC rule are drained from jobs. currently slurm supported.
  • Per Job statistics - detailed breakdown per job for resource utilization and power consumption
  • Beta ARM support
  • additional prometheus exporters for collecting more metrics including GPU, Hardware config and state
  • OOD application for changing a user’s password
  • Improved/extended grafana panels
  • luna 2.1
    • luna cli updates for better posix compliancy: – and - parameters.
    • luna cli to better handle inheritance; warnings/indicators when local overrides apply
    • luna daemon support for pure DHCP deployed clusters, including nodes in a pool/ddns
  • Open Ondemand 4.0.0
  • latest OpenHPC release 2.9 for EL8 and 3.2.1 for EL9
  • Prometheus 3.1.0
  • HA setups support cross mount shared disk exports, allowing passive/standby controllers to access the shared filesystems

Details

AlertX, NHC & hardware exporters

Covering TRIX-610, TRIX-667, TRIX-668, TRIX-707, TRIX-721, TRIX-737, TRIX-747, TRIX-752, TRIX-755

  • New AlertX command-line and GUI tooling to manage Prometheus alerts and rules.
  • NHC drainer reworked with proper logging, an INI config file and controller-only execution, draining flagged nodes from Slurm.
  • Added an nhc/lshw hardware exporter and a hardware-rules generator with live rule reloading.

Per-Job Statistics & monitoring dashboards

Covering TRIX-578

  • Per-job power and resource statistics via a new Slurm-job Prometheus exporter, surfaced in revised Grafana dashboards.

Beta ARM support

Covering TRIX-650, TRIX-651

  • ARM-ready packages, ARM64 Kubernetes support, aarch64 Grafana, and architecture-aware base-image downloads.

Open OnDemand 4.0

Covering TRIX-664, TRIX-690, TRIX-695, TRIX-712

  • Upgrade to Open OnDemand 4.0 with a refactor of the OOD modules/apps, a user password-change application, lchroot (osimage) functionality, and a mandatory FQDN at install time.

HA shared filesystems

Covering TRIX-652, TRIX-738, TRIX-855

  • NFS exports that also mount on controllers so passive/standby controllers reach the shared filesystems, backup of /etc/trinity and all.yml, and PanFS / Lustre / GPFS mount support.

Luna 2.1

Covering TRIX-691, TRIX-675, TRIX-676, TRIX-701, TRIX-708, TRIX-711, TRIX-673, TRIX-699, TRIX-725

  • luna2-daemon: full DHCP/DDNS provisioning — intelligent next-node discovery on plain boots, a DNS plugin for static journal entries, persistent per-interface DHCP settings, and node-state hooks.
  • luna2-cli: POSIX-compliant options (TRIX-699), a --local flag and markers when a node/group diverges from its parent (TRIX-673), and a packing/boot-pause toggle (TRIX-725).

Bug fixes & hardening

  • Numerous fixes across shared-fs/NFS export handling, OOD app permissions, Grafana/Prometheus rule tidy-ups, OMAPI key generation (TRIX-704), iPXE-over-HTTPS (TRIX-693) and the OpenHPC release bump (TRIX-716).

14.4 release

TrinityX 14.4 was released in September 2024 and maintained through update releases up to 14.4u4 (February 2025). It was the most mature 14-series release, adding Kubernetes/K3s as a workload manager, broader monitoring, remotely-runnable Ansible, and significant Luna provisioning and cloud capabilities.

Summary - Kubernetes / K3s support as a selectable workload manager - Remotely-runnable Ansible playbooks - Expanded monitoring: Slurm-job, InfiniBand and ZFS dashboards and exporters - Password and SELinux hardening, with a safety net during init - Cloud (AWS) node provisioning and a self-healing database structure in Luna - Open OnDemand Jupyter and code-server refinements

Details

Kubernetes / K3s

Covering TRIX-545

  • Added Kubernetes/K3s as a selectable workload manager (installed only when listed in workload_managers), including K3s-on-Ubuntu support, long-lived cluster certificates, controller-scoped firewall rules, and a housekeeping job to remove stale Kubernetes nodes.

Remotely-runnable Ansible & init hardening

Covering TRIX-629, TRIX-644, TRIX-577

  • Reworked the Ansible codebase to run against remote hosts again, added a safety net and more tolerant SELinux handling during init, hardened password-quality requirements, and added a hostname-consistency check.

Monitoring & dashboards

Covering TRIX-578, TRIX-490, TRIX-489, TRIX-612

  • New Slurm-job exporter and dashboard, an InfiniBand exporter, a Grafana ZFS dashboard, filesystem-monitoring rules, and refinements to the Slurm and power-consumption panels.

Packaging & repositories

Covering TRIX-607, TRIX-565, TRIX-579, TRIX-616, TRIX-637

  • Configurable RPM download retries/delays, fixed hardcoded external/elrepo URLs, a debootstrap fix, and a luna2-grub2-autofind package for local-disk detection.

Open OnDemand

Covering TRIX-743

  • Refactored the Jupyter app to use per-user virtual environments (with Ubuntu python3-venv support), code-server fixes, and a backport of the osimage OOD app (lchroot).

Luna 2.x

Covering TRIX-145, TRIX-146, TRIX-586, TRIX-594, TRIX-595, TRIX-608, TRIX-548, TRIX-553, TRIX-699, TRIX-673, TRIX-589, TRIX-609

  • luna2-daemon: cloud/AWS node provisioning (TRIX-145/146), a self-healing database structure that adds missing tables/columns on import (TRIX-586), osimage rename that also moves its path (TRIX-594), a "safety net" preventing removal of in-use groups/osimages/bmcsetups/switches (TRIX-595), additional pre/part/post script plugins (TRIX-608), and disk-space verification before packing.
  • luna2-cli: a new Cloud class — list/show/add/change/remove (TRIX-548), kernel-option support on node add/change/clone (TRIX-553), more POSIX-compliant options (TRIX-699), comma-separated roles/scripts (TRIX-589, TRIX-609), and a --local flag with divergence markers when a node/group differs from its parent (TRIX-673).

Bug fixes & hardening

  • Numerous fixes across openldap/HA shared-filesystem handling, DRBD disk naming, reverse-DNS zone creation, network-config templating, and Grafana dashboard tidy-ups.

14.3 release

TrinityX 14.3 was released in September 2024. It centred on public-cloud deployments, bringing Terraform-driven provisioning on AWS and Azure with VPN tunnels back to on-prem, plus the Luna changes needed to run controllers and nodes in the cloud.

Summary - Public-cloud deployment on AWS and Azure, provisioned through Terraform with IPsec/VPN tunnels to on-prem - "Shadow controllers" enabling multi-controller setups without a shared IP, for cloud and remote sites - Luna cloud awareness: cloud detection, cloud node/controller objects, and a new CLI cloud command - Docker-based and Ubuntu/debootstrap image-building improvements - Configurable Prometheus exporter ports and pcs-2.0 packaging - Networking and HA hardening (NetworkManager/resolv.conf, firewalld, DRBD/pacemaker dependencies)

Details

Public-cloud deployment (AWS & Azure)

Covering TRIX-446, TRIX-445, TRIX-540

  • Terraform-driven provisioning of cloud controllers and nodes, with a two-stage apply for idempotency, remote-controller installation, and cloud-storage support for fully in-cloud clusters.
  • IPsec/VPN tunnels (strongSwan/swanctl) between cloud and on-prem, with PSK handling and external-IP discovery for the tunnel endpoints.
  • AlmaLinux Docker-based image support for cloud nodes.

Image building & distributions

  • Improvements to Docker-based image creation and Ubuntu/debootstrap images (missing-script and device fixes, EFI tooling), and PowerTools enabled for non-OpenHPC Slurm installs.

Monitoring & packaging

Covering TRIX-502, TRIX-549

  • Configurable default ports for the Prometheus exporters, and corrected pcs-2.0 package naming.

Networking & HA hardening

  • NetworkManager no longer renders resolv.conf, firewalld zone fixes, OS-specific DRBD/pacemaker dependencies, and an early, clear failure when the shared DRBD disk cannot be initialised.

Luna 2.x

Covering TRIX-145, TRIX-146, TRIX-548, TRIX-567, TRIX-552, TRIX-568, TRIX-556, TRIX-558, TRIX-107, TRIX-553, TRIX-551

  • luna2-daemon: cloud detection and cloud node/controller objects (TRIX-145/146), shadow-controller support so multi-controller setups work without a shared IP, gateway/DNS honoured even when bootproto=dhcp plus a bonding-role example (TRIX-556/558), a disk-space check before packing (TRIX-552), a boot-pause option while an image is being packed, next-free-IP selection when cloning (TRIX-567), and loghost info from the daemon (TRIX-107).
  • luna2-cli: a new cloud command (list/show/add/change/remove, TRIX-548), kernel-option add/change/clone on nodes (TRIX-553), and an osimage kernel-change fix (TRIX-551).

Bug fixes & hardening

  • Numerous fixes across the CI/CD pipelines (moved to an external repo), image-build robustness, ood-vnc cleanup, and installer/prepare ergonomics.

14.2 release

TrinityX 14.2 was released in June 2024. It broadened image and cloud provisioning, added interactive applications, and brought full IPv6 and VLAN support to Luna.

Summary - Cloud and VHD image provisioning with a Terraform-based cloud controller - Multi-distribution image building: Ubuntu, Alma Linux and CentOS base-image playbooks, plus Docker image support - Interactive applications: Jupyter and Code Server, and a new TUI configurator - Open OnDemand 3.1 with PAM authentication as an alternative to Dex - Slurm / OpenHPC refresh and HA hardening - Luna 2.x: full IPv6 support, VLAN and per-node/group kernel options, kickstart provisioning, and full configuration import/export

Details

Cloud & VHD provisioning

Covering TRIX-509, TRIX-533

  • Initial cloud installation path with an integrated cloud controller, Terraform role and group_vars, plus VHD image building (with the associated GRUB fixes and a fixed VHD size).

Multi-distribution image building

Covering TRIX-493, TRIX-535, TRIX-515

  • New Ubuntu base-image creation/download playbook, Alma Linux support, and a CentOS playbook, alongside Docker image support and a template-based reduction of the image-create code. Added an image-name compliancy check and better image-version handling on import.

Interactive applications & Open OnDemand

Covering TRIX-501, TRIX-507, TRIX-525, TRIX-404, TRIX-473, TRIX-476, TRIX-265

  • Jupyter and Code Server interactive applications configured for both controllers and images, and a new TUI configurator.
  • Open OnDemand moved to 3.1 with PAM authentication added as an alternative to Dex (toggle switch) and VNC fixes for Rocky/EL9.

Slurm, OpenHPC & HA

Covering TRIX-522, TRIX-531, TRIX-532, TRIX-453, TRIX-393, TRIX-470

  • Consistent Slurm user/UID handling, slurmdbd fixes, support for non-OpenHPC Slurm packages, and a refresh to the latest OpenHPC repos. HA hardening around corosync key distribution and the pacemaker add-on repo, plus InfiniBand/BMC network configuration promoted from controller to group level.

Luna 2.x

Covering TRIX-236, TRIX-511, TRIX-512, TRIX-133, TRIX-456, TRIX-475, TRIX-536, TRIX-177, TRIX-504, TRIX-455, TRIX-449, TRIX-450, TRIX-448, TRIX-496, TRIX-477, TRIX-480

  • luna2-daemon: full IPv6 support across DNS, DHCP, booting and network plugins (TRIX-236); VLAN-aware network plugins with per-node/group kernel options (TRIX-511, TRIX-512); kickstart as an alternative provisioning method (TRIX-133); full configuration export/import across all tables (TRIX-456); role inclusion/inheritance in the install template (TRIX-177); IP-address reservation to prevent duplicate leases (TRIX-504); and an overhaul of the hard-coded "controller" naming (TRIX-536).
  • luna2-cli: IPv6-aware network listing, show and clear (TRIX-449, TRIX-450), switch/other-device update fixes (TRIX-448), a dynamic per-controller parameter for cluster/network changes (TRIX-496), and secrets and domain-search handling (TRIX-477, TRIX-480).

Bug fixes & hardening

  • Numerous fixes across monitoring dashboards, home-built Prometheus exporters (InfiniBand metrics), Grafana no_proxy handling, RPM/glibc database issues, and CI/CD stabilisation.

14.1 release

TrinityX 14.1 was released in March 2024. It hardened High Availability, overhauled Prometheus/Grafana monitoring, refined Open OnDemand, and introduced journal-based replication in the Luna daemon.

Summary - High-Availability hardening around DRBD, Pacemaker and Corosync - Reworked monitoring: new and fixed Prometheus exporters and a refreshed Grafana dashboard set - Open OnDemand refinements, including an external FQDN for outward-facing services and an InfiniBand graphing app - Identity and authentication improvements via the obol tooling, plus install-time security hardening - Broader image-building and provisioning support (HTTPS debootstrap, entitlement sync, richer hardware data) - Luna daemon HA journal/replication and a substantially extended luna2-cli

Details

High Availability

Covering TRIX-393

  • Hardened the DRBD/Pacemaker/Corosync stack: safer disk validation before configuring DRBD, cleaner post-unstandby resync, corosync token handling, and HA services moved into the Luna role for consistent restart/retry behaviour.

Monitoring & dashboards

Covering TRIX-419, TRIX-421, TRIX-425, TRIX-431, TRIX-435, TRIX-459, TRIX-460, TRIX-461, TRIX-464

  • Introduced an NVIDIA exporter role and fixed several existing exporters, added an allowed_exporters directive, and improved the Prometheus file_sd role and target discovery (by IP, all controllers).
  • A broad Grafana refresh — reworked home dashboard, navigation bar across dashboards, and consistent refresh intervals — with Slurm updated to match the new monitoring.

Open OnDemand

Covering TRIX-148, TRIX-253, TRIX-265, TRIX-282, TRIX-285, TRIX-295

  • Introduced an external FQDN for outward-facing services, an InfiniBand graphing app, auto-reload when an app is not yet initialised, an EL9 OOD-VNC fix, and a slimmed desktop (XFCE only).

Identity, authentication & security

Covering TRIX-175, TRIX-230, TRIX-249, TRIX-398, TRIX-407, TRIX-412, TRIX-413, TRIX-260, TRIX-390

  • obol feature updates with added logging, relocation to /usr/local/sbin, and authentication-tag fixes (including external-auth admin-group handling).
  • Install-time hardening: luna.db set to 0600, and installation halts if SELinux is disabled (requiring a reboot).

Image building & provisioning

Covering TRIX-247, TRIX-315, TRIX-336, TRIX-338, TRIX-378, TRIX-395, TRIX-439

  • HTTPS (instead of FTP) for debootstrap base images, image entitlement sync, global PowerTools handling, and richer hardware data in images and ramdisk (PCI-ids/lspci, dmidecode). Image creation no longer requires grab_exclude to be set, and the Ubuntu image gained netifname kernel options.

Luna 2.x

Covering TRIX-146, TRIX-357, TRIX-373, TRIX-318, TRIX-321, TRIX-363, TRIX-364, TRIX-449, TRIX-450

  • luna2-daemon: journal-based HA replication (TRIX-146) — master/slave osimage and config/kernel sync, replicated routes, table-hash verification, request locking and a housekeeper to keep controllers consistent; plus DNS/DHCP reload-on-mismatch and prevention of networks with clashing IP/subnet config.
  • luna2-cli: bash completion (TRIX-357), double-dot hostlist support across node operations (TRIX-373), IPv6 in network list/show and an IPv4/IPv6 clear flag (TRIX-449, TRIX-450), DNS network management (TRIX-318), monitor status/queue output (TRIX-321), and the _source API for inheritance (TRIX-363, TRIX-364).

Bug fixes & hardening

  • Numerous fixes across DNS/DHCP templates, network validation, OOD apps, exporter/Grafana details, and image-build edge cases.

14 release

TrinityX 14 was released in October 2023. It is the foundational modern TrinityX release: a ground-up rebuild centred on the all-new Luna 2 provisioning stack, a redesigned Grafana-based monitoring experience, and an Open OnDemand portal.

Summary - Introduction of the Luna 2 provisioning stack — a rewritten luna2-daemon (REST API service) and a new luna2-cli, replacing the previous Luna generation - Redesigned monitoring with Grafana dashboards (system info, GPU, IPMI, workload management) backed by Prometheus/telegraf exporters - Open OnDemand 3 portal, with embedded Grafana widgets - Slurm workload-management dashboards with Slurm accounting, on OpenHPC 2.6.1 - Image-file based OS image distribution (osgrab/ospush), replacing the earlier torrent/aria2c approach - Configuration via role overrides and release streams (stable/unstable, main/development) - HA rework and migration of the trix-diag tooling into luna2-utils

Details

Luna 2 — new provisioning daemon & CLI

Covering TRIX-4, TRIX-23, TRIX-32, TRIX-38, TRIX-39, TRIX-87, TRIX-88, TRIX-106, TRIX-112, TRIX-118, TRIX-119, TRIX-130, TRIX-139, TRIX-166, TRIX-179

  • luna2-daemon: a brand-new REST API provisioning service built from the ground up — config/bootstrap handling, a database layer with node/group secrets, dynamic logging, template sanity-checks and reload, a boot API (short/disk/manual) with node-status tracking, and a managed service for dhcpd/dns/luna2.
  • luna2-cli: a complete command-line client covering cluster, node, group, network, osimage, bmcsetup, switch, other-device, secrets, service and power control — with a JWT-authenticated token, an editor flow for multiline/script fields, self-contained table rendering (dropping heavy dependencies), and RPM/pip packaging.

Monitoring & Grafana dashboards

Covering TRIX-184, TRIX-193, TRIX-194, TRIX-195

  • A redesigned dashboard suite — system info, GPU, IPMI and workload-management panels — fed by Prometheus/telegraf exporters, with new vmstat and interrupts plugins and "all"-style selectors across panels.

Open OnDemand 3

Covering TRIX-180

  • Open OnDemand 3 portal integration, including embedded Grafana monitoring widgets and OS-image tag awareness.

Workload management, Slurm & OpenHPC

Covering TRIX-184

  • Reworked workload-management dashboards with a Slurm Accounting connection, running on OpenHPC 2.6.1 for EL8.

OS images, HA & platform plumbing

Covering TRIX-109, TRIX-135, TRIX-190, TRIX-200

  • Image-file based OS image distribution with osgrab/ospush, retiring the torrent/aria2c mechanism; configuration via role overrides and release streams (stable/unstable, main/development); HA pcs rework; chrony local-stratum support; relocation of trix-diag into luna2-utils; and prepare.sh hostname validation.

Bug fixes & hardening

  • Numerous fixes across the new Luna stack, Grafana dashboards, OpenHPC/EL8 repositories, NFS exports, and CI pipelines as the 14 line stabilised.