Workload management

TrinityX is using the OpenHPC stack for user space, including SLURM. This document may also apply for custom slurm installations and through Epel.

SLURM

TrinityX is configured to use the default paths for SLURM. The configuration directory /etc/slurm is shared on all the nodes but links to /trinity/shared/etc/slurm where all the files reside.

For better readability, the files have been split up and included from the main slurm.conf file.

File Description
slurm.conf Main configuration file
acct_gather.conf Slurm configuration file for the acct_gather plugins (see acct_gather.conf)
slurm-health.conf Health check configuration (where applicable)
slurm-partitions.conf Partition configuration
topology.conf Slurm configuration file for defining the network topology (see topology.conf)
cgroup.conf Slurm configuration file for the cgroup support (see cgroup.conf)
slurmdbd.conf Slurm Database Daemon (SlurmDBD) configuration file ([see slurmdbd.conf(https://slurm.schedmd.com/slurmdbd.conf.html)])
slurm-nodes.conf Slurm node configuration file
slurm-user.conf Slurm user configuration file (e.g. QoS, priorities) - deprecated in 15.2 where this file has been merged into slurm.conf

By default Luna generates the configuration for nodes and partitions, where the partition is based on the group name. This method is useful for homogeneous node-type cluster where a default node contains the detailed configuration for CPU-s, cores and RAM. When more complexity is desired, e.g. having different node-types, the automation can be overridden by manually configuration in these files, or the graphical slurm configurator can be used.

Graphical slurm configuration application

The graphical slurm configurator in action:

Short guidance: There are three main sections: 'Hardware', 'Nodes' and 'Partitions'. 'Hardware' allows for configuring presets or properties that can be selected for nodes in 'Nodes' and 'Partitions'. Both 'Nodes' and 'Partitions' sections allow for further tailoring of properties. After configuring what's preferred, a configuration can be tested for syntax validity and previewed before saving. For each save a backup is created and can be loaded to revert changes.

Advanced Slurm configuration automation

Big Fat Note: Below section applies to TrinityX 15.2 and higher.


      For Legacy, please refer to this section.

The Slurm Graphical Configurator behaved differently for 15.1 and below where it would always take full ownership of the configuration of nodes and partitions. This has drastically changed since 15.2, hence the below section.

Philosophy

The files /etc/slurm/slurm-nodes.conf and /etc/slurm/slurm-partitions.conf contain parts that are by default automatically generated. These config files are typically prepared to allow for that automation through two separate blocks "Defaults" and "TrinityX". Outside these blocks, no changes are made through automation whatsoever. Removing the blocks mentioned will permanently stop any automation. Manual changes will then be required to achieve the desired configuration.

The automation entails meta (Defaults) and running configuration (TrinityX) which allows for automatic node and partition generation/configuration as well utilizing the Slurm Graphical Configurator as a utility that manages detailed properties through direct settings or hardware presets. This approach allows luna by default, but also other future utilities to generate and/or automate the actual running configuration. In short: You can have e.g. luna generate the node and partition configuration based on meta information set through the Slurm Graphical Configurator. The best of both worlds.

Defaults block

The defaults block contains meta or templating information to assist in generating purposeful/running configuration inside the TrinityX block. The Slurm Graphical Configurator works only with the defaults block, building the meta information.

The meta configuration contains HWPresetName lines as well as PartitionName and NodeName lines. Generated lines may be be altered manually to comply to the needs. Properties that are not supported by the configurator will be kept as is but honored for the automation. It is possible to extend the meta configuration lines to each's liking. The meta configuration lines typically starts with a # as these lines are not the real configuration, though the syntax is compliant to Slurm's way of configuring resources.

TrinityX block

By default the TrinityX block contains the automation. This means that luna will generate hosts and partitions (based on groups) inside this block, taking the meta configuration in account (Default block). Manual changes inside this block will be overwritten during regeneration, triggered by adding, removing or changing nodes in luna. No changes will be made in the Defaults block.

TrinityX-OOD block

When selecting 'OOD Slurm Configurator manages Nodes and Partitions', as in enabled, the TrinityX block will be changed to TrinityX-OOD. This stops luna automation and nodes and partitions will from here on be managed through the Slurm Graphical Configurator.

This feature can be stopped or disabled which after that point, other, luna by default, will then generate the nodes and partitions when needed.

Examples

The below examples show how a configuration set in the Slurm Graphical Configurator translates to a running, generated configuration, though luna will render the nodes and partitions.

Hardware Presets

Nodes

Partitions

Utility

trix-config-slurm is a utility that can be used to trigger regeneration of the TrinityX block

TrinityX 15.1 and below: Automation versus Configuration

The order of configuration is as follows:

  • Luna configures slurm. This is an automated process
  • More advanced configuration done through the Slurm Graphical configurator
  • Custom configuration

Luna configures

Luna configures the nodes and groups based on luna config by default. It adheres to the block:

# TrinityX will only manage inside 'TrinityX Managed block'.
# If manual override is desired, the 'TrinityX Managed block' can be removed.
#### TrinityX Managed block start ####
NodeName=node003 # GroupName=compute
NodeName=node002 # GroupName=compute
NodeName=node001 # GroupName=compute
NodeName=node004 # GroupName=compute
#### TrinityX Managed block end   ####

There are no automated changes outside the clearly listed managed blocks. This would allow for some additional configuration if the administrator sees the need for it.

Note: When luna automation is desired, it is however important to configure the line:

NodeName=DEFAULT Boards=1 SocketsPerBoard=1 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=100 State=UNKNOWN

where the config should match with the default node hardware configuration.

Slurm Graphical configurator

When using the Slurm Graphical configurator, it'll ask you to confirm using this approach from now on. This typically happens only once after which the managed blocks will be changed to show that the configuration is no longer managed by luna.

The same approach remains as such that no changes are made outside the listed blocks.

Custom configuration

removing the managed blocks altogether will allow for a complete custom configuration. In this case TrinityX relies on the administrator to configure slurm.