Booting over InfiniBand

A node can fetch its kernel, its ramdisk and its image over the InfiniBand fabric instead of Ethernet, and run its provisioning network on IPoIB. Three things have to agree for that: the HCA has to be able to netboot, the ramdisk has to carry and load the InfiniBand drivers, and Luna has to know the network is InfiniBand.

1. The hardware

The HCA's netboot option ROM (FlexBoot on NVIDIA/Mellanox HCAs) is enabled in the HCA firmware and the node's boot order puts it first. That is a firmware setting on the node, not a TrinityX one.
A subnet manager is running on the fabric before the node powers on; IPoIB does not come up without one.

2. The network in Luna

The provisioning network is typed infiniband, and the node's provisioning interface sits on it:

# luna network add ib -N 10.149.0.0/16 -t infiniband
# luna group changeinterface -N ib compute BOOTIF

The type is what makes Luna render an InfiniBand connection profile for the interface on the installed node (datagram transport mode by default).

Keeping the provisioning interface at its default, BOOTIF, is the robust choice: it names whatever interface the node booted from, so it holds whether the HCA comes up as ib0 or as a predictable name such as ibp1s0. Naming the interface (ib0) works too, as long as it is the name the node actually presents — the images carry net.ifnames=0, which gives ib0.

3. The ramdisk: include the drivers, then make them load

The image's kernelmodules lists the drivers the ramdisk carries beyond what it detects. Add the HCA driver and the IPoIB layer to the defaults:

# luna osimage change --kernelmodules "ipmi_devintf, ipmi_si, ipmi_msghandler, mlx5_core, mlx5_ib, ib_ipoib" compute-image

mlx5_core/mlx5_ib is the ConnectX-4 and later family; older HCAs use mlx4_core/mlx4_ib.

Being in the ramdisk is not the same as being loaded. On an EL image the list is handed to dracut as --add-drivers, which only includes the modules; inside the ramdisk the hardware drivers load by themselves when the HCA is found, but ib_ipoib is an upper-layer protocol with no hardware to trigger it and never loads on its own. The kernel options force it:

# luna osimage change --quick-kerneloptions "rd.driver.pre=ib_ipoib" compute-image

dracut offers two of these, both taking a comma-separated list: rd.driver.pre= loads the modules before the hardware is probed, rd.driver.post= after udev has settled. Which one a release needs differs: on EL9 images pre is the form that has been seen to bring the interface up, and on EL8 and Ubuntu 24 images post — take that as the first thing to try rather than as a rule, and if the modules are in the ramdisk (step 4) but the interface never appears, use the other form, listing the HCA driver as well:

# luna osimage change --quick-kerneloptions "rd.driver.post=mlx5_core,mlx5_ib,ib_ipoib" compute-image

Kernel options are read from the node, else the group, else the image, so the line can also live on a group that boots over InfiniBand while the rest of the cluster does not.

Pack the image afterwards:

# luna osimage pack compute-image

4. Verify the ramdisk before booting a node

The pack does not report a module that dracut could not find for the packed kernel. Look inside the ramdisk Luna will serve — its name is the image's initrdfile, under Luna's files directory:

# luna osimage show compute-image | grep initrdfile
# lsinitrd /trinity/local/luna/files/compute-image-1758123456-initramfs-5.14.0-570.12.1.el9_6.x86_64 | grep -E 'ib_ipoib|mlx5'

ib_ipoib.ko and the HCA driver must be listed; for an Ubuntu ramdisk use lsinitramfs. A module missing here is usually one that is not built for the image's kernelversion:

# lchroot compute-image modinfo -k 5.14.0-570.12.1.el9_6.x86_64 ib_ipoib

On the node, the installer console shows the boot interface coming up; a node that stops at "waiting for network" with the modules present in the ramdisk is the loading problem of step 3.