Shared FS disks

Trinity comes with a powerful, but also relatively complex mechanism for setting up shared disks. This method is mostly used for HA configurations but can also serve to (pre)configure controllers and images to incorporate (external) network file systems.

Where supported, TrinityX sets up devices, partitions and filesystems based on the shared_fs_disk configuration. Below sections explain the where and what.

HA requirements:

The shared_fs_disk layout for HA setups should provide for:

• {{ trix_ha }}
• {{ trix_home }}
• {{ trix_shared }}

as an absolute minimum.

A mix for these definitions may exists, for example being served through different means e.g. NFS or DAS, but they have to be defined or present.

Supported devices and file systems

device or source:

• DRBD
• direct (DAS)
• iSCSI

file systems

• ZFS
• EXT2, EXT3, EXT4
• XFS
• NFS

Additional filesystems

TrinityX offers limited support through the shared_fs_disks method for lustre, beegfs, mmfs, gpfs and, panfs file system types in conjunction to NFS (see type). This does not imply TrinityX would install and configure required drivers, but would be capable of adding appropriate mounts to /etc/fstab on the controller and/or inside images out-of-the-box, although this could always be manually added at a later stage, e.g. after the installation. When these file systems are used for HA shared disk purposes, drivers and matching configurations, in short: a capable mountable file system, should be set up before proceeding with the installation of TrinityX.

Other unlisted filesystems are supported as long as they work with the used distribution, but exceed the scope of what can be setup during the installation using the shared_fs_disks method. In those cases these are typically configured afterward with (additional) drivers added to the Luna images or methods used through Luna pre/post.

Default HA shared disk example

The default configuration for shared_fs_disk uses a DRBD disk /dev/vda which has to be changed to match your configuration:

shared_fs_disks:
  - name: 'trinityx'
    type: 'drbd'
    disk: '/dev/vda'    <--- must be changed.
    device: '/dev/drbd0'
    fstype: 'zfs'
    partitions:
    - mount: '{{ trix_home }}'
      fsoptions: '-o xattr=sa'
      xmount: true
    - mount: '{{ trix_shared }}'
      fsoptions: '-o xattr=sa'
      xmount: true
    - mount: '{{ trix_ohpc }}'
      fsoptions: '-o xattr=sa'
      xmount: true
    - mount: '{{ trix_ha }}'
      fsoptions: '-o xattr=sa'

It utilizes a ZFS based pool, which (ZFS) is typically installed during the prepare phase. The big advantage of using ZFS is not having to care about the size of each 'partition'. It also provides out-of-the-box snapshotting functionality, extremely handy for home directories.

All of the above can be customized to match your requirement as mentioned in the Pre-install documentation Make sure that the disk is equal in size and blank on all controllers.

type

The type of carrier. This could be drbd, iscsi, direct (for DAS), nfs or any other 'carrier'. This includes additional filesystems like panfs and lustre. TrinityX does not create remote filesystems and targets for e.g. iscsi, nfs or any network filesystems, including the ones listed in Additional filesystems. These need to be present and available before installation. These filesystems however will be added to the /etc/fstabs for even non-HA setups.

disk

The lowest actual device, only used in DRBD setups. DRBD would render the device based on the provided name, which has to be set as well, e.g. /dev/drbd0. TrinityX sets up DRBD and its configuration.

device

The lowest actual device for non-network file system setups. Used with drbd and direct disk type.

partitions

Segment within a device where fstype is lvm, zfs or a native linux filesystem like ext4 or xfs. TrinityX creates and sets up the partitions.

fstype

fstype could be lvm, zfs, ext4, or any other native linux filesystem. When fstype is lvm or zfs, partitions are expected within the definition. for lvm based disks, partitions fstype are again the native linux filesystems. For zfs, the partition fstype is implied zfs and should not be provided. TrinityX creates the partitions and filesystems.

fsoptions

Filesystem creation time options.

mountoptions

Optional setting that let pacemaker mount with the flags provided, e.g. enabling quotas. Also used for network based filesystems tuning how these are mounted.

xmount

Xmount stands for 'cross mount'. File systems provided by the controller(s) may have a setting called xmount where it would render a configuration to provide access for all HA controllers, a cross mount. As a result not only the active controller has access to these file systems, but the passive controllers too. This is a typical enabled setting for {{ trix_home }} and {{ trix_shared }} definitions. {{ trix_ha }} is only used by the active controller and would not need this to be enabled.

Alternative examples

Below examples are just to show case what could be possible. They most likely do not reflect your setup, but can be used for inspiration.

DRBD disk, using whole disk with ext4

shared_fs_disks:
  - name: 'Home directories'
    mount: '{{ trix_home }}'
    type: 'drbd'
    disk: '/dev/sda2'
    device: '/dev/drbd2'
    fstype: 'ext4'
    fsoptions: '-b 4096 -O quota -E quotatype=usrquota:grpquota'
    mountoptions: 'quota,usrquota,grpquota'
    xmount: true

iSCSI disk based with LVM and partitions

shared_fs_disks:
  - name: 'LVM-disk'
    type: 'iscsi'
    fstype: 'lvm'
    portal: '192.168.0.1'
    target: 'iqn.2003-01.org.linux-iscsi.iscsi-host.x8664:sn.693ca380655e'
    partitions:
    - mount: '{{ trix_home }}'
      size: '8192M'
      fstype: 'ext4'
      fsoptions: '-b 4096'
    - mount: '{{ trix_shared }}'
      size: '16384M'
      fstype: 'ext4'
      fsoptions: '-b 4096'

NFS based shared disk

shared_fs_disks:
  - name: '{{ trix_home }}'
    mount: '{{ trix_home }}'
    type: 'nfs'
    remote: '192.168.0.1:/homes'
    mountoptions: 'nfsvers=4.1,rw,retrans=4,_netdev'
  - name: '{{ trix_shared }}'
    mount: '{{ trix_shared }}'
    type: 'nfs'
    remote: '192.168.0.1:/shared'
    mountoptions: 'nfsvers=4,rw,retrans=4,_netdev'

Direct disk

Direct disks are by default not shared and could technically only be used for {{ trix_ha }}. Xmount not advisable.

shared_fs_disks:
  - name: 'LVMlocal'
    type: 'direct'
    device: /dev/sdz
    fstype: 'lvm'
    partitions:
    - mount: '{{ trix_ha }}'
      size: '32768M'
      fstype: 'ext3'
      fsoptions: '-b 2048'

Mix and match example

Some extreme setups might see a combination of sources providing shared disks for TrinityX:

shared_fs_disks:
  - name: '{{ trix_home }}'
    mount: '{{ trix_home }}'
    type: 'nfs'
    remote: '192.168.0.1:/homes'
    mountoptions: 'nfsvers=4.1,rw,retrans=4,_netdev'

  - name: 'MyShared'
    mount: '{{ trix_shared }}'
    type: 'drbd'
    disk: '/dev/sdb'
    device: '/dev/drbd0'
    fstype: 'ext4'
    fsoptions: '-b 4096 -O quota -E quotatype=usrquota:grpquota'
    mountoptions: 'quota,usrquota,grpquota'
    xmount: true

  - name: 'other'
    type: 'drbd'
    disk: '/dev/disk/by-id/dm-name-vg_rhel-lv_other'
    device: '/dev/drbd1'
    fstype: 'zfs'
    partitions:
    - mount: '{{ trix_ha }}'
      fsoptions: '-o xattr=sa'
    - mount: '{{ trix_easybuild }}'
      fsoptions: '-o xattr=sa'
      xmount: true

  - name: 'my-other-mount-i-want'
    mount: '/opt/applications'
    type: 'nfs'
    remote: '192.168.0.1:/opt/applications'
    mountoptions: 'nfsvers=4,ro,retrans=4,_netdev'

In above example, all minimum required shared disks {{ trix_home }}, {{ trix_shared }} and {{ trix_ha }} are represented for an HA setup, including an extra mount for /opt/applications, not needed by TrinityX, but need to configured on the controllers and images. This is just an example of how various disk types and approaches can be used.

Notes