Fast Storage Server

• Forwards fast raw devices using iSER protocol
• Read-ahead of the raid should be adjusted for the optimum performance

  blockdev --setra 65536 /dev/sd#
  

• Package tgt
  • No iSER support in OpenSuSE RPMS
  • Recompile adding ISCSI_RDMA=1 parameter to make command
• /etc/tgt/targets.conf

  default-driver iser
  <target iqn.ipepdvcompute1.fast>
      <direct-store /dev/disk/by-partlabel/fast>
          bs-type aio
          MaxRecvDataSegmentLength 2097152
          MaxXmitDataSegmentLength 2097152
          FirstBurstLength 8388608
          MaxBurstLength 8388608
          block-size 512
          write-cache off
      </direct-strore>
  </target>
  

Camera Computer

• openiscsi package, iscsid should be started on boot
• Server node configuration

  iscsi_discovery 192.168.11.1 -t iser -f
  iscsi_discovery 192.168.11.3 -t iser -f
  

• It will populate 2 extra /dev/sd? nodes
• Create softraid 0 for optimal performance

  mdadm --create /dev/md3 -n 2 -l 0 /dev/disk/by-path/pci-0000\:06\:00.0-ip-192.168.11.1\:3260-iscsi-iqn.ipepdvcompute1.fast-lun-1 /dev/disk/by-path/pci-0000\:06\:00.0-ip-192.168.11.3\:3260-iscsi-iqn.ipekatrinadei.fast-lun-1
  

• Format it with XFS

  mkfs -t xfs /dev/md3
  

• fstab entry

  /dev/md3 /mnt/server xfs noauto,noatime,allocsize=1GiB,largeio,swalloc,delaylog,nobarrier 0 0
  

Glusterfs Cluster

• 20TB partitions are formatted using XFS and mounted with default options into /mnt/raid
  • /mnt/raid/gluster/meta - folder with cluster metadata
  • /mnt/raid/gluster/storage - storage
• /etc/glusterfs/glusterd.vol on the storage nodes

  volume management
      type mgmt/glusterd
      option working-directory  /mnt/raid/gluster/meta
      option transport-type socket,rdma
      option transport.rdma.bind-address 192.168.11.3
      option transport.rdma.listen-port 24008
      option transport.socket.keepalive-time 10
      option transport.socket.keepalive-interval 2
      option transport.socket.read-fail-log off
  end-volume
  

• /etc/glusterfs/glusterd.vol on the other nodes

  volume management
      type mgmt/glusterd
      option working-directory  /var/lib/glusterd
      option transport-type socket,rdma
      option transport.rdma.bind-address 192.168.11.2
      option transport.rdma.listen-port 24008
      option transport.socket.keepalive-time 10
      option transport.socket.keepalive-interval 2
      option transport.socket.read-fail-log off
  end-volume
  

• All nodes of cluster should be registered from the master node with

  gluster peer probe 192.168.11.#
  

• Currently replicating storage is used, created with:

  gluster volume create storage replica 2 transport rdma 192.168.11.1:/mnt/raid/gluster/storage 192.168.11.3:/mnt/raid/gluster/storage
  gluster volume start storage
  

• Mounted with following line in fstab

  192.168.11.1:storage /pdv glusterfs defaults 0 0