February 2020


Infiniband notes

  • Infiniband is a good idea for multi-node gpu training as it can use RDMA. This allows exchanges between the gpus on each node without first copying the data from the gpu into system ram.
  • QDR:
    • 40 gbps per port
    • 1 micro second latency
    • $40 for a dual port card
  • Bonding of infiniband interfaces is not possible?


  • Bought 3 quad intel nics for $35 on ebay. This should help with lizardfs bandwidth.
  • Bought an m40 gpu with 24gb of vram.
    • Needs a fan and the 3d printed bracket for adding fans to the passive card.