Noah Watkins

github
twitter
linkedin

Setting up RoCE on aarch64 Ubuntu

Notes on setting up RoCE (RDMA over Converged Ethernet) on aarch64 running Ubuntu Server. The instructions below are based off the guide found here http://www.rdmamojo.com/2014/11/08/working-rdma-ubuntu/. However, some package dependencies are not available on arm64. I’ve updated the list to work with the Ubuntu packages available on arm64.

I am going to be using the Utah CloudLab cluster, which contains a bunch of HP Moonshot nodes with the following hardware:

HP Moonshot m400
Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)
64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)
120 GB of flash (SATA3 / M.2, Micron M500)
Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes)

The nodes are running the following software:

[email protected]:~$ uname -a
Linux node.nwatkins-qv1183.emulab-net.utah.cloudlab.us 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 19:05:44 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux
[email protected]:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:   trusty

Install the following packages:

[email protected]:~$ apt-get install libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libcxgb3-1 libmlx4-1 libmthca1 libnes1 libibcommon1

Next we need to install some kernel modules, as Ubuntu currently doesn’t autoload these.

modprobe rdma_cm
modprobe ib_uverbs
modprobe rdma_ucm
modprobe ib_ucm
modprobe ib_umad
modprobe ib_ipoib
modprobe mlx4_ib
modprobe mlx4_en
modprobe iw_cxgb3
modprobe iw_cxgb4
modprobe iw_nes
modprobe iw_c2

Now we can test it using the guide http://community.mellanox.com/docs/DOC-1465. First, start the server:

[email protected]:~$ rdma_server
rdma_server: start
rdma_server: end 0

Then run a client:

[email protected]:~$ rdma_client -s 10.10.1.2
rdma_client: start
rdma_client: end 0
[email protected]:~$ 

And… end 0 means success.

f1c0f49674a6aff32adac1863c38886c68d9d230