object-over-rdma

S3-compatible RDMA GET/PUT for disaggregated JBOF (Just-a-Bunch-of-Flash) storage.

Architecture

Objects are served from NVMe drives attached over NVMe-oF/RoCE. A metadata server translates S3 GET /bucket/key requests into a layout JSON describing which LBA extents on which drives hold the object. The client reads those extents directly via pread(2) on NVMe-oF block devices (/dev/nvmeXn1) and verifies CRC32C on the GPU.

S3 client  →  metadata server  →  layout JSON
                                       ↓
                             read planner (libobject_rdma)
                                       ↓
                         pread(/dev/nvmeXn1) + GPU CRC32C
                                       ↓
                             verified GPU buffer

Components

Component	Path	Role
Read planner	source/read_planner.c	Extent grouping, parallel pread, CRC
JBOF GET helper	vendor/aws-c-s3/source/s3_jbof_get.c	HTTP layout fetch, layout cache, fd pool
JBOF PUT helper	vendor/aws-c-s3/source/s3_jbof_put.c	Placement request, parallel pwrite, commit
Meta-request	vendor/aws-c-s3/source/s3_jbof_meta_request.c	aws-c-s3 dispatch
SigV4 signer	vendor/aws-c-s3/source/s3_jbof_sigv4.c	Self-contained HMAC-SHA256
Mock server	mock/metadata_server.py	Test layout server

Build

# Host-only (unit tests, no CUDA):
mkdir build_host && cd build_host
cmake -DOBJECT_RDMA_WITHOUT_CUDA=ON -DOBJECT_RDMA_BUILD_TESTS=ON ..
make && ctest -V

# Full build (CUDA + nvCOMP required):
mkdir build_demo && cd build_demo
cmake -DOBJECT_RDMA_BUILD_S3_JBOF_DEMO=ON ..
make -j4

Quick start

See docs/usage_example.md for a code walkthrough.

Protocol

See RDMA_PROTOCOL_SPEC.md for the full wire protocol specification (layout JSON schema, NVMe-oF target format, PUT/GET phases).