ofi-intercept

A transparent, workload-agnostic libfabric interception framework. It lets you intercept selected fi_* data-path calls (fi_read/fi_write, fi_send/fi_tsend and their msg/v/recv variants, fi_cq_read/fi_cq_sread, plus the fi_getinfo and fi_cq_open setup calls) and decide, per call, what to do — observe, modify arguments/results, or replace the call entirely. Nothing is intercepted unless you register a hook for it.

It is a library, not a preloadable .so on its own: you link it into your own LD_PRELOAD wrapper together with your hooks.

How it works

fi_read/fi_write/fi_send/fi_tsend/fi_cq_read/… are static inline dispatches through per-object ops tables, so there are no dynamic symbols to LD_PRELOAD. The only exported symbols in the creation chain are fi_getinfo, fi_fabric, fi_fabric2. The framework interposes those, then walks the object graph (fabric → domain → endpoint, and domain → cq) and clones each object's ops table in place, overriding only the slots you hooked and repointing the live object's ->rma/->msg/->tagged/->ops at the clone. The fid handed back to the application is always the provider's real fid, so provider container_of() on the handle stays valid.

The hook model (wrap)

Fill in an lfi_hooks table and call lfi_register() (from a library constructor, before the app creates its fabric). Every field is optional — a NULL slot is left pointing straight at the provider.

Each hook receives the original call arguments plus two trailing parameters:

• next — the real provider op; call it to forward the call.
• user — your context pointer (lfi_hooks.user).

#include "libfabintercept.h"

static ssize_t my_tsend(struct fid_ep *ep, const void *buf, size_t len,
                        void *desc, fi_addr_t dst, uint64_t tag, void *ctx,
                        lfi_tagged_send_fn next, void *user)
{
    record(ep, buf, len, tag);             /* observe  */
    ssize_t r = next(ep, buf, len, desc, dst, tag, ctx); /* forward */
    return r;
    /* or: return some_value;  without calling next() -> replace the op */
}

__attribute__((constructor))
static void install(void)
{
    struct lfi_hooks h = {0};
    h.user        = my_state;
    h.tagged_send = my_tsend;
    /* h.cq_read = my_cq_read;  etc. — set only what you need */
    lfi_register(&h);
}

Object-lifecycle callbacks let you set up and tear down per-object state:

• on_ep_create(ep, domain, user) / on_cq_create(cq, domain, user) — fired once, after the object is wrapped.
• on_ep_close(ep, user) / on_cq_close(cq, user) — fired before the provider closes the object, while it and its domain are still valid (so you can cancel or flush your own deferred state).
• on_domain_info(domain, info, user) / on_ep_info(ep, info, user) — fired with the fi_info the object was created from (the struct passed to fi_domain / fi_endpoint), valid only for the callback. Unlike on_*_create, these carry the negotiated caps/mode and the endpoint's tx_attr/rx_attr (op_flags, msg_order) — attributes you can't recover from a live fid afterwards. on_ep_info fires for fi_endpoint / fi_endpoint2 / fi_scalable_ep.

Building

The library needs libfabric's headers and nothing else. Point LIBFABRIC_PREFIX at an installation (or pass your own -I through CPPFLAGS):

export LIBFABRIC_PREFIX=/usr        # wherever <rdma/fabric.h> lives
make                                # -> libfabintercept.a  (debug, -O2 -g)
make release                        # -> libfabintercept.a  (-O3 -DNDEBUG)
make clean

Then in your wrapper's link step, compile your hooks -fPIC, link the archive, and add -ldl:

g++ -std=c++17 -fPIC -shared -I$LIBFABRIC_PREFIX/include \
    my_hooks.cpp /path/to/ofi-intercept/libfabintercept.a \
    -o libmywrapper_preload.so -ldl
LD_PRELOAD=$PWD/libmywrapper_preload.so <app>

Example consumers

Two complete consumers ship with the library. Each is a single .cpp that builds into its own preloadable .so, and each is a working template to copy:

make timer     # -> libtimer_preload.so
make flags     # -> libflags_preload.so

intercept_timer.cpp measures how long each initiator-side data-path op takes, from issue to CQ completion, and attributes it to the directed NIC→NIC link it ran on. It covers the send, write and read families. The technique is worth understanding even if you never use the tool: on issue it claims a ring slot and forwards the op with the application's op_context redirected at that slot, then on fi_cq_read it recognises its own contexts, stamps the end time, and restores the original context before the application ever sees the completion. It can also sample NIC hardware counters on a background thread, so op timings and counter deltas land on one clock. Each rank writes its own file, so per-rank reports never interleave.

Knobs: INTERCEPT_TIMER_DIR (output directory), INTERCEPT_TIMER_OPS_MB and INTERCEPT_TIMER_NIC_MB (per-file caps), INTERCEPT_TIMER_OPS_MIN_BYTES, INTERCEPT_TIMER_OPS_RANKS, INTERCEPT_TIMER_OPS_NIC, INTERCEPT_TIMER_NIC{,_DEV,_MS,_COUNTERS} (counter sampling), INTERCEPT_TIMER_CQGAP. The file header comment documents each one.

intercept_flags.cpp records the fi_info attributes requested and granted at every setup step — the hints passed to fi_getinfo against the result the provider returned, then the fi_info each fi_domain and fi_endpoint is built from. It is the quickest way to find out what capabilities, mode bits, op_flags and msg_order an application actually negotiated, rather than what its source appears to ask for. Knobs: INTERCEPT_FLAGS_DIR, INTERCEPT_FLAGS_RESULTS_MAX.

What is and is not portable

The framework itself is provider-neutral — it only walks libfabric's own ops tables, so any provider works. So is the op timing in intercept_timer, and so is everything intercept_flags records.

The NIC hardware counter sampling is not. It reads HPE Cassini (cxi) counters from /sys/class/cxi/<dev>/device/telemetry/ and the cxi_rh retry handler's files under /run/cxi/<dev>/, so on any other NIC that half is inert — leave INTERCEPT_TIMER_NIC unset and the op timing still works.

Rank identity is read from SLURM_PROCID, OMPI_COMM_WORLD_RANK, PMIX_RANK, PMI_RANK or RANK, whichever is set, and falls back to a single rank.

Two helpers

• rank_nic_env.sh — wraps a launch to give each rank the right INTERCEPT_TIMER_NIC_DEV. Which NIC a rank should sample is a per-rank fact, but the variable is process-wide; get it wrong and ranks either sample a device that does not exist or fall back to polling every NIC on the node, which distorts the measurement at a 1 ms cadence. Read its header before copying — the mapping it encodes is specific to a node layout.
• ops_db.sh — loads a directory of per-op CSVs into one SQLite database so every rank's ops are queryable by column. See ops_db.md.

Environment

• LFI_TRACE=1 — emit setup-path tracing to stderr (off by default). Never on the data hot path.

Adding a new intercepted op

The op set is whatever the fi_ops_* tables expose; extending it is a small, local edit (no consumer change needed for consumers that don't use the new op):

1. In libfabintercept.h, add an LFI_HOOK(<family>_<op>, <ret>, <provider-op parameter list>) and an lfi_<family>_<op>_hook field to struct lfi_hooks.
2. In libfabintercept.cpp, add a sh_<family>_<op> shim with the provider-op signature (use the FWD_OR_HOOK macro) and an INSTALL(...) line in the matching patched_<family>_ops().

Reading what a consumer recorded

analysis/ holds the tooling for the CSVs intercept_timer writes: typed readers for the op and counter files, a static plotter that puts op durations and NIC counter deltas on one time axis, a counter-delta cache, a table of HPE Cassini counter meanings, and a Streamlit front-end for browsing a run. That directory's README documents the file formats and how to run each script.

License

MIT — see LICENSE. Copyright (c) 2026, Parallel Software and Systems Group, University of Maryland.

Not affiliated with or endorsed by HPE. analysis/cassini_counters.py describes HPE Cassini counters on a best-effort basis and carries its own disclaimer; libfabric itself is a separate project under its own license.