Skip to content
View SGDlab67's full-sized avatar
🎇
Building Web3
🎇
Building Web3

Block or report SGDlab67

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
SGDlab67/README.md

Huu

Rust data engineer. Solana and on-chain data infrastructure.

About

I build streaming data infrastructure in Rust. Most of my work sits where a firehose of on-chain state meets a database that has to answer questions about it: gRPC subscriptions, Anchor account decoding, columnar storage, and the plumbing that keeps a pipeline running unattended for weeks.

klend-indexer is the longest running piece of that. It subscribes to Kamino Lend account writes over a Yellowstone gRPC stream, decodes them with Anchor discriminators, and lands them in ClickHouse. On GCP it accumulated roughly 1.1M account updates and 180K decoded obligation snapshots. It's frozen now, on purpose: Yellowstone can't re-serve old slots, so the history was exported to Parquet before the stream was stopped. Designing around that one constraint taught me more than the rest of the pipeline did.

karat67 came out of running that thing. An indexer can pass every freshness and throughput check while writing empty or malformed rows, and every dashboard built on top inherits the error. karat67 checks indexed data against the chain itself: account shape against the IDL, byte-level reconciliation of sampled accounts, and slot completeness against getBlocks (skipped slots aren't gaps, which is the part people get wrong). Freshness and liveness are deliberately out of scope. Every indexer already has those.

I also do network measurement research at Oregon State's ASTRO Lab under Dr. Zane Ma, on how much validator identity leaks at Solana's gossip layer and how that leak is weighted by stake and concentrated by cloud provider.

Solana is the domain I know best. What sits underneath it is Rust and data engineering, and that part travels.

Off the clock I argue about the conceptual mechanics of Devil Fruits in One Piece.

Skills

Data & Systems

Rust ClickHouse Apache Arrow Docker Google Cloud

Blockchain & Web3

Solana Anchor TypeScript

Also in the toolbox

Python Node.js Linux Git

What I'm working on

  • karat67: integrity checks for Solana indexers, as a Rust crate and a karat CLI. Three checks (shape, reconciliation, completeness), JSON output, built to drop into CI next to an existing pipeline. This is the active one.
  • klend-indexer: Rust to Yellowstone gRPC to ClickHouse. Account-level decode, slot checkpointing, gap detection, lag metrics, and a freeze/resume runbook. Frozen, maintained, the dataset lives in Parquet on GCS.
  • S-NodeFinder: gossip-layer measurement at OSU ASTRO Lab. Collection is running, analysis is paused.
  • Cold-path storage: Parquet and DataFusion beside ClickHouse, so one dataset serves live queries and cheap historical scans. Planned, not built.

Contact

LinkedIn Email

Happy to talk streaming data, Rust, or Solana internals.

"The clock is the dataset: a stream you stop observing is history you cannot get back."

Pinned Loading

  1. klend-indexer klend-indexer Public

    A continuously running Rust Indexer that subscribes to Solana account writes for Kamino Lend over a Yellowstone gRPC stream, decodes them with Anchor discriminators, and lands them in ClickHouse.

    Rust

  2. solana-gossip-observatory solana-gossip-observatory Public

    S-NodeFinder: measuring the degree of anonymity of Solana validators at the gossip layer

    Python

  3. summer-transfer-hook-challenge summer-transfer-hook-challenge Public

    Solana Summer challenge: SPL Token-2022 transfer hook in Rust/Anchor. Course exercise.

    Rust

  4. solana-summer-t22 solana-summer-t22 Public

    Solana Summer exercise: Token-2022 extensions in Rust/Anchor.

    Rust