Skip to content
View shivpratapsinghpanwar's full-sized avatar

Block or report shivpratapsinghpanwar

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse

Shivpratap Singh Panwar

I build what a humanoid sees and how it moves — perception, RL policies, and the road from sim → sim → hardware.

Humanoid Perception & RL Engineer · Robot Vision · Locomotion Policies · Sim-to-Real · Industrial CV · Edge Inference

Email LinkedIn Portfolio Kaggle ORCID

Open to roles in humanoid / robot perception, robot learning (RL, sim-to-real) and industrial computer vision.


🧭 What I own on a humanoid

graph LR
    subgraph See["👁 Perception"]
        A["Multi-camera RGB-D<br/>detection · segmentation<br/>pose · depth"] --> B["VLA / scene<br/>understanding"]
    end
    subgraph Move["🦿 Policy"]
        C["Task & reward design<br/>curricula · randomisation"] --> D["PPO in MuJoCo Warp<br/>(mjlab · RSL-RL)"]
    end
    subgraph Ship["🔩 Sim → Sim → Hardware"]
        E["Second-simulator<br/>validation"] --> F["ONNX export<br/>C++ runner · HAL"] --> G["Robot"]
    end
    B --> C
    D --> E
    B -. "INT8 / INT4 on edge" .-> G
Loading
Perception the vision stack a robot actually runs: multi-task models (detection, instance segmentation, pose, depth) quantised FP16 → INT4 for edge targets, plus vision-language-action models in the control loop
Policies whole-body locomotion and motion tracking trained from scratch with PPO: reward design, gait mechanics, push/drag curricula, domain randomisation
Sim → sim → hardware model verification against CAD, actuator limits and latency modelling, cross-simulator checks, ONNX → C++ deployment behind a hardware-abstraction layer

🔩 Day job — sim → sim → hardware at Kody Technolab

I build the path a humanoid policy takes from a training simulator to the robot:

  • The robot model is checked, not trusted — URDF → MJCF conversion with an automated verifier that compares the simulated model against the CAD before any training starts.
  • Training — velocity-tracking and motion-tracking tasks with PPO in MuJoCo Warp / mjlab, with mass randomisation and datasheet torque limits.
  • Then the reality gap, on purpose — an actuation-latency model, a second-simulator pass, and ONNX export into a C++ state-machine runner behind a hardware-abstraction layer.
  • One measured lesson: a policy with 0 % falls in an idealised simulator fell 100 % of the time once a 45 ms actuation delay was modelled. Retrained with the delay in the loop, it went back to 0 %. That is the gap that breaks robots on day one, and it was caught before any hardware was at risk.

I also build the perception layer for multi-camera robots (detection, instance segmentation, pose, depth, quantised FP16 → INT4) and perception for a medical screening robot. Robot, customer and architecture specifics stay with my employer; everything below is work I own outright.


🦿 g1-robust-locomotion — whole-body RL for the Unitree G1 (23-DoF)

Trained from scratch, with no pretrained policy and no imitation data, in MuJoCo Warp with RSL-RL PPO on free-tier T4s:

  • Running that is actually running — a speed-adaptive gait clock shortens the cycle from 0.6 s to 0.4 s with commanded speed and pushes stance below 0.5, opening a true flight phase. A fixed clock turns "running" into a fast walk.
  • Take hits from any direction — velocity impulses mixed with sustained horizontal drags from random headings, so balance recovery is learned as a skill rather than memorised against one shove.
  • Reverse locomotion is a trained command (lin_vel_x ∈ [−1.2, 2.0]), on a three-stage command curriculum.
  • Anti-cheat reward shaping — standing-still and feet-air terms stop the classic marching-in-place reward farm.
  • Built to survive session limits — every run is checkpointed and resumable across 9–12 h GPU session kills.

Base stacks I build on: unitree_rl_mjlab (fork) · τ0-VLA (fork, tracked for VLA work).


👁 Humanoid perception & VLA on the edge

Humanoid_Perception_for_Edge — vision-language-action for a humanoid, on a CPU

Tiangong 2 Pro standing in MuJoCo, outside view and head-camera view SmolVLA driving the upper body in a closed loop

  • Tiangong 2 Pro in MuJoCo with a head-mounted RGB-D camera (real intrinsics, metric depth), and a 17-joint upper-body action contract resolved by joint name, so a model can't silently drive the wrong joint.
  • A safety filter (±0.30 rad envelope plus rate limiting) between the model and the actuators, and SmolVLA in a 50 Hz control loop.
  • Every number is measured on a laptop: SmolVLA-base, fp32 on an i5-10210U, takes p50 9.6 s / p95 11.5 s per 50-action chunk at 6.6 GB peak RSS. That gap to real time is exactly what TernaVLA attacks. This milestone is plumbing; the motion is not trained yet, and the README says so.

TernaVLA · ternary-vlm — 1.58-bit models for robots without a GPU

  • TernaVLA benchmarks ternary VLA backbones on a laptop CPU (real layer shapes): the 0.5B tier runs 0.44 s per action chunk at 8 threads, which is real time, against 1.48 s for 1B and 2.42 s for 2B. bitnet.cpp runs the real 2B model 21 % faster than ONNX Runtime.
  • ternary-vlm: a SigLIP2 vision encoder on BitNet b1.58 2B, with LoRA that is quantised jointly with the ternary weights and merges bit-exactly (tested) for llama.cpp export. Research in progress; paper in preparation.

🏭 Industrial computer vision & data tooling

EasyAnnotate — annotation with nothing to set up

pip install easyannotate

EasyAnnotate demo

Boxes, polygons, oriented boxes and keypoints, with SAM 2.1 and YOLO-World assist, a fast review loop for model predictions, dataset operations, and export to YOLO / COCO / VOC / LabelMe / CSV. Local-first, with AI tools sized to your machine. Measured: the editor opens a 20k-image project with the first image in 1.6 s and page loads of 5–7 ms; SAM takes 0.044 s per click on an RTX 2050 and 0.17 s on CPU. CI, tests, desktop releases.

data-doctor — diagnose vision datasets before they lie to you

pip install vision-data-doctor

A delivered production train/val split contained 81 groups of byte-identical images spanning both sides, so 40 % of validation was leaked. The model's 0.75 recall did not survive an honest split, and nothing in the training stack warned. data-doctor makes that a 10-second CI check:

  • Hashes nominate; only pixels convict. SHA-256 catches exact copies, and two perceptual hash families across all 8 dihedral orientations nominate near-duplicates. Every candidate is then verified on decoded pixels, so a reported pair reads 99.4% similar, rot90, never a hash coincidence.
  • It also checks for train/val leakage, COCO structural defects and corrupt files. It exits 1 on failure, and Pillow is its only dependency.

Hobby project: conveyor-belt rock segmentation and sizing (Mask R-CNN, GigE camera, operator dashboard).


🧪 Also built

Project What it is
Synthetic_Data_Factory Synthetic images for data-scarce vision (SD 1.5 + LoRA, from-scratch DDPM), screened for memorisation and tested with paired multi-seed A/B runs. Reported as measured: macro-F1 +0.005 ± 0.019, consistent with zero. The writeup says so instead of selling a headline.
KrishiDisha.ai First-author IEEE ICoEIT 2025 platform: field-photo crop-disease detection, tool-grounded multilingual assistant, crop/fertiliser/yield models, live weather and market prices. Field-validated with 150+ farmers.
rsna-knee-abnormality-detection 12 findings from knee MRI, with weak labels mined from multilingual radiology reports (regex 0.738 → LLM 0.816 → blend 0.846 macro AUC).
edgefront_JEV Hosted model or small local model? Measures accuracy, latency, calibration and cost on your own task (ONNX Runtime, int8).

⚡ Toolbox

Robot learning MuJoCo · MuJoCo Warp · mjlab · RSL-RL (PPO) · reward & curriculum design · domain randomisation · actuator/latency modelling · URDF/MJCF
Deployment ONNX · C++ control runners · TensorRT · ONNX Runtime · OpenVINO · TFLite · bitnet.cpp / llama.cpp · FP16 / INT8 / INT4
Edge targets NVIDIA Jetson · Android robots · embedded boards · laptop-class CPUs · GPU servers
Perception YOLO family · RF-DETR · D-FINE · Mask/Faster R-CNN · SAM 1/2/3 · pose · depth · RGB-D · VLA (SmolVLA, LeRobot) · classical CV & projective geometry
Core Python · C++ · PyTorch · OpenCV · MediaPipe · FastAPI · GitHub Actions

📄 Publications

  • KrishiDisha: Revolutionizing Agriculture with Intelligent Recommendations, Disease Detection, and Yield Prediction — first author · IEEE ICoEIT 2025, pp. 1028–1040 · doi:10.1109/ICoEIT63558.2025.11211713
  • Web-BCD: A Machine and Deep Learning Approach for Breast Cancer Detection — IBM Technical Report, Dec 2024 · ROC-AUC 0.96, deployed for clinician use
  • A Review on Understanding the Patterns of Student Dropout — first author · Studies in Smart Technologies, Springer, 2025 · doi:10.1007/978-981-97-9006-7_20

💼 Experience

ML Engineer · Kody Technolab — humanoid perception & RL (Nov 2025 – present; intern Apr–Oct 2025) · DL Research Mentee · IBM India (2024) B.Tech CSE (AI/ML) · Medi-Caps University · CGPA 8.67


I read the papers properly, design for the compute the robot actually has, and don't ship a number that hasn't earned it.

Pinned Loading

  1. edgefront_JEV edgefront_JEV Public

    Hosted model or small local model? Measure accuracy, latency, calibration and cost of both on your own task, then route between them (ONNX Runtime, int8).

    Python 7

  2. KrishiDisha.ai KrishiDisha.ai Public

    AI decision support for Indian farmers: own field-trained crop disease detection, crop/fertilizer/yield advice, weather, mandi prices, marketplace and a multilingual assistant (Flask + PyTorch)

    Python 4 1

  3. Synthetic_Data_Factory Synthetic_Data_Factory Public

    Synthetic training images for data-scarce vision (SD 1.5 + LoRA, from-scratch DDPM), with memorization screening and multi-seed A/B tests that measure whether the data actually helps.

    Python 3

  4. ternary-vlm ternary-vlm Public

    TernaVLM: parameter-efficient vision adaptation of a natively ternary (1.58-bit) LM with exact ternary export. Research in progress, paper in preparation.

    Python 1

  5. EasyAnnotate EasyAnnotate Public

    Image annotation with nothing to set up: draw, review model predictions fast, export a correct dataset. Local-first; AI tools sized to your machine.

    Python

  6. TernaVLA TernaVLA Public

    Ternary (1.58-bit) vision-language-action models for edge humanoids, measured on CPUs. Active research.

    Python