ReproBench: Automated Paper Reproduction Scorecard
A tooling suite that runs reproduced model checkpoints against held-out benchmarks and auto-generates a comparison scorecard against reported paper results.
Overview
ReproBench standardizes how I track paper reproductions: a YAML spec per paper declares the claimed metrics, the CI pipeline runs the checkpointed model against the declared benchmark, and MLflow logs are diffed against the paper's reported numbers to flag any drift after dependency upgrades.
Architecture
A GitHub Actions workflow triggers on checkpoint updates, spins up an evaluation container, computes metrics, and posts a comparison table as a PR comment; historical runs are stored in MLflow for trend tracking across library version bumps.
Tech Stack
Screenshots
Lessons Learned
A surprising number of "reproduction regressions" turned out to be caused by silent breaking changes in third-party library defaults (e.g., a normalization default changing between torchvision versions) rather than actual code bugs — automated regression tracking caught these early.
ls -la ./related-projects/
Edge-NIDS: On-Device Intrusion Detection for IoT
A quantized CNN intrusion-detection model deployed to an ARM Cortex-M microcontroller for real-time IoT traffic monitoring without cloud connectivity.
FedDiagnose: Cross-Hospital Federated Diagnostic Modeling
A simulated multi-hospital federated learning platform for chest X-ray classification, keeping patient imaging data local to each institution.
ClinSumm: LoRA-Tuned Clinical Note Summarizer
A parameter-efficient fine-tuned language model that summarizes de-identified clinical notes into structured, clinician-readable briefs.