TaewoooPark

OSICBench — Data skill for Claude Code

Data community

Operating Scientific Instruments via Code — a benchmark for AI agents.

How to install OSICBench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open TaewoooPark/OSICBench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What OSICBench does

Operating Scientific Instruments via Code — a benchmark for AI agents. Physics goal + instrument manual in; any code style out; scored only on what physically happened: data fidelity vs hidden ground truth, hardware safety from a flight recorder, fault resilience, and interaction efficiency.

Alternatives in Data

Agent Loop Skills — Loop until it's better — drop-in agentic loops (autoresearch, scientific writing, data analysis, code/SQL/prom 155 ★
Fix Tickets — Autonomously fix one or more speckit-companion GitHub issues with the SpecKit Companion pipeline — clean branc 85 ★
Seismic Workflow — Guide seismic data analysis from loading through rock physics to visualization 60 ★

README

OSIC-Bench

**Operating Scientific Instruments via Code — a benchmark for AI agents.**

Can an AI agent write the code that runs a real experiment? Not "does the code look right" — did the instrument produce **correct data**, did the hardware end in a **safe state**, did the run **survive the faults** that real benches throw at you?

OSIC-Bench measures exactly that. Agents get a physics goal and an instrument manual; they write control code in any style they like; the code runs against a simulated instrument farm with hidden ground-truth physics, realistic protocol quirks, and scheduled fault injection. **Scoring never looks at the code — only at what physically happened.**

Quick start

pip install -e .

# run one reference solution against one task, then grade it
osicbench run --task tasks/t03_diode_iv \
    --submission tasks/t03_diode_iv/reference/ref_procedural.py \
    --seed 7 --out runs/demo/t03_diode_iv_s7 --label demo

# the whole self-validation gate: references must pass, mutants must fail
osicbench validate --tasks tasks --seeds 2 --jobs 6 --out runs/validate

# aggregate labeled runs into a report (pass rates, CIs, paired stats)
osicbench report --runs runs/demo/ --out reports/

# replay any run's flight recorder as a human-readable timeline
osicsim-replay runs/demo/t03_diode_iv_s7/farm/recorder.jsonl

For model or skill comparisons, freeze the expected task/seed/sample matrix before authoring with [`adapters/matrix_runner.py plan`](adapters/README.md). This keeps missing submissions in the denominator and groups repeated samples correctly. The single-run example above remains supported but reports unverified coverage without a manifest. Keep validation sweeps and unrelated experiments outside the run directory being compared.

For subscription-authenticated Codex and Claude Code evaluations, see the [model and native-harness protocol](experiments/model_matrix/README.md). It defines isolated runtime checks, model/reasoning conditions,