AI Evaluation, Data Annotation & Benchmarking Subcontracts
Multiple Frontier AI Companies · Present
- Worked across AI data annotation, dataset creation, RLHF/SFT data generation, expert model evaluation, calibration, rubric creation, task validation, and quality assurance for frontier AI systems.
- Created and evaluated repository-level software-engineering tasks across Python, Go, and Rust, including backend systems, APIs, games, debugging, and infrastructure-oriented problems across repositories such as etcd, Kubernetes, Vault, and Traefik.
- Performed expert RLHF / preference evaluation of model-generated patches for highly technical repository-level tasks, assessing correctness, implementation quality, regressions, edge cases, and issue resolution using detailed rubrics and test cases.
- Worked on highly challenging SWE-bench-style repository problems for frontier models, selecting difficult issues from areas of expertise and defining task requirements, expected behavior, rubrics, and validation tests for broken or incomplete model-generated solutions.
- Created 60+ terminal-based benchmark tasks across mathematics, formal methods, scientific computing, software engineering, code debugging, reverse engineering, low-level programming, business knowledge, and quantitative reasoning.
- Contributed to CI/CD and evaluation-harness engineering, supporting reproducible task execution, environment setup, trajectory capture, automated verification, test execution, and model-output evaluation.
- Created and reviewed a large volume of computer-use tasks across Linux, Windows, and macOS, covering GUI workflows, application interaction, file manipulation, browser workflows, and multi-step real-world computer operations.
- Worked across RL and SFT data pipelines, including task creation, data generation, preference evaluation, calibration, reviewer workflows, and quality validation for computer-use agents.
- Worked with the tooling team on environment and evaluation infrastructure across Linux, Windows, and macOS, contributing to task execution, evaluator workflows, benchmark reliability, and cross-platform automation.
- Took on tooling-engineering and forward-deployed engineering responsibilities, contributing to workflow optimization, evaluation efficiency, automation, and benchmark cost-reduction initiatives.
- Automated generation and validation of task JSON specifications, producing 90-100% instruction/evaluator alignment across generated task specifications and reducing manual effort in benchmark preparation.
- Created benchmark tasks designed around defined single-agent and multi-agent performance thresholds to evaluate frontier systems on complex task execution and multi-agent coordination.
- Took on lead and reviewer responsibilities, reviewing and calibrating benchmark tasks and validating task specifications, evaluation criteria, quality, and scoring consistency.
- Created and evaluated 10 unique long-horizon, greenfield repository tasks, targeting difficult software-engineering scenarios and measurable performance separation between evaluated coding agents.
- Designed high-difficulty tasks targeting a measurable performance separation of approximately 20% between evaluated systems while maintaining challenging, realistic repository-generation requirements.


