Python · TensorFlow · Kubernetes · Docker · Java
ABOUT THE ROLE
Build datasets, evaluation rubrics, and reward signals that help improve large language models. You will work with AI research teams to test data-collection approaches, investigate model failure modes, and develop measurements that inform training runs.
WHAT YOU'LL DO
- Design data slices that reveal meaningful model failure modes across areas such as finance, software, and enterprise workflows.
- Build and refine evaluation rubrics and reward signals for RLHF and RLVR pipelines.
- Model annotator behavior and run experiments to improve model capabilities.
- Develop quantitative methods to measure dataset quality, diversity, and downstream impact.
- Create and manage real-world and synthetic data pipelines.
- Work with research teams to turn training objectives into concrete data and evaluation specifications.
WHAT WE'RE LOOKING FOR
- One to four years of hands-on experience designing machine learning data-collection strategies and evaluation rubrics.
- Familiarity with large language model training and evaluation methods.
- Experience creating data slices to reveal model failure modes across domains such as finance, code, or enterprise workflows.
- Ability to design lightweight experiments, work through ambiguous results, and draw actionable insights.
- A practical, builder-oriented approach and interest in how data quality and structure shape model behavior.
COMPENSATION & BENEFITS
Base salary is $260,000 to $290,000 annually. Visa sponsorship is available.
LOCATION
On-site in San Francisco, California.