# Research Engineer, Evals at True Talent Solutions Partners

- Company: True Talent Solutions Partners
- Status: Open
- Workplace: On-site
- Location: San Francisco
- Level: Mid-level
- Discipline: AI & ML
- Employment: Full-time
- Posted: 2026-11-04
- Skills: evaluation systems, web data extraction, LLM-ready information
- Apply: https://jobs.ashbyhq.com/true-talent-solutions-partners/14a085ef-388b-48c1-a37a-477255aa5d52

## Description

ABOUT THE ROLE

Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions.


WHAT YOU'LL DO

 - Design metrics for extraction quality across diverse websites, formats, and edge cases.

 - Build scalable pipelines and evaluation harnesses to measure output quality rigorously.

 - Generate and curate ground-truth datasets for trustworthy evaluation.

 - Close the feedback loop by translating evaluation results into model and product improvements.

 - Make quality results clear and actionable for the broader team.


WHAT WE'RE LOOKING FOR

 - Four or more years of relevant engineering experience, including at least two years focused on evaluation work.

 - Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents.

 - Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up.

 - Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.

 - Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.

 - Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisio

More True Talent Solutions Partners roles: https://deviantjobs.com/companies/true-talent-solutions-partners