evaluation systems · web data extraction · LLM-ready information
ABOUT THE ROLE
Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions.
WHAT YOU'LL DO
- Design metrics for extraction quality across diverse websites, formats, and edge cases.
- Build scalable pipelines and evaluation harnesses to measure output quality rigorously.
- Generate and curate ground-truth datasets for trustworthy evaluation.
- Close the feedback loop by translating evaluation results into model and product improvements.
- Make quality results clear and actionable for the broader team.
WHAT WE'RE LOOKING FOR
- Four or more years of relevant engineering experience, including at least two years focused on evaluation work.
- Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents.
- Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up.
- Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.
- Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.
- Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisio