AI · language models
ABOUT THE ROLE
Develop benchmarks and evaluation methods for assessing advanced language models. This research and engineering role combines hands-on AI research with building practical evaluation systems, helping shape how models are tested before deployment.
WHAT YOU'LL DO
- Evaluate newly released AI models across a range of capabilities and use cases.
- Build benchmarks from the ground up, including dataset creation and coordination with labelers.
- Develop research findings and contribute to publications on evaluation methods.
- Improve automated methods for evaluating generated text.
- Work with engineering teams to implement and scale evaluation approaches in production systems.
- Collaborate with AI labs and enterprise customers to understand evaluation needs and define research initiatives.
WHAT WE'RE LOOKING FOR
- At least 3 years of full-time AI or machine learning research experience, especially in benchmarking, model evaluation, NLP, or language models.
- A track record of published research, ideally in NLP or benchmarking, alongside an interest in translating research into applied systems.
- Strong Python skills and experience building clean, maintainable software, including HTTP services with Django, Flask, or similar frameworks.
- Hands-on experience with PyTorch or TensorFlow, language modeling, and relevant AI/ML infrastructure and techniques.
- Experience designing benchmarks and evaluation methodologies.
- Strong computer science fundamentals and experience collaborating through Git workflows, code reviews, and cross-function