Client Roles
ABOUT THE ROLE
Join a growing data infrastructure team building large-scale web data systems for AI research and model development. You will develop reliable crawling and data acquisition infrastructure across diverse websites, helping research teams create datasets for training and evaluating advanced models.
WHAT YOU'LL DO
- Build and maintain web crawlers across social, travel, multilingual, and other domains.
- Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
- Adapt crawling methods to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
- Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Work with research teams to align data collection and dataset curation with modeling needs.
- Monitor crawl coverage, performance, and data quality, and improve systems as websites change.
- Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
WHAT WE'RE LOOKING FOR
- At least 2 years of experience building web crawlers or large-scale data pipelines; 4 or more years is preferred, with flexibility for candidates holding a PhD.
- Experience operating data pipelines at terabyte scale or greater, including distributed systems or parallel processing.
- Strong programming skills in Go, Rust, Python, Java, or C++.
- Deep knowledge of HTTP, networking, browser behavior, and techniques for handling anti-bot systems, rate limits, and dynamic websites.
- Familiarity with headless browsers