Python · Java · Kafka · Spark · Hadoop
ABOUT THE ROLE
Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.
WHAT YOU'LL DO
- Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.
- Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.
- Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Build datasets for research and model training in collaboration with research teams.
- Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.
- Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
WHAT WE'RE LOOKING FOR
- Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.
- Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.
- Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.
- Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites