T
True Talent Solutions Partners

Research Crawling Engineer

Remote·$160K–250K

web crawling · data acquisition

ABOUT THE ROLE

Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.

WHAT YOU'LL DO

  • Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
  • Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
  • Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
  • Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.
  • Collaborate with research teams to shape datasets for research and model training.
  • Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.
  • Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

WHAT WE'RE LOOKING FOR

  • Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.
  • Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.
  • Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.
  • Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-b
Read the rest on jobs.ashbyhq.com