# Research Crawling Engineer at True Talent Solutions Partners

- Company: True Talent Solutions Partners
- Status: Open
- Workplace: Remote
- Location: Remote
- Level: Mid-level
- Discipline: Data
- Employment: Full-time
- Salary: $160K–250K
- Posted: 2026-11-12
- Skills: Python, Java, Kafka, Spark, Hadoop
- Apply: https://jobs.ashbyhq.com/true-talent-solutions-partners/d3c16dfd-954d-40a4-a209-ab607cb0bf2b

## Description

ABOUT THE ROLE

Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.


WHAT YOU'LL DO

 - Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.

 - Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.

 - Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.

 - Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

 - Build datasets for research and model training in collaboration with research teams.

 - Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.

 - Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.


WHAT WE'RE LOOKING FOR

 - Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.

 - Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.

 - Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.

 - Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites

More True Talent Solutions Partners roles: https://deviantjobs.com/companies/true-talent-solutions-partners