# Research Crawling Engineer at True Talent Solutions Partners

- Company: True Talent Solutions Partners
- Status: Open
- Workplace: Remote
- Location: Remote
- Level: Mid-level
- Discipline: Data
- Employment: Full-time
- Posted: 2026-11-08
- Skills: Client Roles
- Apply: https://jobs.ashbyhq.com/true-talent-solutions-partners/a085d7b8-e83f-4fa8-8261-32d4bb1fad88

## Description

ABOUT THE ROLE

Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.


WHAT YOU'LL DO

 - Build and maintain web crawlers across social, travel, and multilingual websites.

 - Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.

 - Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.

 - Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

 - Collaborate with research teams to create datasets suited to model training and research needs.

 - Monitor crawl coverage, performance, and data quality, adapting systems as websites change.

 - Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.


WHAT WE'RE LOOKING FOR

 - At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.

 - Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.

 - Strong programming skills in Go, Rust, Python, Java, or C++.

 - Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.

 - Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.

 - Experie

More True Talent Solutions Partners roles: https://deviantjobs.com/companies/true-talent-solutions-partners