# Research Crawling Engineer at True Talent Solutions Partners

- Company: True Talent Solutions Partners
- Status: Open
- Workplace: Remote
- Location: Remote
- Level: Mid-level
- Discipline: AI & ML
- Employment: Full-time
- Salary: $160K–250K
- Posted: 2026-11-18
- Skills: web crawling, data acquisition
- Apply: https://jobs.ashbyhq.com/true-talent-solutions-partners/a6efae72-b53c-42bb-bae0-228d9c8c97d1

## Description

ABOUT THE ROLE

Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.


WHAT YOU'LL DO

 - Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.

 - Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.

 - Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.

 - Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.

 - Collaborate with research teams to shape datasets for research and model training.

 - Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.

 - Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.


WHAT WE'RE LOOKING FOR

 - Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.

 - Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.

 - Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.

 - Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-b

More True Talent Solutions Partners roles: https://deviantjobs.com/companies/true-talent-solutions-partners
