machine learning · distributed training · inference clusters
ABOUT THE ROLE
Join an AI research team building large-scale models for physical systems, with weather as an initial application. You will develop the infrastructure for distributed model training and inference, helping turn large volumes of observational data into useful predictions and insights.
WHAT YOU'LL DO
- Design, deploy, and maintain distributed machine learning training and inference clusters.
- Build scalable pipelines for petabyte-scale datasets and manage model training across the machine learning lifecycle.
- Test training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
- Profile and debug low-level GPU operations to improve performance.
- Evaluate research and apply relevant techniques to infrastructure and training workflows.
- Analyze training experiments, including results from ablation studies, to inform model development.
WHAT WE'RE LOOKING FOR
- Two to ten years of experience building large-scale machine learning infrastructure for pretrained or foundation models.
- Hands-on experience with compute clusters, data infrastructure, inference systems, and distributed training.
- Strong skills in optimizing large-scale training, data, or inference workloads, including GPU performance debugging.
- Proficiency with distributed training frameworks such as FSDP or DeepSpeed, and experience with CUDA and JAX.
- Experience with Python, C++, Linux, NVIDIA GPUs, and cloud platforms such as GCP, AWS, or Azure.
- Familiarity with Docker, Kubernetes, distributed task management, model ser