Principal ML Engineer

General Information

Locations : Redwood City, California, United States of America Location: Vancouver

State: British Columbia

Country: Canada

Role ID 216087

Worker Type Regular Employee

Studio/Department Other

Work Model Hybrid

Description & Requirements

Electronic Arts creates next-level entertainment experiences that inspire players and fans around the world. Here, everyone is part of the story. Part of a community that connects across the globe. A place where creativity thrives, new perspectives are invited, and ideas matter. A team where everyone makes play happen.

At EA, we believe games are powerful because they bring together multiple ways people engage: play, watch, create, and connect. And increasingly, the biggest entertainment platforms aren't just places to consume content—they're places where communities build.

Creator-made content is already a proven part of EA's history—from community creation tools in Battlefield to The Gallery in The Sims 4. We believe new creative technologies and tools will expand how players engage with and contribute to our experiences, supported by thoughtful product design, safety systems, and global reach. Our focus is on enabling more players to participate in creative expression by making creation easier, safer, and more rewarding.

As a Principal ML Engineer, Training Pipelines, you will own the systems that prepare and deliver multimodal game data for large-scale model training. You will set the architecture for these training input pipelines and improve data throughput, accelerator utilization, reproducibility, and recovery from failures.

Your responsibilities will initially focus on hands-on individual contribution and technical leadership, with potential to take on people-management responsibilities as the team grows. You will partner with researchers and central data and compute teams to turn evolving model requirements into reusable training capabilities.

Responsibilities:

You will architect and build shared training input pipelines from certified game capture to accelerators, covering storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across multiple training frameworks.

You will define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, acceptance criteria, and reproducible splits.

You will build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and corrupt data, including elastic restarts and changes to cluster size or parallelism.

You will design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, titles, and capture conditions, accounting for worker/rank locality and distributed parallelism.

You will benchmark and enforce throughput, data-wait time, step time, and accelerator utilization and stalls. Yu will model FLOPs utilization (MFU) where applicable, per-rank skew, scaling efficiency, restart time, reliability, and cost targets through thousands of accelerators.

You will diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators, including stragglers, faulty hardware, corrupt samples, and imbalanced batches.

You will integrate compilers, runtimes, optimized kernels, low-precision training, and hardware-accelerated media processing. You will maintain efficient data delivery and recovery under FSDP/HSDP, tensor, context, and pipeline parallelism across heterogeneous accelerator environments.

You will lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads. You will translate model and data-access requirements into scalable systems building on shared infrastructure services, while researchers retain model-research ownership.

Qualifications:

20+ years of development experience, including owning production systems for distributed-training input, with measurable improvements in throughput, accelerator utilization, step time, reliability, recovery time, or cost

Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for single jobs spanning thousands of accelerators

Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and optimization on one or more accelerator platforms such as GPUs, TPUs, or AWS Trainium

Pre-training experience and an understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency

Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.

Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, and elastic restart; experience with schemas, dataset versioning, lineage, quality, and reproducible training/evaluation

Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults

Pay Transparency - North America

COMPENSATION AND BENEFITS

The ranges listed below are what EA in good faith expects to pay applicants for this role in these locations at the time of this posting. If you reside in a different location, a recruiter will advise on the applicable range and benefits. Pay offered will be determined based on a number of relevant business and candidate factors (e.g. education, qualifications, certifications, experience, skills, geographic location, or business needs).

PAY RANGES

* British Columbia (depending on location e.g. Vancouver vs. Victoria) *$204,600 - $287,500 CAD

* California (depending on location e.g. Los Angeles vs. San Francisco) *$227,300 - $344,700 USD

Pay is just one part of the overall compensation at EA.

In the US, we offer a package of benefits including paid time off (3 weeks per year to start), 80 hours per year of sick time, 16 paid company holidays per year, 10 weeks paid time off to bond with baby, medical/dental/vision insurance, life insurance, disability insurance, and 401(k) to regular full-time employees. Certain roles may also be eligible for bonus and other incentive programs.

For Canada, we offer a package of benefits including vacation (3 weeks per year to start), 10 days per year of sick time, paid top-up to EI/QPIP benefits up to 100% of base salary when you welcome a