Head of Quality, Technical Staff (AI Training Data Startup)

A young, quick-moving applied research startup produces realistic training environments and datasets for advanced AI models. It needs a hands-on quality owner who decides what is good enough to ship, builds the automated systems that enforce that standard, and later assembles the team that works alongside them.

What you’ll own

Personally examine outputs from people-generated data and from environments, spending much of your time inside the data itself

Create automated evaluators, QA agents and scoring pipelines to detect reward hacking, faulty verifiers and mismatched data distributions ahead of delivery

Set measurable quality bars and pass criteria covering environments, tasks, verifiers, reward signals and datasets

Sign off on deliverables for engagements with leading AI labs, cloud providers and large enterprises

Recruit, onboard and manage quality reviewers and QA engineers while the group expands

Qualifications

One to five years as an LLM evaluation engineer, QA engineer or comparable role

Experience creating automated evaluators, QA tooling or data-quality systems for ML training data, agents or large pipelines

Technical understanding of what gives a dataset or environment real learning signal

Practice reviewing LLM or agent outputs and traces by hand, along with evals, RL environments or annotation pipelines

Ability to write the automation yourself: graders, judge models, QA agents and quality dashboards

Exacting attention to detail

Plus: past founder or zero-to-one experience

On-site in the San Francisco Bay Area. Visa transfers are considered.