Applied AI Engineer (AI Agent Evaluation Startup)

A Series A startup that evaluates and monitors AI agents in production is hiring applied AI engineers. The team wants curious, self-directed builders who can tackle open-ended problems and make its analysis agents both more accurate and more scalable across enormous volumes of data.

Your work

Improve retrieval and search so the company’s agents can find the right information fast and reach sound judgments

Create specialized harnesses for inspecting and manipulating agent trajectories, since off-the-shelf tools fall short

Architect an isolated sandbox in which an evaluation agent operates around the clock, powering self-improving autonomous systems for customers

Dig into voluminous, untidy production logs of agent interactions, pulling out evaluation signals and lifting data quality via curation, pipelines and benchmarking

Build agent-based systems that analyze long-running agent behavior in production

Profile

Experience building AI agents and evals, in applied or generative AI rather than a purely ML-centric or academic path

Typically one to four years of industry experience, with an upper bound around ten

Computer science degree; backgrounds from leading universities, top tech companies or high-growth startups are valued

High agency and comfort with ambiguity

Bonus: data curation, benchmarking and data pipeline experience; Python

On-site in the San Francisco Bay Area. Visa transfers are straightforward, and new visa sponsorship is possible for exceptional candidates.