Senior AI Engineer, Applied AI Solutions

Job Description

Position Summary:

We are building an applied AI team that turns business workflows into production agent systems. As a Senior AI Engineer, you will design, build and run multi-agent solutions on Google Cloud, from first prototype to a platform other teams build on. The role needs strong software engineering fundamentals, sound judgment about what a model should and should not do, and the ability to work directly with business stakeholders in ambiguous situations.

Key Responsibilities:

Agent Design and Architecture:

Design and build multi-agent systems with Google Agent Development Kit (ADK), using orchestrator, sub-agent, sequential and agent-as-tool patterns, and deploy them to Vertex AI Agent Engine.

Break business workflows into steps that a model handles and steps that deterministic code handles, so calculations, business rules, permissions and approvals never depend on model output.

Build human-in-the-loop controls, including confidence-based clarification, approval steps and bounded retry loops for actions that are irreversible or high value.

Add grounding and output checks with model callbacks, so figures and facts in responses trace back to tool results and unsupported content is corrected or withheld.

Manage agent state and memory across sessions, including long-term user memory, artifacts and session state, without bloating model context.

Tools, Integration and Data:

Build MCP servers and A2A services that expose enterprise systems to agents, with clear tool schemas, scoped permissions and errors the agent can act on.

Integrate agents with enterprise data and platforms such as BigQuery, Cloud SQL, Cloud Storage, Vertex AI Search, Microsoft 365, SharePoint and Microsoft Graph.

Build retrieval pipelines where needed, combining keyword and semantic search with permission filtering, and measure retrieval quality separately from answer quality.

Select and combine models across providers, including the Gemini family and Claude on Vertex AI Model Garden, with tiering and escalation to balance quality, latency and cost.

Security and Identity:

Design permission-aware systems in which user identity comes from the platform, never from the prompt, and every data access respects the user's entitlements.

Apply least-privilege access with separate deploy and runtime service accounts, Workload Identity Federation, Secret Manager and OAuth.

Defend agents against prompt injection by treating external content as untrusted data, limiting tool privileges and keeping audit logs of agent actions.

Evaluation and Quality:

Build evaluation suites with golden datasets, deterministic assertions, tool-trajectory checks and repeated runs to account for model non-determinism.

Run regression comparisons before every model or prompt change, measuring output quality, format validity, consistency and latency against a baseline.

Enforce quality with automated tests, typed Python, lint rules and evaluation gates in CI, and trace requirements through to tests and acceptance criteria.

Platform, Delivery and Operations:

Build reusable platform components, including project scaffolding, configuration-driven deployment, and shared modules for identity, secrets, observability and memory, so new agents ship faster on a proven pattern.

Operate GitOps deployment pipelines across development, staging and production, with plan-on-pull-request, apply-on-merge, drift detection and safe promotion between environments.

Deploy services on Cloud Run and Vertex AI Agent Engine, using containers, CI/CD and infrastructure as code.

Instrument agents with structured logging, tracing, token and cost tracking, and model usage reporting, and use the data to tune performance and spend.

Stakeholder Partnership and Delivery:

Work directly with business leaders to map workflows, define success metrics, and turn ambiguous requests into written requirements and technical designs.

Write design documents, decision records and deviation logs that explain trade-offs and how the system works around platform constraints.

Lead solutions from prototype through user acceptance testing to production, and hand over with runbooks and operational documentation.

Track the AI ecosystem, run focused experiments with new models, tools and frameworks, and recommend adoption based on evidence.

Qualifications & Experience:

Required:

5+ years of software engineering experience, including 2+ years building and operating LLM-based applications in production.

Strong Python skills, including typed code, automated testing (pytest), Pydantic and modern backend development (APIs, microservices).

Hands-on experience building agentic systems with tool use and multi-step orchestration, using Google ADK, LangGraph, LlamaIndex or a similar framework.

Production experience on Google Cloud, including Vertex AI, Gemini models, Cloud Run, BigQuery, IAM and Secret Manager.

Experience designing evaluation for LLM systems, including golden datasets, automated checks and regression testing across model versions.

Working knowledge of how large language models are trained, evaluated and served, including tokenization, embeddings, fine-tuning, preference tuning and inference settings.

Solid fundamentals in data structures, algorithms, concurrency and failure handling, including retries, idempotency and timeouts.

Experience with Docker, CI/CD pipelines and infrastructure as code.

Ability to translate business problems into technical designs and explain trade-offs clearly to technical and non-technical audiences.

Nice to Have:

Experience with Vertex AI Agent Engine (Agent Platform), Gemini Enterprise and Agent Engine Memory Bank.

Experience building MCP servers or A2A services.

Experience with Claude, Perplexity or other model providers alongside Gemini, and with comparing them on cost, latency and quality.

Experience integrating with Microsoft 365, SharePoint, Microsoft Graph, Salesforce or Snowflake.

Experience with enterprise security patterns, including OAuth, Workload Identity Federation, permission-aware retrieval and audit logging.

TypeScript or Node.js experience, particularly for developer tooling or command line tools.

Experience building internal developer platforms or reusable frameworks adopted by other teams.

Background in agency or cross-functional product environments.

#LI-AB1

#LI-HYBRID

#HM

Horizon Media is proud to be an equal opportunity workplace. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements.

Salary Range $195,000.00 - $225,000.00

A successful applicant’s actual base salary may vary based on factors such as individual’s skill sets, experience, training, education, licensure/certifications, and qualifications for the role. As an organization, we take an aptitude and competency-based hiring approach. We provide a competitive total rewards package including a discretionary bonus and a variety of benefits including health insurance coverage, life and disability insurance, retirement savings plans, company paid holidays and unlimited paid time off (PTO), mental health and wellness resources, pet insurance, childcare resources, identity theft insurance, fertility assistance programs, and fitness reimbursement.