Evaluation And Observability Product Jobs

1 job found

Today

Senior Product Operations Manager, Evaluation

Harvey San Francisco, CA

San Francisco, CA

Today

$150k - $210k/yr Full time Senior Ai Workflows Data Analytics

Apply Now

Job Description

Harvey is hiring a technical, systems-minded Senior Product Operations Manager, Evaluation to build and scale the evaluation engine behind its platform.

Responsibilities

Build and scale the systems that power model and product evaluations across Harvey
Run intake, triage, and prioritization for the evaluation request queue, routing capacity to the highest-value coverage gaps
Embed evaluation workflows and readiness checkpoints into the product development lifecycle
Create and maintain a single source of truth for evaluation status, results, history, and launch readiness
Operationalize expert-designed evaluation methodologies into scalable, repeatable processes
Manage human data providers and stand up an internal contract-attorney pipeline to ensure evaluation quality meets legal standards
Partner with Engineering and Research to improve evaluation tooling, automation, and dashboards
Drive evaluation readiness for major product and model launches across geographies and jurisdictions
Document and operationalize evaluation governance as evaluation complexity increases
Define how Harvey ensures model accuracy, reliability, and trust at global scale

Requirements

4–7+ years of experience in technical program management, product operations, research operations, or evaluation/benchmarking roles
Experience with ML/AI evaluations, benchmarking frameworks, or scientific workflows
Comfort with statistical methodologies and SQL or Python (or similar tools) to interpret evaluation data, natively or with AI tool support
Strong business acumen with an ROI-focused mindset for scaling
Ability to work deeply with legal experts and operationalize complex evaluation methodologies
Strong cross-functional coordination across Product, Engineering, Research, and data providers/vendors
High attention to detail with a bias toward clarity, rigor, and reproducibility
Ability to navigate an evolving landscape and bring order to complex systems
Strong communication skills, including translating technical nuance for diverse stakeholders
Willingness to do whatever it takes to make evaluation systems successful, including documentation and diagnosing pipeline issues

Technologies

SQL
Python

Benefits

Comprehensive health, dental and vision coverage
Retirement benefits: 401k match up to 4%
Flexible PTO
Eligible to participate in the equity plan and benefits program
Offers Equity
Offers Bonus

Location and Employment

San Francisco, CA (hybrid)
Full time
Department: Product

Compensation

USD 150,000–210,000 per year
Offers Equity
Offers Bonus

Harvey is committed to providing reasonable accommodations to applicants with disabilities. Requests can be made by emailing accommodations@harvey.ai.

View full job page →

Get Evaluation And Observability Product Jobs alerts

New Evaluation And Observability Product Jobs, daily.