Llm Evaluation Product Jobs

1 job found

Today

Sr Product Manager, Technical, Core Shopping Data Science

Amazon.com Services LLC Seattle, WA

Seattle, WA

Today

$151k - $205k/yr Full time Ai Workflows Customer Experience Data Analytics

Apply Now

Job Description

At Amazon.com Services LLC, the Core Shopping Data Science team builds the data-driven foundation for a shopping experience designed to maximize long-term free cash flow. In this role, you will set the quality bar for core shopping experiences and create the measurement and inspection mechanisms that help teams make confident decisions across search and beyond.

You will start with the search results page, then extend quality measurement to the Homepage and Detail Page. The work combines offline and LLM-based evaluation, scalable audit loops, and online integration so quality signals can reach ranking and serving systems, not only periodic reporting.

What you’ll own

Quality standards that every metric definition, annotation SOP, and automated measurement approach must meet before it is trusted or published.
A portfolio of shopping quality metrics aligned to that bar, spanning areas such as relevance, duplication, and brand quality, and expanding as new defect classes are identified.
How quality defects are found, including sampling strategy, audit cadence, anecdote review, and systematic inspection of pages customers actually saw.
Conversion of qualitative signal into quantified metrics, partnering with UX Research and leadership to translate customer anecdotes and research findings into defect definitions that can be sampled, scored, and tracked.
The audit loop and SOP bar: define what gets audited and against which criteria, review findings, and require corrections in annotation SOPs or LLM prompts.
Continuous improvement when measurement misses a defect class: define the corrected standard and drive the team to build to it.
Movement from offline to online quality measurement, enabling quality signals to be available where experiences are ranked and served rather than only in periodic reports.
Partnering with central platform teams to embed quality metrics into online evaluation and serving paths.
Decision enablement for how online quality signals change decisions, including experimentation guardrails, faster detection of quality regressions, and closed loop correction of defective experiences.
A multi-year roadmap for shopping quality measurement and inspection: which defects to measure next, which surfaces to expand to, and what each expansion unblocks for the business.
Extending the charter beyond search by defining defect taxonomy from scratch for surfaces with no existing measurement, including what a quality defect is on that page, how it is sampled, how it is scored, and how it rolls up.
Alignment across teams and surfaces, including Organic Search, Sponsored Products, Sponsored Brands, and International, so evolving definitions remain consistent with the experiences the metrics judge.

Minimum qualifications

5+ years of product or program management, product marketing, business development, or technology experience
Bachelor’s degree
Experience with feature delivery and tradeoffs of a product
Experience owning and driving roadmap strategy and definition
Experience with end to end product delivery
Experience contributing to engineering discussions around technology decisions and strategy related to a product
Experience managing technical products or online services
Experience representing and advocating for a variety of critical customers and stakeholders during executive-level prioritization and planning

Tools and technologies

Tableau
Qlikview
QuickSight
LLM based evaluation
LLM prompts

Compensation and logistics

Location: Seattle, WA (onsite)
Salary: USD 151,200 - 204,600 per year

Benefits

Health insurance (medical, dental, vision, prescription, Basic Life & AD&D, and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
401(k) matching
Paid time off
Parental leave
Sign-on payments and restricted stock units (RSUs)

Preferred qualifications

Experience using analytical tools such as Tableau, Qlikview, or QuickSight
Experience building and driving adoption of new tools

Team context

The Core Shopping Data Science team focuses on the long term and big picture to ensure the full Amazon shopping experience balances strategic trade-offs. Data science supports feature owners and systems through developing and vending metrics, building tools and datasets that inject data into decision-making, and delivering deep analyses to inform high-touch decisions.

View full job page →

Get Llm Evaluation Product Jobs alerts

New Llm Evaluation Product Jobs, daily.