Product Quality

Product Quality Jobs

2 jobs found

Today

Senior Product Manager, Search & Recommendations Quality

Expedia Seattle, WA

Seattle, WA

Today

$173k - $277k/yr Full time Manager Ai Workflows Data Analytics

Apply Now

Job Description

Build and scale how Expedia Group measures search and recommendations quality across brands. You will define evaluation strategy, partner closely with ML teams to assess ranking and recommendation tradeoffs, and use experimentation and benchmarking to shape a roadmap grounded in measurable traveler outcomes.

What you’ll work on

Own the product vision and roadmap for search and recommendations quality measurement, translating target outcomes and KPIs (such as relevance, ranking quality, recommendation engagement, and search-to-booking success) into delivery milestones.
Partner with ML Science and ML Engineering to define and evaluate model quality using offline metrics (for example, NDCG@10) and online proxies (for example, Average Booking Position, recommendation click-through, and conversion), enabling technical tradeoff decisions.
Own and scale the evaluation system for search relevance, including a curated query corpus, human relevance annotation, and LLM-as-judge scoring. Report results through the broader Search Health program and extend the approach to recommendation surfaces (such as destination recommendations, cross-sell, and rebooking recommendations).
Define and maintain product metrics for search relevance and recommendation quality, running regular metric reviews and ensuring the roadmap stays accountable to outcomes across both surfaces.
Lead competitive relevance and recommendation benchmarking against key players, translating findings into a prioritized set of investments across product, ranking, and personalization.
Coordinate experimentation and win-loss analysis for ranking, relevance, and recommendation changes, recommending strategies to improve quality across brands.
Coach and influence PMs and stakeholders across the wider Search & Recs organization on measurement best practices.

Skills and qualifications

Bachelor’s degree in a technical, quantitative, or related field (or equivalent practical experience), plus 8+ years of product management experience (or 6+ years with a Master’s).
Experience owning product metrics, experimentation, or evaluation frameworks for ranking, recommendations, personalization, or search systems.
Strong technical depth, including direct partnership with ML Science and ML Engineering on model evaluation and tradeoffs.
Proven ability to design product experiments independently, interpret ranking and relevance (or recommendation engagement) data, and translate insights into a defensible roadmap.
Track record of driving milestone and release planning for complex, cross-functional initiatives, with proactive risk mitigation.
Strong written and verbal communication, including experience presenting to and influencing senior stakeholders.

Tools and methods

NDCG@10
LLM-as-judge

Benefits

Medical, dental, and vision coverage
Paid time off
Employee Assistance Program
Wellness and travel reimbursement
Travel discounts
International Airlines Travel Agent Network (IATAN) membership

Preferred qualifications

Experience with search ranking and/or recommendation systems (such as learning-to-rank models, collaborative filtering, NDCG, engagement and conversion metrics).
Familiarity with LLM-as-judge or other AI-assisted relevance and recommendation evaluation methods.
Experience in marketplace or travel/e-commerce search and recommendations, balancing traveler relevance with supplier and commercial considerations.
Experience running or contributing to competitive benchmarking programs across search and/or recommendation surfaces.

Compensation: USD 173,000 - 277,000 per yearly. Seattle cash range: $173,000.00 to $242,500.00. Starting pay varies by location, available budget, and individual knowledge, skills, and experience. Pay ranges may be modified in the future.

Accommodation: Expedia Group is committed to an inclusive and accessible recruiting experience. If you need an accommodation or adjustment due to a disability during the application or recruiting process, submit a request at https://expedia.service-now.com/askeg?id=job_accommodation.

View full job page →

Today

Sr Product Manager, Technical, Core Shopping Data Science

Amazon.com Services LLC Seattle, WA

Seattle, WA

Today

$151k - $205k/yr Full time Ai Workflows Customer Experience Data Analytics

Apply Now

Job Description

At Amazon.com Services LLC, the Core Shopping Data Science team builds the data-driven foundation for a shopping experience designed to maximize long-term free cash flow. In this role, you will set the quality bar for core shopping experiences and create the measurement and inspection mechanisms that help teams make confident decisions across search and beyond.

You will start with the search results page, then extend quality measurement to the Homepage and Detail Page. The work combines offline and LLM-based evaluation, scalable audit loops, and online integration so quality signals can reach ranking and serving systems, not only periodic reporting.

What you’ll own

Quality standards that every metric definition, annotation SOP, and automated measurement approach must meet before it is trusted or published.
A portfolio of shopping quality metrics aligned to that bar, spanning areas such as relevance, duplication, and brand quality, and expanding as new defect classes are identified.
How quality defects are found, including sampling strategy, audit cadence, anecdote review, and systematic inspection of pages customers actually saw.
Conversion of qualitative signal into quantified metrics, partnering with UX Research and leadership to translate customer anecdotes and research findings into defect definitions that can be sampled, scored, and tracked.
The audit loop and SOP bar: define what gets audited and against which criteria, review findings, and require corrections in annotation SOPs or LLM prompts.
Continuous improvement when measurement misses a defect class: define the corrected standard and drive the team to build to it.
Movement from offline to online quality measurement, enabling quality signals to be available where experiences are ranked and served rather than only in periodic reports.
Partnering with central platform teams to embed quality metrics into online evaluation and serving paths.
Decision enablement for how online quality signals change decisions, including experimentation guardrails, faster detection of quality regressions, and closed loop correction of defective experiences.
A multi-year roadmap for shopping quality measurement and inspection: which defects to measure next, which surfaces to expand to, and what each expansion unblocks for the business.
Extending the charter beyond search by defining defect taxonomy from scratch for surfaces with no existing measurement, including what a quality defect is on that page, how it is sampled, how it is scored, and how it rolls up.
Alignment across teams and surfaces, including Organic Search, Sponsored Products, Sponsored Brands, and International, so evolving definitions remain consistent with the experiences the metrics judge.

Minimum qualifications

5+ years of product or program management, product marketing, business development, or technology experience
Bachelor’s degree
Experience with feature delivery and tradeoffs of a product
Experience owning and driving roadmap strategy and definition
Experience with end to end product delivery
Experience contributing to engineering discussions around technology decisions and strategy related to a product
Experience managing technical products or online services
Experience representing and advocating for a variety of critical customers and stakeholders during executive-level prioritization and planning

Tools and technologies

Tableau
Qlikview
QuickSight
LLM based evaluation
LLM prompts

Compensation and logistics

Location: Seattle, WA (onsite)
Salary: USD 151,200 - 204,600 per year

Benefits

Health insurance (medical, dental, vision, prescription, Basic Life & AD&D, and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
401(k) matching
Paid time off
Parental leave
Sign-on payments and restricted stock units (RSUs)

Preferred qualifications

Experience using analytical tools such as Tableau, Qlikview, or QuickSight
Experience building and driving adoption of new tools

Team context

The Core Shopping Data Science team focuses on the long term and big picture to ensure the full Amazon shopping experience balances strategic trade-offs. Data science supports feature owners and systems through developing and vending metrics, building tools and datasets that inject data into decision-making, and delivering deep analyses to inform high-touch decisions.

View full job page →

Product Quality Jobs

Senior Product Manager, Search & Recommendations Quality

Sr Product Manager, Technical, Core Shopping Data Science

Get Product Quality Jobs alerts