Large Language Models Product Jobs

1 job found

Today

Product Owner (Conversational AI)

Pennymac Carrollton, TX

Carrollton, TX

Today

$90k - $150k/yr Full time Agentic Ai Ai Workflows Artificial Intelligence

Apply Now

Job Description

Pennymac is hiring a Product Owner (Conversational AI) in Carrollton, TX (onsite) to help shape how conversational AI experiences are designed, deployed, and improved across the company. This role leads prompt development and a company-wide prompt and persona strategy, while building the multi-agent orchestration that powers voice AI initiatives across CDL/Sales, Servicing, and Fulfillment.

What you’ll do

Own the product vision for conversational AI agents, including the company-wide persona, brand voice, and overarching prompt strategy.
Architect multi-agent orchestration by defining how primary routing agents interpret user intent and hand off work to specialized sub-agents for domain needs such as Sales, Servicing, and Fulfillment. Define logic boundaries, tool access, and context parameters per sub-agent.
Develop and optimize prompts continuously by building, testing, and tuning complex system prompts, system instructions, and few-shot examples. Optimize prompts across the AWS tech stack to balance conversational quality, token efficiency, and response latency.
Set up measurable performance tracking with granular evaluation frameworks. Monitor metrics such as hallucination rates, contextual accuracy, task completion and containment, and user drop-off to drive ongoing prompt optimization cycles.
Coordinate across CDL/Sales, Servicing, and Fulfillment as a central partner to negotiate and refine business requirements, enabling parallel development without technical bottlenecks.
Define the Golden Dataset of test scenarios that feed engineering automated LLM-as-a-judge evaluation pipelines. Lead structured A/B testing and data analysis to improve agent responses against baseline configurations.
Drive strategy and ROI by aligning agent performance to key business KPIs, including CSAT and AHT.
Partner with engineering to integrate prompt designs with core infrastructure, latency management pipelines, and real-time safety guardrails.

What you bring

5 to 7 years of Product Management or Product Ownership experience, ideally in mortgage, fintech, or complex customer servicing environments.
Demonstrated experience building AI/ML products, with a focus on conversational AI, Large Language Models (LLMs), and deep prompt engineering strategies in an enterprise AWS environment.
Proven ability to design multi-agent or complex state-machine conversational architectures.
Strong analytical experience, including building effectiveness dashboards, running A/B tests, and using AI-specific metrics such as hallucination tracking and prompt latency.
Ability to translate complex business logic into precise technical configurations and natural language flows.
Exceptional leadership and stakeholder management skills.

Tools and technologies

AWS
Large Language Models (LLMs)
LLM-as-a-judge evaluation pipelines
A/B testing

Benefits

Comprehensive Medical, Dental, and Vision
Paid Time Off programs including vacation, holidays, illness, and parental leave
Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
Retirement benefits, life insurance, 401k match, and tuition reimbursement
Philanthropy Programs including matching gifts, volunteer grants, charitable grants, and corporate sponsorships

Compensation and work model

Salary: $90,000 - $150,000 per year
Work Model: OFFICE

View full job page →

Get Large Language Models Product Jobs alerts

New Large Language Models Product Jobs, daily.