Product Owner (Conversational AI)

Pennymac is hiring a Product Owner (Conversational AI) in Carrollton, TX (onsite) to help shape how conversational AI experiences are designed, deployed, and improved across the company. This role leads prompt development and a company-wide prompt and persona strategy, while building the multi-agent orchestration that powers voice AI initiatives across CDL/Sales, Servicing, and Fulfillment.

What you’ll do

• Own the product vision for conversational AI agents, including the company-wide persona, brand voice, and overarching prompt strategy.
• Architect multi-agent orchestration by defining how primary routing agents interpret user intent and hand off work to specialized sub-agents for domain needs such as Sales, Servicing, and Fulfillment. Define logic boundaries, tool access, and context parameters per sub-agent.
• Develop and optimize prompts continuously by building, testing, and tuning complex system prompts, system instructions, and few-shot examples. Optimize prompts across the AWS tech stack to balance conversational quality, token efficiency, and response latency.
• Set up measurable performance tracking with granular evaluation frameworks. Monitor metrics such as hallucination rates, contextual accuracy, task completion and containment, and user drop-off to drive ongoing prompt optimization cycles.
• Coordinate across CDL/Sales, Servicing, and Fulfillment as a central partner to negotiate and refine business requirements, enabling parallel development without technical bottlenecks.
• Define the Golden Dataset of test scenarios that feed engineering automated LLM-as-a-judge evaluation pipelines. Lead structured A/B testing and data analysis to improve agent responses against baseline configurations.
• Drive strategy and ROI by aligning agent performance to key business KPIs, including CSAT and AHT.
• Partner with engineering to integrate prompt designs with core infrastructure, latency management pipelines, and real-time safety guardrails.

What you bring

• 5 to 7 years of Product Management or Product Ownership experience, ideally in mortgage, fintech, or complex customer servicing environments.
• Demonstrated experience building AI/ML products, with a focus on conversational AI, Large Language Models (LLMs), and deep prompt engineering strategies in an enterprise AWS environment.
• Proven ability to design multi-agent or complex state-machine conversational architectures.
• Strong analytical experience, including building effectiveness dashboards, running A/B tests, and using AI-specific metrics such as hallucination tracking and prompt latency.
• Ability to translate complex business logic into precise technical configurations and natural language flows.
• Exceptional leadership and stakeholder management skills.

Tools and technologies

• AWS
• Large Language Models (LLMs)
• LLM-as-a-judge evaluation pipelines
• A/B testing

Benefits

• Comprehensive Medical, Dental, and Vision
• Paid Time Off programs including vacation, holidays, illness, and parental leave
• Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
• Retirement benefits, life insurance, 401k match, and tuition reimbursement
• Philanthropy Programs including matching gifts, volunteer grants, charitable grants, and corporate sponsorships

Compensation and work model

Salary: $90,000 - $150,000 per year
Work Model: OFFICE