Ai Evaluation & Testing Product Jobs

1 job found

Today

Technical Product Owner - AI & Agents

Rainforest Atlanta, GA

Atlanta, GA

Today

Full time Agentic Ai Ai Agent Ai Agent Engineering

Apply Now

Job Description

Rainforest is an early stage payments-as-a-service startup building a way to make monetizing payments for vertically focused software platforms fair and simple. On the AI & Agents team, you will shape agentic products for payments programs and help bring agentic workflows into how the company operates.

In this deeply technical role (onsite in Atlanta, GA), you will own key parts of Rainforest’s agent harness and the systems that keep agents accurate and reliable as models, prompts, and routing strategies evolve. You will work hands-on with engineering, drive model selection and evaluation, and represent Rainforest’s agentic products with clients and the broader market.

What you’ll do

Set product direction for the agent harness, focusing on knowledge, skills, and tools that create real value beyond a thin wrapper on an LLM
Define and build evaluations, regression testing, and telemetry to maintain accuracy as model and prompt changes happen
Tune agent behavior in practice, from prompts to knowledge base and skills, working closely with engineering
Drive model selection and routing decisions across multiple LLMs, balancing capability, latency, and cost
Design how client agents interoperate with Rainforest using MCP and agent-to-agent communication
Make product changes directly using AI coding tools, then partner with engineers to review and ship updates
Act as an internal thought leader on AI and agents, helping teams adopt agentic workflows
Represent Rainforest’s agentic products externally with clients and the broader market
Help shape how AI and agents are used across Rainforest by partnering with teams company-wide

What you bring

Hands-on experience building agentic products or agent harnesses, including tool use, memory, prompt assembly, and evaluation
Deep, current knowledge of the LLM landscape, including frontier and open-weight models and where each performs well
Daily fluency with AI coding tools and comfort making real code changes
Experience building evals or regression suites for LLM-based systems
Familiarity with MCP, agent-to-agent protocols, embeddings, and multi-model architectures
A strong technical background from product, engineering, or an adjacent role
Exceptional written and verbal communication, able to explain complex systems to technical and non-technical audiences
Comfort with ambiguity and a high level of autonomy in a fast-moving startup
Payments or fintech experience is a plus

Technologies you’ll work with

LLMs, MCP, agent-to-agent communication
Embeddings, multi-model architectures
AI coding tools

Benefits

Comprehensive health benefits package
Unlimited paid time off
Paid parental leave
Fun and flexible working environment
Continuous investment in people and culture

Rainforest is looking for a deeply technical Product Manager for the AI & Agents team. This is not a traditional PM role; you will help build, evaluate, tune, and ship the agentic systems that power Rainforest’s payments programs.

View full job page →

Get Ai Evaluation & Testing Product Jobs alerts

New Ai Evaluation & Testing Product Jobs, daily.