Workshop Hiring is looking for a Technical Product Manager for Pedagogy & AI Quality to help ensure Medley’s AI-powered instructional scaffolds consistently meet high standards for learning impact and user experience. In this hybrid Boston role, you will translate research-backed instructional practice into measurable quality targets, strengthen evaluation and observability, and partner closely with engineering to ship improvements.
Medley supports multilingual learners by layering individualized reading and writing scaffolds directly onto digital materials already used by schools, including curriculum platforms, Google Docs, websites, and PDFs. Supports are tailored to a student’s English proficiency level and home language, helping students access grade-level content in real time while staying engaged with the same rigorous material as their peers.
What you will do
- Convert research-based instructional practices in literacy, special education, and multilingual learner instruction into clear, usable quality standards for Medley scaffolds, including targets for AI output quality.
- Build and maintain datasets, rubrics, and evaluation suites, and perform regular error analysis and trace review using a mix of human labeling, LLM-as-judge, and rule-based evaluations.
- Write, test, and refine prompts, run model comparisons and regression tests, and make prompt or model changes to ensure outputs consistently meet target metrics and quality thresholds.
- Review educator signals, support ticket data, and product analytics, and conduct user research with students and teachers to guide product decisions.
- Maintain a prioritized backlog of AI quality and scaffold UX issues; write clear specs and partner with engineering to scope, build, and verify fixes, product updates, and new features.
What you bring
- Pedagogical expertise, especially in scaffolding literacy instruction for struggling readers, special education, and/or multilingual learner education, ideally supported by experience such as classroom teaching, intervention, instructional coaching, or curriculum design.
- Hands-on experience with AI evals, including human labeling, LLM-as-judge rubric design, defining AI metrics targets, and conducting error analysis of LLM outputs.
- AI product fluency, using LLMs daily and designing complex AI systems for products.
- Experience defining new features and collaborating with engineers to deliver them, including clear communication of problems and solutions and comfort with technical tradeoffs and UX questions.
- Strong initiative, sharp judgment, and clear writing, with the ability to act on issues using data and instinct and communicate effectively to both technical and non-technical audiences.
Tools and technologies
- LLM-as-judge
- LangSmith
- Braintrust
- Arize
- Langfuse
- WCAG
Compensation
- USD 130,000 - 150,000 per year
Additional context
This is an opportunity to tighten rubrics for high-quality scaffolds for reading and writing support across multilingual learning and special education, and across subjects. You will expand metrics, strengthen robust eval systems and review processes, and gradually own the quality and usability of the scaffolds suite as it grows to support more students with diverse needs.
Over time, you will work with the head of product and engineering leadership to ensure Medley continues serving teachers and students in partner districts with scaffolds that meet the highest levels of quality. Medley is already used by over 50 districts across the US.
Nice to have
- Graduate degree and/or significant direct experience in special education, reading/literacy, ESL/ELD or bilingual education.
- Familiarity with AI eval and observability tools (for example, LangSmith, Braintrust, Arize, Langfuse) or annotation platforms.
- Proficiency in a language other than English.
- Prior work at an edtech company, or on content quality, assessment design, or psychometrics.
- Familiarity with accessibility standards (for example, WCAG) and assistive technology.
