• Smartsheet is building the next generation of AI-powered work management through SmartAssist, our intelligent agent platform. As we scale from early demos to production-grade agents, quality is the critical frontier and we are looking for a Agent Quality Engineer to own it
• This is not a QA role. It’s a deeply technical, high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation. You will diagnose why our agents fail, design the systems that catch regressions, and drive measurable improvements across our orchestrator and subagent fleet
• You will work closely with our Agent Engineering and AI Platform teams, embedded in a team that has already shipped evaluation infrastructure on Databricks/MLflow and is building toward a mature Agent Development Lifecycle (ADLC)
• Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist’s orchestrator and subagents
• Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
• Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
• Extend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation for production traffic
• Close the feedback loop ensure that every change has a measurable, attributable quality signal
• Collaborate with our Agent Architecture lead to distinguish quality problems that require prompt/context solutions from those that require structural fixes
• Establish repeatable methodology that scales beyond any single agent or subagent
• In your first 6 months, you will have:
• Delivered measurable, validated quality improvement on at least one SmartAssist agent across our defined quality dimensions
• Expanded evaluation coverage to close the most significant blind spots in our current framework
• Established a repeatable quality improvement methodology the broader team can apply
• Why This Role:
• You will have real ownership, not a supporting role on someone else’s roadmap
• SmartAssist is shipping to real users; this work has immediate, visible impact
• You will be part of a fast-moving team that deeply values engineering rigor and actively seeks diverse perspectives
Benefits
• Lucrative Employee Stock Purchase Program (15% discount)
• 401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
• Monthly stipend to support your work and productivity
• Flexible Time Away Program, plus Incidental Sick Leave
• Up to 24 weeks of Parental Leave
• Personal paid Volunteer Day to support our community
• Opportunities for professional growth and development including access to LinkedIn Learning online courses
• Company Funded Perks, including a counseling membership, primary care membership, local retail discounts, and your own personal Smartsheet account
• US employees receive 12 paid holidays per year
• US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
• Teleworking options from any registered location (role specific)
• HSA, 100% employer-paid premiums, or Buy-up medical/vision and dental coverage options for full-time employees
• Equity - Restricted Stock Units (RSUs) with all offers
• Monthly contributions toward your pension
• Mployer-paid Private Medical and Dental, additional cost for family members