CAD $80,000–$90,000 / yearView role Source checked 2026-10-29
SPECIALIST CAREERS · CANADA & THE US
AI evaluation & testing jobs
Measure model behavior and test the systems around it. Find LLM evaluation, AI test automation and quality engineering opportunities.
Evaluation designRegression testingPython & test automationExperiment analysis
APPLY DIRECTLY WITH EMPLOYERS
12 current roles
CAD $90,000–$117,000 / yearView role Source checked 2026-10-29
Anthropic Curated from employer
Cyber Evaluations Engineer
United States · Remote-friendly · San Francisco / Washington, DC
USD $300,000–$405,000 / yearView role Source checked 2026-10-30
Anthropic Curated from employer
Research Engineer, Post-Training Model Evaluations
San Francisco / Seattle · Hybrid
USD $500,000–$850,000 / yearView role Source checked 2026-10-30
Anthropic Curated from employer
Performance Engineer, Inference Systems
San Francisco / New York City / Seattle · Hybrid
USD $350,000–$850,000 / yearView role Source checked 2026-10-30
USD $207,000–$385,000 / yearView role Source checked 2026-10-30
Decagon Curated from employer
Research Engineer, Safety
San Francisco / New York City · In-office · Full-time
USD $200,000–$400,000 / yearView role Source checked 2026-10-30
USD $200,000–$250,000 / yearView role Source checked 2026-10-30
Apollo Research Curated from employer
Research Scientist/Engineer (Evaluations)
San Francisco · In-person · London option also listed · Full-time
Pay: see employerView role Source checked 2026-10-30
Baseten Curated from employer
Forward Deployed Engineer (Training)
San Francisco / New York · Hybrid · Full-time
USD $200,000–$400,000 / yearView role Source checked 2026-10-30
LangChain Curated from employer
Deployed Engineer, Professional Services (San Francisco)
San Francisco, California · Hybrid · Up to 20% travel · Full-time
USD $150,000–$300,000 base / yearView role Source checked 2026-10-30
Firecrawl Curated from employer
Agent Experience Engineer
Toronto / San Francisco · Hybrid · 3+ office days · Full-time
CAD $217,000–$233,000 / yearView role Source checked 2026-10-30
No matching roles right now.
Try another skill or remove a filter. We’re starting with a small, focused selection.
Each role shows its source-check date and known location restrictions. A listing can close before its review window ends; confirm current availability with the employer. How we review listings.
UNDERSTAND THE WORK
AI evaluation & testing
career guide.
By TaskBull AIJobs · Updated October 30, 2026
AI evaluation spans research and application quality
Evaluation work asks whether a model or AI application performs well enough for a particular use. Research teams may build measurements for model development. Product teams may test retrieval, agent behavior, APIs and connected workflows. These jobs can share techniques while needing different software, analysis and domain experience. Read what is being evaluated and how the results influence decisions.
Engineering roles are different from annotation tasks
AI evaluator is also used for freelance rating, labeling or human-feedback work. This hub focuses on engineering and related technical evaluation roles in our Canada and US selection. A position building an evaluation platform is not interchangeable with a short-term data-rating task. Check employment type, responsibilities and qualifications before treating a search-title match as the right opportunity.
Build an evaluation project around a decision
Choose a specific question, such as whether a retrieval change improves answer quality without introducing a regression. Explain your test cases, scoring approach, error analysis and limits. Keep a held-out set where appropriate and show how results can be reproduced. For quality engineering, demonstrate how checks fit into delivery and how a failing test leads to investigation rather than a misleading pass/fail label.
Read the tooling, location and experience requirements
Evaluation vacancies can require Python, data analysis, automated testing or research methods. Senior quality roles may include CI/CD, stakeholder ownership and team leadership. Hybrid roles can specify several office days each week, and remote roles still have eligibility restrictions. Use the direct employer application to confirm the latest requirements, compensation and whether sponsorship is available.
Sources and reading
Career guidance draws on the examples below and TaskBull’s editorial analysis. Employer sources describe their own roles; they do not endorse this board.
BEFORE YOU APPLY
Your career
questions.
Are LLM evaluation jobs entry-level?
Some related tasks are, but engineering and research positions can require substantial experience. We preserve the employer’s requirements rather than labeling every evaluation role entry-level.
How are AI testing jobs different from conventional QA?
Many use familiar testing and delivery practices, with additional attention to variable model outputs, retrieval quality and evaluation design. The mix depends on the application and employer.