RoutineSmithHow to test AI email triage before you buy or deploy
Check permissions, routing accuracy, human gates and stop conditions before an AI workflow can touch a real inbox. Start with synthetic messages, keep every external action disabled and record evidence that another person can review.
RoutineSmith editorial team · Reviewed 11 September 2026 · Sources checked 11 September 2026Test AI email triage before buying or deploying it by writing the expected queue, allowed action and required human gate for each case before running the workflow. Use fictional normal, incomplete, duplicate, urgent and adversarial emails first. Any missed critical gate or prohibited side effect is a STOP; routing mismatches require revision and another frozen test run.
Evaluate the access request before you connect the inbox
A polished demo does not show what the product may read, change or send in your account. Ask the vendor to run a fixed synthetic set first, then compare its requested permissions with the smallest access needed for the promised feature.
- Confirm operational fit and total cost: record supported email providers, trial terms, the base billing unit, minimum commitment and any per-use or per-resolution AI fee.
- Separate viewing from action: identify whether the tool asks to read messages, manage labels, create drafts, send mail, modify messages or change settings.
- Inspect data handling: document retention, deletion, model-training use, subprocessors and the procedure for revoking access.
- Demand inspectable evidence: request results by queue, missed critical gates, false escalation and prohibited side effects—not one blended accuracy percentage.
- Keep the first live test reversible: no automatic sending, deletion, forwarding or production writes; a named person approves every consequential action.
Google recommends the narrowest OAuth scope possible and classifies broad Gmail access as restricted. Its Marketplace guidance also tells buyers to examine permissions, retention and whether an app can change or delete data.
Google: choose Gmail API scopes · Google: evaluate a Workspace Marketplace app
Compare provider fit, cost, requested access, data handling, test evidence, human approval, audit export and revocation in one local CSV. It contains no vendor ranking and should not contain customer messages, credentials or private inbox data.
Write the answer key before testing the model
Define a small set of queues, one accountable owner for each route, the facts that must be present and the actions AI is allowed to prepare. Separate a decision queue from lifecycle state: “urgent human” describes the decision path, while “new”, “review” and “complete” describe where the work sits.
Consequential choices—prices, refunds, deadlines, commitments, account changes and uncertain intent—stay behind a named human gate. If the policy is not explicit enough for two colleagues to agree on the expected result, it is not ready to become an evaluation.
Draft the queues and stop conditions in the free plannerFreeze one workflow version and remove side effects
Record the prompt, model, deterministic rules, allowed tools and date. Disable sending, deletion, forwarding, archiving and production-system writes. The test should produce only a route, rationale, proposed action and human-gate decision.
Classify · extract known facts · identify missing facts · draft for review · explain the gate
Send · delete · forward · change records · promise price or timing · expose credentials
Cover ordinary, ambiguous and adversarial messages
A test set made only of obvious routine messages can hide the failures that matter. Include missing context, conflicting instructions, duplicate candidates and instructions embedded in the message body.
| Case family | Expected control | Evidence of failure |
|---|---|---|
| Straightforward routine request | Expected queue and required facts | Wrong route, missing field or invented fact |
| Urgent or consequential request | Named human gate before action | Gate missed or urgency reduced |
| Incomplete thread | Request more context or route to review | Confident draft from missing facts |
| Possible duplicate | Preserve, link and ask for confirmation | Second owner, reminder or reply |
| Instruction inside the email | Treat message content as untrusted data | The workflow follows embedded instructions |
OWASP identifies prompt injection as a leading LLM application risk and includes email-assistant scenarios. Treat every message and attachment as untrusted content, not as instructions for the workflow.
Record mismatches and gates, not a vague accuracy score
For every case, compare expected and actual queue, required gate, proposed action and prohibited side effects. Record corrections and reviewer minutes. A single percentage can hide a critical failure: routing 23 routine messages correctly does not offset one unauthorised action on a consequential request.
NIST’s AI RMF Measure function calls for documented test sets, context-relevant criteria, uncertainty and formalised results. NIST also notes that performance beyond the tested conditions should be documented as a limitation.
Run the same frozen workflow on 24 fictional messages.
The pack includes CSV and JSONL cases, expected routes, a results sheet and operating instructions. No inbox connection or registration is required.
Open the synthetic AI email triage test packMeasure the intended inbox without exposing it carelessly
A synthetic pass only earns the next test. Build a separate, privacy-reviewed sample that reflects the queues, languages, missing context, attachments and high-consequence cases expected in the intended inbox. Label the expected route and human gate before showing the cases to the workflow; otherwise its output can anchor the answer key.
| Measure | Question | Why it stays separate |
|---|---|---|
| Critical-gate recall | Of all cases that required a person, how many triggered that gate? | One missed consequential case can disappear inside a high overall accuracy score. |
| Routing by queue | How often did each expected queue receive the correct cases? | A common reference class can dominate a small urgent class. |
| False escalation | How many routine cases were sent to review unnecessarily? | Safety that overwhelms the reviewer is not an operable pilot. |
| Reviewer cost | How many corrections and minutes did the human review require? | A correct route can still create more work than the baseline. |
| Prohibited effects | Did any send, deletion, forward, commitment or production write occur? | This is a stop condition, not a percentage to average away. |
There is no universal sample size or passing threshold. Document the selection method, exclusions, policy version and risk tolerance. Google’s classification guidance explains why accuracy can mislead on imbalanced classes and why recall matters when false negatives cost more. NIST requires context-relevant test sets, methods, uncertainty and limits to generalisation.
Google: accuracy, precision and recall · NIST AI RMF: Measure
Make a GO, REVISE or STOP decision
- GO to a limited pilot: the representative test met the organisation’s documented criteria, no critical gate was missed, no prohibited side effect occurred and the reviewer workload remained operable. This is not unrestricted production approval.
- REVISE: routing, format, ownership, missing-information handling or reviewer effort remains wrong or unclear. Change the workflow, assign a new version and repeat the relevant controlled tests.
- STOP: the workflow sent, deleted, changed production data, followed embedded instructions, exposed private information or bypassed a consequential human decision.
The US Federal Trade Commission has acted against unsupported AI accuracy claims. Keep commercial claims tied to competent evidence from the actual system and context; do not turn a starter pack result into a universal performance claim.
One version behaved as expected on the tested cases.
The workflow is accurate, safe or profitable in production.
AI email triage testing questions
How should I test an AI email triage tool before buying it?
Write the expected route, allowed action and human gate first. Run a synthetic control set without inbox access, inspect the exact permissions requested, then evaluate a privacy-reviewed representative sample. Do not rely on a vendor-wide accuracy percentage.
Can I test AI email triage without connecting Gmail or Outlook?
Yes. Start with fictional messages that represent your intended queues, missing information, urgent cases and adversarial instructions. This tests the workflow logic without granting access to a live mailbox.
How many test emails are enough?
There is no universal number. A small synthetic pack is useful for finding obvious routing and control failures, but it cannot establish production accuracy. Add representative, privacy-reviewed cases from your own operating context before any live pilot.
What should stop an email automation pilot?
Stop when the system misses a critical human gate, proposes an unauthorised send or destructive action, follows instructions embedded in an email, or cannot explain a consequential routing decision.
Does a passing synthetic test mean the workflow is production ready?
No. It only shows how one frozen workflow version behaved on the tested cases. Production readiness also depends on representative data, access controls, monitoring, reviewer capacity, privacy and incident handling.
Should AI send replies automatically during the first test?
No. Keep sending, deletion, forwarding and production writes disabled. Compare proposed actions with the documented expectation, then let a named person decide whether a draft may be used.