LLM Eval for Cold Email

LLM eval for cold email is a systematic quality check applied to every AI-generated cold email before it reaches a prospect. The eval combines automated rule checks (word count, banned phrases, format) with LLM-as-judge scoring (tone, relevance, naturalness) and human spot-checks. The output is a pass/fail decision per email plus a quality score that tracks agent performance over time.

The principle: a cold email eval should mirror what a skilled SDR manager would check in a 15-second review. Is it short? Is it relevant? Does it sound human? Is the signal real? Is the ask appropriate? Codify that review into automated + LLM + human layers.

The 3-Layer Eval

Layer 1: Automated rule checks (every email, instant)

Layer 2: LLM-as-judge (every email or flagged sample)

Use a second LLM call to rate the email on qualitative dimensions the automated checks can't measure.

Judge prompt:

You are a cold email quality reviewer for B2B SaaS outbound.
Rate this email on 5 dimensions, each 1-5.

Dimensions:
1. RELEVANCE: Does the personalization connect to the
   prospect's actual situation based on the input data?
   (5 = clearly relevant, 1 = generic or irrelevant)

2. NATURALNESS: Does it read like a human peer wrote it,
   not a bot or a marketer?
   (5 = completely natural, 1 = obviously AI-generated)

3. SIGNAL QUALITY: Is the signal (the reason for reaching
   out) specific, recent, and verifiable?
   (5 = specific verifiable signal, 1 = no signal or generic)

4. ASK APPROPRIATENESS: Is the CTA reasonable? (15 min,
   not 45. No "demo". No calendar link in Email 1)
   (5 = perfect ask, 1 = aggressive or inappropriate)

5. OVERALL: Would you send this email as-is to a VP at a
   target account?
   (5 = send immediately, 1 = would never send)

Input data provided to the email writer:
{input_data}

Email to evaluate:
Subject: {subject}
Body: {body}

Respond with JSON:
{
  "relevance": N,
  "naturalness": N,
  "signal_quality": N,
  "ask_appropriateness": N,
  "overall": N,
  "issues": ["list of specific issues found"],
  "pass": true/false (pass if overall >= 4 AND no dimension < 3)
}

Judge rules:

• Use Sonnet for the judge (fast, accurate enough for scoring). Don't use the same model instance that generated the email
• Pass threshold: overall ≥ 4 AND no individual dimension < 3. A 5/5 on naturalness doesn't compensate for a 2/5 on relevance
• Log all judge scores. The trend over time shows whether the generator prompt is improving or degrading
• Calibrate the judge monthly against human ratings. If the judge consistently scores 0.5+ points higher than human reviewers, adjust the judge prompt

Layer 3: Human spot-check (10-20% sample)

A human reviewer checks a random sample of emails that passed Layers 1 and 2.

Human review checklist (15-20 seconds per email):

• [ ] Signal is real (could you verify it with a quick search?)
• [ ] Personalization connects to a relevant problem (not just trivia)
• [ ] Tone sounds like a peer, not a vendor or a bot
• [ ] You would actually send this email to a real prospect
• [ ] No claims about the prospect that aren't in the input data

Human review rules:

• Review 10-20% of emails that passed automated + LLM checks
• Rate each 1-5 on the same dimensions as the LLM judge
• Compare human scores to LLM judge scores. They should agree within ±0.5 on average. If they diverge, recalibrate the judge

Hallucination-Specific Checks

Hallucination is the highest-risk failure mode in AI-generated cold email. One fabricated claim ("congrats on the acquisition" when no acquisition happened) permanently damages the relationship.

Hallucination detection methods

Hallucination rules

• Cross-check every proper noun. If the email mentions a company name, person name, or event that isn't in the input data, it's fabricated. Flag and reject
• Cross-check every number. If the email cites "3x improvement" and the input data contains no such number, it's fabricated
• "Not found" is safer than guessing. If the input data is missing a field, the email should skip that element, not fill it with a plausible-sounding claim
• Hallucination rate target: < 2% per batch. If more than 2% of emails contain fabricated claims, the generator prompt needs anti-hallucination reinforcement

Eval Scoring and Thresholds

Per-email scoring

Automated checks: PASS / FAIL (binary)
  → Must pass all automated checks to proceed

LLM judge score: 1-5 per dimension, 5 dimensions
  → Overall score = average of 5 dimensions
  → Pass: overall ≥ 4.0 AND no dimension < 3.0

Human review (if sampled): 1-5 per dimension
  → Same thresholds as LLM judge

Final decision:
  Automated PASS + Judge PASS → Send
  Automated PASS + Judge FAIL → Regenerate with feedback
  Automated FAIL → Regenerate (don't judge, just fix the rule violation)

Batch-level scoring

Eval-Driven Prompt Improvement

Finding patterns in failures

1. Export all failed emails from the last batch
2. Categorize failures:
   - Rule violations (which rule? how often?)
   - Low relevance scores (is the personalization generic?)
   - Low naturalness scores (what makes it sound like AI?)
   - Hallucinations (what was fabricated? why?)
3. Identify the top 2 failure categories
4. Modify the generator prompt to address those categories
5. Re-run the eval on the golden set
6. Deploy if scores improve without regression

Common failure patterns and prompt fixes

Production Monitoring

Weekly eval dashboard

When to pause and fix

Building the Golden Set for Cold Email

What to include

Golden set rules

• Include the input data alongside each expected output. The eval needs to verify the output against the input. Without input data, accuracy checks are impossible
• Update the golden set quarterly. Add new failure modes, new edge cases, and new examples of ideal output as your ICP and messaging evolve
• Use real prospect data (anonymized if necessary). Fake data ("Acme Corp, 50 employees") produces different outputs than real data ("Ramp, 180 employees, Series B"). Use real data for realistic eval

Measurement

Anti-Pattern Check

• No automated checks before LLM judge. The judge evaluates 200 emails. 30 of them are over word count. The judge spends compute (and money) evaluating emails that should have been caught by a free regex check. Automated checks first, always
• LLM judge uses the same prompt as the generator. The judge and the generator have the same blind spots. Use a different prompt with explicit evaluation criteria. The judge should be a critic, not a clone of the writer
• No hallucination-specific check. The generic quality eval rates an email 4.5/5. But the email says "congrats on the Series C" when the company raised a Series B. Generic quality scoring doesn't catch factual errors. Cross-check every proper noun and number
• Golden set has 5 examples. Not enough to evaluate reliably. 30-40 examples minimum across ideal, good, bad, and edge cases
• Eval exists but nobody reads the results. Weekly dashboard goes to a Slack channel nobody monitors. If the scores are declining and nobody notices for 3 weeks, the eval is theater. Assign an owner. Review weekly
• Same eval since launch, never updated. The ICP shifted. New banned phrases emerged. The messaging changed. The eval criteria must evolve with the product. Update quarterly
• Human spot-check stopped after Month 1. "The agent is good now, we don't need to check." Quality drifts. Prompts degrade. Models update. 10% human spot-check is permanent. Never drop to 0%