Home/
Part XII — Building Real Products (End-to-End Projects)/35. Project 4: Customer Support Triage Assistant/35.5 Evaluation and privacy controls
35.5 Evaluation and privacy controls
Overview and links for this section of the guide.
On this page
PII Redaction
Customer emails contain credit card numbers, addresses, and phone numbers. Before sending email text to the LLM, run it through a scrubber (like Microsoft Presidio or a simple regex).
Replace `4000-1234-5678-9010` with `[CREDIT_CARD]`. The model doesn't need the number to know it's a billing question.
Evaluation Set
Before launching, export 100 past emails and label them manually. Run your bot on them. - Accuracy: Did it get the category right? - Safety: Did it promise a refund it shouldn't have?
The "Drift" Problem
Customer language changes. A new product launch ("VibeWidget 2000") might confuse the model if it hasn't seen it before. Re-run evals weekly.