What I Was Trying to Accomplish
Last week, a colleague went on leave and I inherited their shared inbox.
Two hundred unread customer emails. Questions, complaints, follow-ups, a few "just checking in" messages, and a handful that didn't fit any category. My job was to get through them, figure out what was urgent, and summarize the state of things for my manager.
I didn't want to read all 200 emails line by line. I wanted a summary. I wanted to know what was urgent, what was routine, and what could wait.
So I tried three AI-assisted approaches. Here's what happened.
What I Used
Tool: A general-purpose AI assistant (free tier)
Also used: My email client's built-in search and label system
Input: 200 customer emails, exported as plain text
Time available: About 2 hours
Goal: A prioritized summary for my manager, plus a plan for handling the rest
The Safety Concern
Before I start, here's the thing I was worried about.
Customer emails contain names, order numbers, and sometimes personal details. I didn't want to paste raw emails into an AI tool without thinking about it.
So I took a few precautions.
First, I removed names and order numbers before pasting anything into the AI tool. I replaced them with placeholders like [CUSTOMER] and [ORDER].
Second, I didn't paste anything with financial, legal, or medical details. Those I handled manually.
Third, I kept the AI output as a draft. Anything that went to my manager or to a customer was reviewed by me first.
If you're doing something similar, I'd recommend the same precautions. The AI tool doesn't need to know who the customer is. It just needs to know what the issue is.
Approach 1: Summarize Everything at Once
My first approach was the obvious one. I pasted all 200 emails into the AI tool and asked for a summary.
The prompt:
"Summarize these 200 customer emails. Group by category. Identify urgent issues. Keep it under 500 words."
The result:
The AI produced a summary in about 30 seconds. It grouped emails into categories: complaints, questions, follow-ups, and other. It identified "urgent" issues. It kept it under 500 words.
Why it failed:
The summary was too vague. It said things like "several customers reported issues with delivery." But it didn't say which customers, what the issues were, or how urgent they were.
The AI also missed context. One email was from a customer who had already been refunded. The AI flagged it as urgent. Another email was a follow-up on an issue that had been resolved. The AI flagged it as new.
The summary was a starting point. But it wasn't usable.
Approach 2: Summarize in Batches
My second approach was to break the emails into batches.
I grouped them by category first—complaints, questions, follow-ups, other—then pasted each batch separately.
The prompt:
"Summarize these 50 customer emails. List each issue in one sentence. Mark urgent items with a star."
The result:
This was better. The AI produced a list of 50 one-sentence summaries. I could scan them quickly. I could see which ones were urgent.
Why it worked better:
Smaller batches meant the AI had less to process. The output was more specific. The summaries were shorter and easier to scan.
Why it still wasn't enough:
The AI still missed context. It couldn't tell the difference between a new complaint and a follow-up on a resolved complaint. It couldn't tell the difference between an urgent issue and a routine one.
I had to add that context myself. But at least the AI gave me a starting point.
Approach 3: Summarize and Flag
My third approach was the best one.
I pasted batches of emails and asked the AI to summarize, but also to flag anything that needed human review.
The prompt:
"Summarize these emails. For each one, tell me: (1) what the customer wants, (2) whether it's urgent, (3) whether it needs human review. If you're not sure, say 'needs review.'"
The result:
The AI produced a table. Three columns: what they want, urgent, needs review.
It flagged about 40% of the emails as "needs review." That was helpful. It told me where to focus.
Why it worked best:
The AI admitted uncertainty. Instead of guessing, it flagged items it wasn't sure about. That gave me a clear next step.
Why it still needed me:
The AI couldn't tell the difference between a customer who was angry and a customer who was just asking a question. It couldn't tell the difference between a routine request and a serious complaint. I had to read those emails myself.
What the AI Did Well
It was fast. Summarizing 200 emails took about 20 minutes total, including the batch processing.
It grouped by category. Complaints, questions, follow-ups—it sorted them automatically.
It flagged urgency. Not perfectly, but better than nothing.
It gave me a starting point. Instead of reading 200 emails, I read 80 flagged ones.
What Still Needed My Attention
Context. The AI couldn't tell the difference between new issues and resolved ones.
Tone. The AI couldn't tell the difference between an angry customer and a curious one.
Prioritization. The AI flagged too many things as urgent. I had to narrow it down.
Final judgment. Every email that went to my manager was reviewed by me first.
The Math
Without AI: Reading and summarizing 200 emails would have taken about 4 hours.
With AI: Summarizing, flagging, and reviewing took about 2 hours.
Time saved: About 2 hours. Not 4. Just the worst 2 hours—the scanning, the sorting, the blank page.
What I'd Do Differently
Remove more identifiers before pasting. I removed names and order numbers. I should have removed more.
Break into smaller batches. 50 emails per batch worked better than 200 at once.
Ask for flags, not judgments. The AI is better at flagging uncertainty than at making judgment calls.
Would I Trust This Result Without Checking It?
No.
The AI summary was a starting point. It wasn't a final answer. I wouldn't send it to a customer or a manager without reviewing it first.
But it did save me time. It gave me a way to scan 200 emails without reading all 200. It helped me figure out where to focus.
The Part That Worked—and the Part That Didn't
Worked: Batching, flagging, categorizing.
Didn't work: Summarizing everything at once, trusting the AI to make judgment calls.
If you're trying something similar, I'd recommend batching and flagging. Don't trust the AI to decide what's urgent. Use it to help you find where to look.
