Key takeaways
- Review every urgent call and a structured weekly sample of the rest, not a random handful.
- Over-sample nights and weekends: only 30.3% of employed people work on an average weekend day or holiday[1], so fewer people catch errors then.
- Score each call on the same checklist: hard facts, urgency, outcome, approved answers, tone and compliance.
- Turn every repeated error into a fix to a rule, knowledge base entry or script, and track it to closure.
Why weekly, and why a structured sample
An AI phone agent that worked well in its first month can drift. Your fees change, a new ZIP code is added, the on-call rota changes, a vendor updates its software, the season turns and callers start describing frozen pipes instead of slow drains. None of that announces itself. A short weekly review catches drift while it is still small.
Listening to a few random calls is not enough. Random samples are dominated by easy daytime calls, which are the least likely to go wrong. A structured sample deliberately includes the calls that carry the most risk, so a 30-minute review covers what actually matters.
Weekly is the right rhythm for most plumbing shops once the first month is over. Daily review suits the first two weeks and any week after a major change. Monthly is too slow: by the time a problem shows up in a monthly review, it has already affected dozens of callers.
Build the weekly sample
For a shop handling a few hundred AI calls a week, a sample of 20 to 30 calls is usually enough. Build it from these groups:
- Every call marked urgent. These carry the most risk and the most value.
- A set of routine-marked calls from nights and weekends, to catch emergencies that were missed.
- Calls in languages other than English, reviewed by a fluent speaker.
- Calls where the caller asked for a person, to see why.
- Calls that ended within the first 30 seconds, to check the greeting and early questions.
- Calls that produced a booking the office later changed or cancelled.
- A few random daytime calls, as a control.
Over-sample nights and weekends on purpose. Plumbers are often on call for emergencies, so evening and weekend work is common[2], yet on an average Saturday, Sunday or holiday in 2025 only 30.3% of employed Americans worked, against 80.5% on an average weekday[1]. Your office is usually closed then, so mistakes made at the weekend have nobody to catch them until Monday.
Source: U.S. Bureau of Labor Statistics, 2025[1]
Show the data
| Item | Value |
|---|---|
| Average weekday | 80.5% |
| Average Saturday, Sunday or holiday | 30.3% |
Do not forget non-English calls. In 2019, 39% of people who spoke Spanish at home reported speaking English less than very well[3]. If the agent handles Spanish calls, an English-only reviewer cannot judge them; someone fluent must.
The scorecard
| Check | Question | Score |
|---|---|---|
| Hard facts | Are address, ZIP, callback number and name correct in the summary? | Pass or fail |
| Urgency | Was the urgency right, judged by your written rules? | Pass or fail |
| Outcome | Did the call end in the right outcome: booked, urgent route, request or handover? | Pass or fail |
| Approved answers | Did every answer match the knowledge base, with no guesses? | Pass, minor issue, fail |
| Summary | Does the summary match the recording, with nothing added or missing? | Pass, minor issue, fail |
| Tone | Was the caller treated patiently and clearly? | Good, acceptable, poor |
| Compliance | Recording notice and AI disclosure given at the start? | Pass or fail |
Keep the scorecard identical from week to week. Changing the questions makes trends meaningless. If you need a new check, add it as an extra line and leave the others alone.
How to score one call in three minutes
Read the summary first
Note what the office would have done based on it alone.
Check the transcript for hard facts
Find the address, ZIP and callback number in the transcript and compare them character by character.
Listen to the opening and the close
The first 30 seconds show the greeting, disclosure and recording notice. The last 30 show what the caller was told would happen.
Skim the middle for the problem
Confirm the problem and urgency in the caller's words match the summary and the flag.
Score and note
Fill in the scorecard and write one line on any failure: what happened and which rule or entry caused it.
Weekly metrics alongside the sample
The sample tells you why; a few system-wide numbers tell you how much. Track these every week, from your phone and call records:
- Calls handled, by hour band: business hours, evenings, nights, weekends.
- Share of calls ending in a defined outcome, rather than a hang-up or loop.
- Urgent calls and the time from call start to the on-call tech being reached.
- Requests for a person, and at what point in the call.
- Office changes to AI bookings, as a share of AI bookings.
- Hard-fact error rate from the sample.
- Complaints or reviews mentioning the phone experience.
Plot each on a simple line over the weeks. A single bad week is noise; three weeks in the same direction is a trend worth acting on.
Thresholds that trigger action
- Any missed emergency: fix the urgency rule the same day and review all calls of that type for the past week.
- Any hard-fact error: check the read-back script and the summary for that call type; two in a week means a change is needed.
- Rising requests for a person at the same point in calls: write an approved answer or a faster handover for that question.
- Office changes to bookings above your target: tighten the booking rule that produced them.
- Compliance failure: treat as urgent; check the greeting configuration immediately.
Turning findings into fixes
Most problems fall into one of four types, and each has a different owner. A rule problem, such as the wrong urgency or booking decision, goes to whoever owns the call rules. A knowledge problem, such as a wrong or missing answer, goes to the knowledge base owner. A script problem, such as a confusing question or a weak read-back, goes to whoever writes the call flow. A system problem, such as dropped calls or wrong transfers, goes to the vendor with examples. Log each fix with the date and the calls that prompted it, and check the next week's sample to confirm it worked.
The 30-minute weekly meeting
- Five minutes: the weekly metrics and any trend.
- Ten minutes: play two or three calls, one good and one or two with problems.
- Ten minutes: agree fixes, owners and deadlines.
- Five minutes: check last week's fixes worked.
Invite the people closest to the calls: the office manager, a CSR and, every few weeks, an on-call tech. They will spot things a manager listening alone will miss.
Compliance checks every week
Federal law allows recording with one party's consent, but some states require all parties' consent[4]. Check in every sampled call that the recording notice and AI disclosure were given before the caller shared personal details. If the agent makes any outbound calls, such as callbacks, also check that those calls fit your consent rules. A greeting setting changed by accident is easy to miss and costly to discover late.
Re-test after every change
Reviewing live calls catches problems after callers have met them. A short set of standing test calls catches them before. Keep a list of ten scripted scenarios that cover your riskiest situations, and run them after any change to rules, fees, hours, the on-call rota or the vendor's software:
- A burst pipe at night described without the word emergency.
- Water near electrics or coming through a light fitting.
- A caller in your second language.
- A ZIP code just outside your service area.
- A request to move an existing appointment.
- A price question about a large job.
- A caller who asks for a person straight away.
Record the result of each test alongside the date and the change that prompted it. Over time this becomes a simple regression record: proof that the scenarios that matter most still work, and a quick way to spot which change broke something when one does.
Example: one week's review
Example: in week nine, a shop reviews 24 calls. All four urgent calls reached the on-call tech within two minutes. One Saturday call marked routine described water dripping through a light fitting; the urgency rule only mentioned ceilings, not light fittings. The knowledge base owner adds water near electrics to the urgent list that afternoon. Two Spanish calls scored well. Requests for a person rose slightly, all about invoices, so the office adds an approved answer pointing billing questions to a callback. The next week, the light-fitting scenario is tested and passes.
Frequently asked questions
How many AI calls should we review each week?
For most plumbing shops, 20 to 30 calls in a structured sample, plus every call marked urgent. Review more after any major change.
Who should review AI calls?
The office manager or a senior CSR, with a fluent speaker for non-English calls and an on-call tech joining the review every few weeks.
What is the most important thing to check?
Urgency and hard facts. A missed emergency or a wrong address does real harm; minor wording issues can wait.
Can the vendor do the monitoring for us?
Vendor reports help, but they cannot judge your rules, customers and area as well as your own team. Keep your own weekly review.
References
U.S. Bureau of Labor Statistics (2025). American Time Use Survey, Table 4: Employed persons working and time spent working on days worked, 2025 (opens in a new tab) โ
U.S. Bureau of Labor Statistics (2025). Occupational Outlook Handbook: Plumbers, Pipefitters, and Steamfitters (opens in a new tab) โ
U.S. Census Bureau (2022). Nearly 68 Million People Spoke a Language Other Than English at Home in 2019 (opens in a new tab) โ
Legal Information Institute, Cornell Law School (2026). 18 U.S. Code ยง 2511 - Interception and disclosure of wire, oral, or electronic communications prohibited (opens in a new tab) โ