Technology AI

Monitoring AI call quality week by week

A weekly routine for reviewing AI-handled plumbing calls: how to build the sample, a scorecard, the metrics to track, thresholds and a 30-minute meeting.

AssistCall research teamPublished Reviewed by Faruque Amin, VP of Sales and Marketing6 min read

On this page
  1. Why weekly, and why a structured sample
  2. Build the weekly sample
  3. The scorecard
  4. How to score one call in three minutes
  5. Weekly metrics alongside the sample
  6. Thresholds that trigger action
  7. Turning findings into fixes
  8. The 30-minute weekly meeting
  9. Compliance checks every week
  10. Re-test after every change
  11. Example: one week's review
  12. Frequently asked questions
  13. References
Technology AI: Monitoring AI call quality week by weekArticle coverTECHNOLOGY AIMonitoring AI callquality week by weekAva for PlumbersInsightsTechnology AI: Monitoring AI call quality week by weekArticle coverTECHNOLOGY AIMonitoring AI callquality week by weekAva for Plumbers

Key takeaways

  • Review every urgent call and a structured weekly sample of the rest, not a random handful.
  • Over-sample nights and weekends: only 30.3% of employed people work on an average weekend day or holiday[1], so fewer people catch errors then.
  • Score each call on the same checklist: hard facts, urgency, outcome, approved answers, tone and compliance.
  • Turn every repeated error into a fix to a rule, knowledge base entry or script, and track it to closure.

Why weekly, and why a structured sample

An AI phone agent that worked well in its first month can drift. Your fees change, a new ZIP code is added, the on-call rota changes, a vendor updates its software, the season turns and callers start describing frozen pipes instead of slow drains. None of that announces itself. A short weekly review catches drift while it is still small.

Listening to a few random calls is not enough. Random samples are dominated by easy daytime calls, which are the least likely to go wrong. A structured sample deliberately includes the calls that carry the most risk, so a 30-minute review covers what actually matters.

Weekly is the right rhythm for most plumbing shops once the first month is over. Daily review suits the first two weeks and any week after a major change. Monthly is too slow: by the time a problem shows up in a monthly review, it has already affected dozens of callers.

Build the weekly sample

For a shop handling a few hundred AI calls a week, a sample of 20 to 30 calls is usually enough. Build it from these groups:

  • Every call marked urgent. These carry the most risk and the most value.
  • A set of routine-marked calls from nights and weekends, to catch emergencies that were missed.
  • Calls in languages other than English, reviewed by a fluent speaker.
  • Calls where the caller asked for a person, to see why.
  • Calls that ended within the first 30 seconds, to check the greeting and early questions.
  • Calls that produced a booking the office later changed or cancelled.
  • A few random daytime calls, as a control.

Over-sample nights and weekends on purpose. Plumbers are often on call for emergencies, so evening and weekend work is common[2], yet on an average Saturday, Sunday or holiday in 2025 only 30.3% of employed Americans worked, against 80.5% on an average weekday[1]. Your office is usually closed then, so mistakes made at the weekend have nobody to catch them until Monday.

Share of employed Americans who worked on an average day, 2025
Share of employed Americans who worked on an average day, 2025Comparison. Average weekday: 80.5%; Average Saturday, Sunday or holiday: 30.3%.80.5%Average weekdayAverage weekday: 80.5%30.3%Average Saturday, Sunday or holidayAverage Saturday, Sunday or holiday: 30.3%Share of employed Americans who worked on an average day, 2025Comparison. Average weekday: 80.5%; Average Saturday, Sunday or holiday: 30.3%.80.5%Average weekdayAverage weekday: 80.5%30.3%Average Saturday, Sunday or holidayAverage Saturday, Sunday or holiday: 30.3%

Source: U.S. Bureau of Labor Statistics, 2025[1]

Show the data
ItemValue
Average weekday80.5%
Average Saturday, Sunday or holiday30.3%

Do not forget non-English calls. In 2019, 39% of people who spoke Spanish at home reported speaking English less than very well[3]. If the agent handles Spanish calls, an English-only reviewer cannot judge them; someone fluent must.

The scorecard

A seven-point scorecard for each reviewed call.
CheckQuestionScore
Hard factsAre address, ZIP, callback number and name correct in the summary?Pass or fail
UrgencyWas the urgency right, judged by your written rules?Pass or fail
OutcomeDid the call end in the right outcome: booked, urgent route, request or handover?Pass or fail
Approved answersDid every answer match the knowledge base, with no guesses?Pass, minor issue, fail
SummaryDoes the summary match the recording, with nothing added or missing?Pass, minor issue, fail
ToneWas the caller treated patiently and clearly?Good, acceptable, poor
ComplianceRecording notice and AI disclosure given at the start?Pass or fail

Keep the scorecard identical from week to week. Changing the questions makes trends meaningless. If you need a new check, add it as an extra line and leave the others alone.

How to score one call in three minutes

  1. Read the summary first

    Note what the office would have done based on it alone.

  2. Check the transcript for hard facts

    Find the address, ZIP and callback number in the transcript and compare them character by character.

  3. Listen to the opening and the close

    The first 30 seconds show the greeting, disclosure and recording notice. The last 30 show what the caller was told would happen.

  4. Skim the middle for the problem

    Confirm the problem and urgency in the caller's words match the summary and the flag.

  5. Score and note

    Fill in the scorecard and write one line on any failure: what happened and which rule or entry caused it.

Weekly metrics alongside the sample

The sample tells you why; a few system-wide numbers tell you how much. Track these every week, from your phone and call records:

  • Calls handled, by hour band: business hours, evenings, nights, weekends.
  • Share of calls ending in a defined outcome, rather than a hang-up or loop.
  • Urgent calls and the time from call start to the on-call tech being reached.
  • Requests for a person, and at what point in the call.
  • Office changes to AI bookings, as a share of AI bookings.
  • Hard-fact error rate from the sample.
  • Complaints or reviews mentioning the phone experience.

Plot each on a simple line over the weeks. A single bad week is noise; three weeks in the same direction is a trend worth acting on.

Thresholds that trigger action

  • Any missed emergency: fix the urgency rule the same day and review all calls of that type for the past week.
  • Any hard-fact error: check the read-back script and the summary for that call type; two in a week means a change is needed.
  • Rising requests for a person at the same point in calls: write an approved answer or a faster handover for that question.
  • Office changes to bookings above your target: tighten the booking rule that produced them.
  • Compliance failure: treat as urgent; check the greeting configuration immediately.

Turning findings into fixes

Most problems fall into one of four types, and each has a different owner. A rule problem, such as the wrong urgency or booking decision, goes to whoever owns the call rules. A knowledge problem, such as a wrong or missing answer, goes to the knowledge base owner. A script problem, such as a confusing question or a weak read-back, goes to whoever writes the call flow. A system problem, such as dropped calls or wrong transfers, goes to the vendor with examples. Log each fix with the date and the calls that prompted it, and check the next week's sample to confirm it worked.

The 30-minute weekly meeting

  1. Five minutes: the weekly metrics and any trend.
  2. Ten minutes: play two or three calls, one good and one or two with problems.
  3. Ten minutes: agree fixes, owners and deadlines.
  4. Five minutes: check last week's fixes worked.

Invite the people closest to the calls: the office manager, a CSR and, every few weeks, an on-call tech. They will spot things a manager listening alone will miss.

Compliance checks every week

Federal law allows recording with one party's consent, but some states require all parties' consent[4]. Check in every sampled call that the recording notice and AI disclosure were given before the caller shared personal details. If the agent makes any outbound calls, such as callbacks, also check that those calls fit your consent rules. A greeting setting changed by accident is easy to miss and costly to discover late.

Re-test after every change

Reviewing live calls catches problems after callers have met them. A short set of standing test calls catches them before. Keep a list of ten scripted scenarios that cover your riskiest situations, and run them after any change to rules, fees, hours, the on-call rota or the vendor's software:

  • A burst pipe at night described without the word emergency.
  • Water near electrics or coming through a light fitting.
  • A caller in your second language.
  • A ZIP code just outside your service area.
  • A request to move an existing appointment.
  • A price question about a large job.
  • A caller who asks for a person straight away.

Record the result of each test alongside the date and the change that prompted it. Over time this becomes a simple regression record: proof that the scenarios that matter most still work, and a quick way to spot which change broke something when one does.

Example: one week's review

Example: in week nine, a shop reviews 24 calls. All four urgent calls reached the on-call tech within two minutes. One Saturday call marked routine described water dripping through a light fitting; the urgency rule only mentioned ceilings, not light fittings. The knowledge base owner adds water near electrics to the urgent list that afternoon. Two Spanish calls scored well. Requests for a person rose slightly, all about invoices, so the office adds an approved answer pointing billing questions to a callback. The next week, the light-fitting scenario is tested and passes.

Frequently asked questions

How many AI calls should we review each week?

For most plumbing shops, 20 to 30 calls in a structured sample, plus every call marked urgent. Review more after any major change.

Who should review AI calls?

The office manager or a senior CSR, with a fluent speaker for non-English calls and an on-call tech joining the review every few weeks.

What is the most important thing to check?

Urgency and hard facts. A missed emergency or a wrong address does real harm; minor wording issues can wait.

Can the vendor do the monitoring for us?

Vendor reports help, but they cannot judge your rules, customers and area as well as your own team. Keep your own weekly review.

References

  1. U.S. Bureau of Labor Statistics (2025). American Time Use Survey, Table 4: Employed persons working and time spent working on days worked, 2025 (opens in a new tab) โ†‘

  2. U.S. Bureau of Labor Statistics (2025). Occupational Outlook Handbook: Plumbers, Pipefitters, and Steamfitters (opens in a new tab) โ†‘

  3. U.S. Census Bureau (2022). Nearly 68 Million People Spoke a Language Other Than English at Home in 2019 (opens in a new tab) โ†‘

  4. Legal Information Institute, Cornell Law School (2026). 18 U.S. Code ยง 2511 - Interception and disclosure of wire, oral, or electronic communications prohibited (opens in a new tab) โ†‘

Book a demo

See Ava handle your calls.

Book a short demo. Weโ€™ll walk through your calls and workflow, and agree what Ava should take care of.

Book a demo