When teams adopt AI quality assurance, the first question is usually "how accurate is the AI?". In practice, how accurately AI scores depends heavily on how well the scorecard is written. An item like "good service attitude" can get different scores from two senior reviewers, so of course AI struggles to judge it consistently.
A good scorecard has three traits: each item describes a behavior you can hear, each judgment can point to a line in the transcript, and the things that truly cannot go wrong are marked as critical items and handled separately.
This guide covers scorecard structure, how to write items, a template, and the calibration process. You do not need to start from scratch; start from the scorecard you use today.
Five layers of a scorecard
| Layer | Meaning | Example |
|---|---|---|
| Section | A stage of the call | Opening, resolution, closing |
| Item | A single behavior to check within a section | Verify the customer's identity |
| Judgment | What counts as passing | Pass, fail, not applicable |
| Weight | How much the item is worth | 10 points |
| Critical item | If failed, the whole call is flagged | Sharing account details before verifying identity |
Sections let reports show who is weakest where: one agent opens well but always forgets to confirm at the end; another resolves issues quickly but skips verification. A total score averages these differences away.
Write items as behaviors you can hear
Like a human reviewer, AI judges from the transcript. If an item has no matching line in the transcript, it is hard to judge consistently.
| Hard to judge | Rewritten as a judgeable behavior |
|---|---|
| Good service attitude | When the customer expresses frustration, the agent acknowledges it before moving into the process |
| Effectively confirmed the need | Before offering a solution, the agent restates the customer's problem in their own words |
| Sufficiently professional | Plan names, terms and conditions the agent quotes match the company's published information |
| Complete closing | Before ending, the agent asks whether there is anything else and thanks the customer |
| Compliant | At the start of the call, the agent says the call is being recorded |
Three rules for rewriting:
Design for "not applicable"
Not every call goes through every step. A customer asking for opening hours never reaches most of the resolution section; a customer who never complains should not fail "calm the customer".
Allow three judgments for every item: pass, fail, not applicable. Exclude not-applicable items from the denominator, otherwise simple calls get marked down for steps that never came up, and the scores stop meaning anything.
How to calculate a weighted score
Weighted score = sum of weights of passed items ÷ sum of weights of applicable items × 100
Example: a scorecard has 6 items weighted 10, 20, 20, 15, 15 and 20, totaling 100. On one call, the 15-point "calm the customer" item is not applicable, the 20-point "confirm the need" item fails, and the rest pass.
Without excluding the not-applicable item, the call would score 65 and look much worse than it was.
Critical items: handled on their own, not averaged
Some mistakes are unacceptable no matter how well the rest of the call went. Mark these as critical items: if one fails, the call goes straight to a manager instead of being averaged away with the other items.
Common critical items:
Keep critical items few. If every item is critical, the review queue grows too long for anyone to read, which is the same as having no critical items.
Template: inbound customer service scorecard
| Section | Item | Weight | Critical |
|---|---|---|---|
| Opening | States the company name and own name | 5 | No |
| Opening | Discloses that the call is recorded | 5 | Yes |
| Resolution | Verifies identity before sharing personal data | 15 | Yes |
| Resolution | Restates the customer's problem before offering a solution | 15 | No |
| Resolution | Acknowledges the customer's frustration when expressed | 10 | No |
| Resolution | Solution and terms match published company information | 20 | No |
| Resolution | Uses no banned phrases | 10 | Yes |
| Closing | Asks whether the customer has any other questions | 10 | No |
| Closing | Explains next steps and timing | 5 | No |
| Closing | Thanks the customer and ends the call | 5 | No |
This is a starting point. Adjust it to your business, regulatory requirements and company policies.
Service scorecard vs telesales scorecard
| Item | Inbound service | Outbound telesales |
|---|---|---|
| Purpose of the call | Solve the customer's problem | Introduce a product, get consent or close |
| Opening focus | Confirm identity and reason for calling | State who is calling and why, ask if it is a good time |
| Core items | Confirming the need, correct solution | Complete product explanation, clear fees and terms |
| Compliance focus | Identity verification, personal data | Required disclosures, improper promises, continuing to pitch after a clear refusal |
| Typical critical items | Personal data leaks | Exaggerated or false promises |
Sales calls usually carry higher compliance risk, so prioritize items such as "were fees and terms explained clearly" and "did the agent keep pitching after a clear refusal". For Taiwan's telemarketing rules, see the AI telemarketing rollout and compliance guide.
Calibration: align AI with your senior reviewers
Do not go live as soon as the scorecard is written. Calibrate it first:
1. Pick a batch of calls: different lines of business, different agents, high and low scores.
2. Score them by hand and by AI: one pass by a senior reviewer, one by AI.
3. Compare item by item: find the items with the most disagreement.
4. Read the quoted lines: because AI judgments quote the transcript, the cause is usually an item description that can be read two ways.
5. Rewrite and score again: until the disagreement is at a level the team accepts.
Calibration often reveals that the original scorecard had no consensus even among human reviewers. That alone is worth the exercise.
Rules change, so version them
Regulations change, products launch and scripts are revised, and the scorecard changes with them. When it does, keep two things in mind:
The scorecard is the foundation of AI QA. Write items as behaviors you can hear, make every judgment point to a line, design for not applicable, mark what truly cannot go wrong as critical, and calibrate against senior reviewers on a batch of calls. Then AI can score every call to your standard.
Pathors Voxcal uses your existing scorecard: every judgment quotes the transcript, calls that fail a critical item go to review automatically, and when rules change they are saved as a new version so past calls can be rescored.
Related reading:
Frequently Asked Questions
How many items should a call QA scorecard have?
There is no fixed number; 10 to 20 is common. More items cover more ground, but each one must be backed by something in the transcript. A shorter list of items that are judged accurately beats a long list of vague ones.
How are critical items different from regular items?
Regular items add up to a score according to their weights. If a critical item fails, the call goes straight to a manager no matter how high the total score is. Use critical items for identity verification, personal data, banned phrases and false promises.
How should not-applicable items be scored?
Exclude them from the denominator. The weighted score is the sum of weights of passed items divided by the sum of weights of applicable items, times 100. That way simple calls are not penalized for steps that never came up.
Can AI QA use our existing scorecard?
Yes, and starting from it is recommended. Pathors Voxcal, for example, uses the company's own scorecard with custom items, weights and critical items. During rollout some items usually need to be rewritten as more concrete behaviors.
After the scorecard changes, can old scores still be compared?
Not directly, because old and new scores use different rules. Save the scorecard as a new version and rescore a past period of calls under the new rules so before and after are comparable.
