Evaluate a credit card support bot

Evaluate a customer support bot for a credit card issuer: build a dataset of support conversations, define compliance-aware rubrics, run evaluations, review flagged cases, and gate regressions in CI.

Scenario

You run a support bot for a credit card issuer. It answers questions about billing disputes, card cancellation, credit limits, and fraud reports. A wrong answer here is not a bad user experience, it is a compliance incident: an invented fee, a missed fraud escalation, or a skipped disclosure can carry regulatory consequences.

This tutorial walks the full loop with the Akhara SDK: dataset, rubrics, evaluation run, expert review, and a CI gate.

Estimated time	Prerequisites	Difficulty
30 minutes	pip install akhara, AKHARA_API_KEY	Intermediate

The failure modes you evaluate for:

Failure mode	Example	Severity
Wrong policy answer	Quotes a 90-day dispute window when the cardholder agreement says 60	High
Hallucinated terms or fees	Invents a “$25 expedited dispute fee” that does not exist	Critical
Missed escalation	Handles a fraud report in-bot instead of routing to the fraud team	Critical
Missing disclosure	Cancels a card without the required balance and rewards-forfeiture notice	High
Off-policy tone	Pressures a cardholder out of filing a legitimate dispute	Medium

Step 1: Create the project and dataset

Create a project for the bot and a dataset of support conversations. Cover every intent you care about, including adversarial phrasings:

setup.py

from akhara import Akhara

client = Akhara()  # Uses AKHARA_API_KEY env var

project = client.projects.create(
    name="cardholder-support-bot",
    description="Support bot for credit card issuer: disputes, cancellation, limits, fraud",
)

dataset = client.datasets.create(
    name="cardholder-support-golden",
    project="cardholder-support-bot",
    description="Golden set of support conversations across dispute, cancellation, limit, and fraud intents",
)

Aim for a balanced golden set:

Intent	Samples	What to include
Billing disputes	40	Unauthorized charges, duplicate charges, merchant errors, dispute-window edge cases
Card cancellation	30	Outstanding balance, rewards forfeiture, retention offers, joint accounts
Credit limit	30	Increase requests, decrease requests, declined requests, income re-verification
Fraud reports	30	Lost card, stolen card, account takeover, transactions the cardholder does not recognize
Out of scope	20	Investment advice, other products, questions the bot must refuse or redirect

Step 2: Log support conversations

Log each conversation as a sample. Put the bot’s answer in output and encode the policy-correct behavior in expected:

log_samples.py

# A billing dispute the bot should handle in-bot
client.samples.create(
    dataset="cardholder-support-golden",
    input={
        "intent": "billing_dispute",
        "conversation": [
            {"role": "customer", "text": "There's a $89.99 charge from a merchant I've never heard of."},
        ],
    },
    output={
        "response": (
            "I can help you dispute that. You have 60 days from the statement date "
            "to file. I've opened a dispute for the $89.99 charge and issued a "
            "temporary credit while we investigate, which typically takes up to "
            "10 business days."
        ),
        "actions": ["open_dispute", "issue_temporary_credit"],
    },
    expected={
        "must_include": ["60 days", "temporary credit"],
        "must_not": ["fee", "charge to dispute"],
        "required_actions": ["open_dispute"],
        "escalate_to_human": False,
    },
    metadata={"intent": "billing_dispute", "policy_version": "cardholder-agreement-2026-01"},
)

# A fraud report the bot must escalate
client.samples.create(
    dataset="cardholder-support-golden",
    input={
        "intent": "fraud_report",
        "conversation": [
            {"role": "customer", "text": "Someone stole my card and there are charges I didn't make."},
        ],
    },
    output={
        "response": (
            "I'm sorry that happened. I've locked your card so no new charges can "
            "go through, and I'm connecting you with our fraud team now. You won't "
            "be held responsible for verified unauthorized charges."
        ),
        "actions": ["lock_card", "escalate_to_fraud_team"],
    },
    expected={
        "must_include": ["locked", "fraud team"],
        "required_actions": ["lock_card", "escalate_to_fraud_team"],
        "escalate_to_human": True,
    },
    metadata={"intent": "fraud_report", "policy_version": "cardholder-agreement-2026-01"},
)

The metadata.policy_version field ties every sample to the cardholder agreement it was graded against, so a policy update tells you exactly which samples to re-verify.

Step 3: Define the rubrics

Score five dimensions. Deterministic checks catch the mechanical failures; LLM judges with versioned rubrics grade the policy-sensitive ones:

Rubric	What it checks	Evaluator
Policy accuracy	Dispute windows, cancellation terms, limit rules match the cardholder agreement	llm_judge with card_policy_accuracy_v1
Compliance disclosures	Required notices present (dispute rights, forfeiture on cancellation, fraud liability)	policy_adherence
Escalation	Fraud and complaint cases route to a human; routine cases stay in-bot	Deterministic action check
Tone	Professional, no pressure against legitimate disputes, no blame	llm_judge with support_tone_v1
No hallucinated terms	No invented fees, rates, timelines, or product terms	llm_judge with no_fabricated_terms_v1

evaluators.py

evaluators = [
    {
        "type": "contains_all",
        "config": {"fields": ["must_include"]},
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "card_policy_accuracy_v1"},
    },
    {
        "type": "policy_adherence",
        "config": {"policy_set": "card_issuer_disclosures_v2"},
    },
    {
        "type": "escalation_check",
        "config": {
            "expected_field": "expected.escalate_to_human",
            "action_field": "output.actions",
            "escalation_actions": ["escalate_to_fraud_team", "transfer_to_agent"],
        },
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "support_tone_v1"},
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "no_fabricated_terms_v1"},
    },
]

Treat any fabricated fee, rate, or contractual term as a critical failure, not a quality nuance. A bot that invents a “$25 dispute fee” exposes the issuer to UDAAP claims even if every other answer is perfect.

Step 4: Run the evaluation

run_evaluation.py

evaluation = client.evaluations.create(
    name="cardholder-support-regression",
    project="cardholder-support-bot",
    dataset="cardholder-support-golden",
    evaluators=evaluators,
)

results = client.evaluations.wait(evaluation.id, stage="automated")

print(f"Policy accuracy: {results.scores['card_policy_accuracy']}")
print(f"Disclosure adherence: {results.scores['policy_adherence']}")
print(f"Escalation accuracy: {results.scores['escalation_check']}")
print(f"Fabricated terms rate: {results.scores['fabricated_terms_rate']}")

Inspect the worst failures directly:

failures = results.get_failures(evaluator="llm_judge")
for failure in failures[:5]:
    print(f"Intent: {failure.sample.metadata['intent']}")
    print(f"Response: {failure.sample.output['response']}")
    print(f"Issue: {failure.issue}")
    print("---")

Step 5: Review flagged samples with expert review

Automated judges are good at “the answer omits the 60-day window” and weaker at “this retention script crosses a line.” Route the policy-sensitive failures to reviewers with financial services compliance background:

human_review.py

client.projects.update(
    "cardholder-support-bot",
    human_review={
        "enabled": True,
        "reviewer_pool": "financial_compliance_experts",
        "routing_rules": [
            {
                "condition": "fabricated_terms_detected",
                "action": "route_to_review",
                "priority": "urgent",
            },
            {
                "condition": "escalation_check == 'failed'",
                "action": "route_to_review",
                "priority": "urgent",
            },
            {
                "condition": "card_policy_accuracy < 0.8",
                "action": "route_to_review",
                "priority": "high",
            },
            {
                "condition": "random_sample",
                "rate": 0.05,
                "action": "route_to_review",
            },
        ],
    },
)

Reviewers grade flagged conversations in the dashboard at app.akhara.ai against the same rubrics, and their adjudications feed back into the golden set as new expected values. See Human review design for reviewer calibration and inter-rater reliability.

Step 6: Gate regressions in CI

Version the suite as YAML and fail the pipeline when a gate breaks. Escalation and fabricated-terms gates are strict floors, not soft targets:

evaluations/cardholder_support_gate.yaml

name: Cardholder Support Gate
version: 1.0.0

evaluators:
  - type: llm_judge
    config:
      rubric: card_policy_accuracy_v1
  - type: policy_adherence
    config:
      policy_set: card_issuer_disclosures_v2
  - type: escalation_check
    config:
      expected_field: expected.escalate_to_human
      action_field: output.actions
      escalation_actions: ["escalate_to_fraud_team", "transfer_to_agent"]
  - type: llm_judge
    config:
      rubric: no_fabricated_terms_v1

gates:
  - name: policy_accuracy_gate
    metric: card_policy_accuracy
    operator: gte
    threshold: 0.90

  - name: disclosure_gate
    metric: policy_adherence
    operator: gte
    threshold: 0.95

  - name: escalation_gate
    metric: escalation_check
    operator: gte
    threshold: 0.99

  - name: fabricated_terms_gate
    metric: fabricated_terms_rate
    operator: lte
    threshold: 0.0

baseline:
  type: production
  model_tag: production-current
  max_regression:
    card_policy_accuracy: 0.02
    policy_adherence: 0.01

Run it in your pipeline with the same gate script pattern as the CI/CD tutorial:

pip install akhara pyyaml
python scripts/run_evaluation.py \
  --model-path models/candidate.bin \
  --config evaluations/cardholder_support_gate.yaml \
  --dataset cardholder-support-golden \
  --output evaluation_results.json

The script exits non-zero when a gate fails or the candidate regresses past the baseline cap, which blocks the merge. Use ci_mode so pending human review does not block the pipeline; require completed review before production promotion instead.

Target metrics

Metric	Target
Policy answer accuracy	> 90%
Disclosure adherence	> 95%
Escalation accuracy	> 99%
Fabricated terms	0
Tone score	> 85%

Next steps

• Safety gating before production: stricter gates for policy-sensitive agents
• Deploy evaluations in CI/CD: the full gate script and GitHub Actions wiring
• Continuous monitoring in production: track the same metrics on live traffic