> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Evaluate a credit card support bot

> Evaluate a customer support bot for a credit card issuer: build a dataset of support conversations, define compliance-aware rubrics, run evaluations, review flagged cases, and gate regressions in CI.

## Scenario

You run a support bot for a credit card issuer. It answers questions about billing disputes, card cancellation, credit limits, and fraud reports. A wrong answer here is not a bad user experience, it is a compliance incident: an invented fee, a missed fraud escalation, or a skipped disclosure can carry regulatory consequences.

This tutorial walks the full loop with the Akhara SDK: dataset, rubrics, evaluation run, expert review, and a CI gate.

| Estimated time | Prerequisites                          | Difficulty   |
| -------------- | -------------------------------------- | ------------ |
| 30 minutes     | `pip install akhara`, `AKHARA_API_KEY` | Intermediate |

The failure modes you evaluate for:

| Failure mode               | Example                                                                   | Severity |
| -------------------------- | ------------------------------------------------------------------------- | -------- |
| Wrong policy answer        | Quotes a 90-day dispute window when the cardholder agreement says 60      | High     |
| Hallucinated terms or fees | Invents a "\$25 expedited dispute fee" that does not exist                | Critical |
| Missed escalation          | Handles a fraud report in-bot instead of routing to the fraud team        | Critical |
| Missing disclosure         | Cancels a card without the required balance and rewards-forfeiture notice | High     |
| Off-policy tone            | Pressures a cardholder out of filing a legitimate dispute                 | Medium   |

## Step 1: Create the project and dataset

Create a project for the bot and a dataset of support conversations. Cover every intent you care about, including adversarial phrasings:

```python title="setup.py" theme={null}
from akhara import Akhara

client = Akhara()  # Uses AKHARA_API_KEY env var

project = client.projects.create(
    name="cardholder-support-bot",
    description="Support bot for credit card issuer: disputes, cancellation, limits, fraud",
)

dataset = client.datasets.create(
    name="cardholder-support-golden",
    project="cardholder-support-bot",
    description="Golden set of support conversations across dispute, cancellation, limit, and fraud intents",
)
```

Aim for a balanced golden set:

| Intent            | Samples | What to include                                                                          |
| ----------------- | ------- | ---------------------------------------------------------------------------------------- |
| Billing disputes  | 40      | Unauthorized charges, duplicate charges, merchant errors, dispute-window edge cases      |
| Card cancellation | 30      | Outstanding balance, rewards forfeiture, retention offers, joint accounts                |
| Credit limit      | 30      | Increase requests, decrease requests, declined requests, income re-verification          |
| Fraud reports     | 30      | Lost card, stolen card, account takeover, transactions the cardholder does not recognize |
| Out of scope      | 20      | Investment advice, other products, questions the bot must refuse or redirect             |

## Step 2: Log support conversations

Log each conversation as a sample. Put the bot's answer in `output` and encode the policy-correct behavior in `expected`:

```python title="log_samples.py" theme={null}
# A billing dispute the bot should handle in-bot
client.samples.create(
    dataset="cardholder-support-golden",
    input={
        "intent": "billing_dispute",
        "conversation": [
            {"role": "customer", "text": "There's a $89.99 charge from a merchant I've never heard of."},
        ],
    },
    output={
        "response": (
            "I can help you dispute that. You have 60 days from the statement date "
            "to file. I've opened a dispute for the $89.99 charge and issued a "
            "temporary credit while we investigate, which typically takes up to "
            "10 business days."
        ),
        "actions": ["open_dispute", "issue_temporary_credit"],
    },
    expected={
        "must_include": ["60 days", "temporary credit"],
        "must_not": ["fee", "charge to dispute"],
        "required_actions": ["open_dispute"],
        "escalate_to_human": False,
    },
    metadata={"intent": "billing_dispute", "policy_version": "cardholder-agreement-2026-01"},
)

# A fraud report the bot must escalate
client.samples.create(
    dataset="cardholder-support-golden",
    input={
        "intent": "fraud_report",
        "conversation": [
            {"role": "customer", "text": "Someone stole my card and there are charges I didn't make."},
        ],
    },
    output={
        "response": (
            "I'm sorry that happened. I've locked your card so no new charges can "
            "go through, and I'm connecting you with our fraud team now. You won't "
            "be held responsible for verified unauthorized charges."
        ),
        "actions": ["lock_card", "escalate_to_fraud_team"],
    },
    expected={
        "must_include": ["locked", "fraud team"],
        "required_actions": ["lock_card", "escalate_to_fraud_team"],
        "escalate_to_human": True,
    },
    metadata={"intent": "fraud_report", "policy_version": "cardholder-agreement-2026-01"},
)
```

The `metadata.policy_version` field ties every sample to the cardholder agreement it was graded against, so a policy update tells you exactly which samples to re-verify.

## Step 3: Define the rubrics

Score five dimensions. Deterministic checks catch the mechanical failures; LLM judges with versioned rubrics grade the policy-sensitive ones:

| Rubric                 | What it checks                                                                         | Evaluator                                  |
| ---------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------ |
| Policy accuracy        | Dispute windows, cancellation terms, limit rules match the cardholder agreement        | `llm_judge` with `card_policy_accuracy_v1` |
| Compliance disclosures | Required notices present (dispute rights, forfeiture on cancellation, fraud liability) | `policy_adherence`                         |
| Escalation             | Fraud and complaint cases route to a human; routine cases stay in-bot                  | Deterministic action check                 |
| Tone                   | Professional, no pressure against legitimate disputes, no blame                        | `llm_judge` with `support_tone_v1`         |
| No hallucinated terms  | No invented fees, rates, timelines, or product terms                                   | `llm_judge` with `no_fabricated_terms_v1`  |

```python title="evaluators.py" theme={null}
evaluators = [
    {
        "type": "contains_all",
        "config": {"fields": ["must_include"]},
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "card_policy_accuracy_v1"},
    },
    {
        "type": "policy_adherence",
        "config": {"policy_set": "card_issuer_disclosures_v2"},
    },
    {
        "type": "escalation_check",
        "config": {
            "expected_field": "expected.escalate_to_human",
            "action_field": "output.actions",
            "escalation_actions": ["escalate_to_fraud_team", "transfer_to_agent"],
        },
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "support_tone_v1"},
    },
    {
        "type": "llm_judge",
        "config": {"rubric": "no_fabricated_terms_v1"},
    },
]
```

<Warning>
  Treat any fabricated fee, rate, or contractual term as a critical failure, not a quality nuance. A bot that invents a "\$25 dispute fee" exposes the issuer to UDAAP claims even if every other answer is perfect.
</Warning>

## Step 4: Run the evaluation

```python title="run_evaluation.py" theme={null}
evaluation = client.evaluations.create(
    name="cardholder-support-regression",
    project="cardholder-support-bot",
    dataset="cardholder-support-golden",
    evaluators=evaluators,
)

results = client.evaluations.wait(evaluation.id, stage="automated")

print(f"Policy accuracy: {results.scores['card_policy_accuracy']}")
print(f"Disclosure adherence: {results.scores['policy_adherence']}")
print(f"Escalation accuracy: {results.scores['escalation_check']}")
print(f"Fabricated terms rate: {results.scores['fabricated_terms_rate']}")
```

Inspect the worst failures directly:

```python theme={null}
failures = results.get_failures(evaluator="llm_judge")
for failure in failures[:5]:
    print(f"Intent: {failure.sample.metadata['intent']}")
    print(f"Response: {failure.sample.output['response']}")
    print(f"Issue: {failure.issue}")
    print("---")
```

## Step 5: Review flagged samples with expert review

Automated judges are good at "the answer omits the 60-day window" and weaker at "this retention script crosses a line." Route the policy-sensitive failures to reviewers with financial services compliance background:

```python title="human_review.py" theme={null}
client.projects.update(
    "cardholder-support-bot",
    human_review={
        "enabled": True,
        "reviewer_pool": "financial_compliance_experts",
        "routing_rules": [
            {
                "condition": "fabricated_terms_detected",
                "action": "route_to_review",
                "priority": "urgent",
            },
            {
                "condition": "escalation_check == 'failed'",
                "action": "route_to_review",
                "priority": "urgent",
            },
            {
                "condition": "card_policy_accuracy < 0.8",
                "action": "route_to_review",
                "priority": "high",
            },
            {
                "condition": "random_sample",
                "rate": 0.05,
                "action": "route_to_review",
            },
        ],
    },
)
```

Reviewers grade flagged conversations in the dashboard at [app.akhara.ai](https://app.akhara.ai) against the same rubrics, and their adjudications feed back into the golden set as new `expected` values. See [Human review design](/evaluation/docs/evaluation-framework/human-review-design) for reviewer calibration and inter-rater reliability.

## Step 6: Gate regressions in CI

Version the suite as YAML and fail the pipeline when a gate breaks. Escalation and fabricated-terms gates are strict floors, not soft targets:

```yaml title="evaluations/cardholder_support_gate.yaml" theme={null}
name: Cardholder Support Gate
version: 1.0.0

evaluators:
  - type: llm_judge
    config:
      rubric: card_policy_accuracy_v1
  - type: policy_adherence
    config:
      policy_set: card_issuer_disclosures_v2
  - type: escalation_check
    config:
      expected_field: expected.escalate_to_human
      action_field: output.actions
      escalation_actions: ["escalate_to_fraud_team", "transfer_to_agent"]
  - type: llm_judge
    config:
      rubric: no_fabricated_terms_v1

gates:
  - name: policy_accuracy_gate
    metric: card_policy_accuracy
    operator: gte
    threshold: 0.90

  - name: disclosure_gate
    metric: policy_adherence
    operator: gte
    threshold: 0.95

  - name: escalation_gate
    metric: escalation_check
    operator: gte
    threshold: 0.99

  - name: fabricated_terms_gate
    metric: fabricated_terms_rate
    operator: lte
    threshold: 0.0

baseline:
  type: production
  model_tag: production-current
  max_regression:
    card_policy_accuracy: 0.02
    policy_adherence: 0.01
```

Run it in your pipeline with the same gate script pattern as the [CI/CD tutorial](/evaluation/docs/tutorials/ci-cd):

```bash theme={null}
pip install akhara pyyaml
python scripts/run_evaluation.py \
  --model-path models/candidate.bin \
  --config evaluations/cardholder_support_gate.yaml \
  --dataset cardholder-support-golden \
  --output evaluation_results.json
```

The script exits non-zero when a gate fails or the candidate regresses past the baseline cap, which blocks the merge. Use `ci_mode` so pending human review does not block the pipeline; require completed review before production promotion instead.

## Target metrics

| Metric                 | Target |
| ---------------------- | ------ |
| Policy answer accuracy | > 90%  |
| Disclosure adherence   | > 95%  |
| Escalation accuracy    | > 99%  |
| Fabricated terms       | 0      |
| Tone score             | > 85%  |

## Next steps

* [Safety gating before production](/evaluation/docs/tutorials/safety-gating): stricter gates for policy-sensitive agents
* [Deploy evaluations in CI/CD](/evaluation/docs/tutorials/ci-cd): the full gate script and GitHub Actions wiring
* [Continuous monitoring in production](/evaluation/docs/tutorials/monitoring): track the same metrics on live traffic