Metrics Reference

Quantitative measures for AI evaluation. Each metric is designed for interpretability and audit-ready documentation.

Accuracy metrics

Classification accuracy

Measures the percentage of cases where the AI assigned the correct label or level, for example the priority a support assistant assigns to tickets.

Classification Accuracy = (Correct Classifications) / (Total Cases) × 100%

Weighted Accuracy = Σ(weight_i × correct_i) / Σ(weight_i)

Where weights reflect the severity of each level.

Metric	Formula	Interpretation
Raw Accuracy	Correct / Total	Simple correctness rate
Weighted Accuracy	Σ(w × correct) / Σw	Accounts for severity differences
Under-rating Rate	Under-rated / Total	Cases assigned less urgent than needed
Over-rating Rate	Over-rated / Total	Cases assigned more urgent than needed

Under-rating vs over-rating: In most risk-bearing domains, under-rating (missing critical cases) is far more dangerous than over-rating (unnecessary escalations). Weight your metrics accordingly.

Sensitivity and specificity

Critical metrics for evaluating detection of specific conditions or red flags.

Sensitivity (Recall) = True Positives / (True Positives + False Negatives)
"Of all actual positives, how many did we catch?"

Specificity = True Negatives / (True Negatives + False Positives)
"Of all actual negatives, how many did we correctly rule out?"

Positive Predictive Value (PPV) = TP / (TP + FP)
"When we say positive, how often are we right?"

Negative Predictive Value (NPV) = TN / (TN + FN)
"When we say negative, how often are we right?"

Choosing a priority metric

Detection target	Priority metric	Target	Rationale
Account compromise	Sensitivity	≥ 99%	Cannot miss takeovers
Crisis disclosure	Sensitivity	≥ 99%	Safety-critical detection
Prohibited content	Sensitivity	≥ 98%	Policy and legal exposure
Routine follow-up	Specificity	≥ 85%	Avoid unnecessary escalations
Clinical red flags (healthcare vertical)	Sensitivity	≥ 99%	Time-critical intervention

sensitivity_calculation.py

from akhara import Akhara

client = Akhara()

# Get evaluation results with sensitivity/specificity breakdown
results = client.evaluations.get("eval_abc123")

for target in results.detection_targets:
    print(f"""
Target: {target.name}
  Sensitivity: {target.sensitivity:.1%}
  Specificity: {target.specificity:.1%}
  PPV: {target.ppv:.1%}
  NPV: {target.npv:.1%}
  F1 Score: {target.f1:.3f}

  Confusion Matrix:
    TP: {target.true_positives}  FP: {target.false_positives}
    FN: {target.false_negatives}  TN: {target.true_negatives}
""")

Rubric-based scoring

For complex outputs like long-form responses or generated documents, rubric-based scoring provides structured evaluation across multiple dimensions.

rubric_definition.py

rubric = {
    "name": "Support Response Quality",
    "version": "2.1",
    "dimensions": [
        {
            "name": "Factual Accuracy",
            "weight": 0.30,
            "criteria": [
                {"score": 5, "description": "All facts accurate, appropriate terminology"},
                {"score": 4, "description": "Minor terminology issues, facts correct"},
                {"score": 3, "description": "Some inaccuracies, no user harm"},
                {"score": 2, "description": "Multiple inaccuracies, potential confusion"},
                {"score": 1, "description": "Significant errors, harmful guidance"}
            ]
        },
        {
            "name": "Completeness",
            "weight": 0.25,
            "criteria": [
                {"score": 5, "description": "All required elements present, thorough"},
                {"score": 4, "description": "Minor omissions, key info captured"},
                {"score": 3, "description": "Some gaps but usable"},
                {"score": 2, "description": "Missing important elements"},
                {"score": 1, "description": "Critically incomplete"}
            ]
        },
        {
            "name": "Policy Adherence",
            "weight": 0.25,
            "criteria": [
                {"score": 5, "description": "All policy constraints respected, clear disclosures"},
                {"score": 4, "description": "Policy respected, minor disclosure gaps"},
                {"score": 3, "description": "Basic policy compliance"},
                {"score": 2, "description": "Policy gaps present"},
                {"score": 1, "description": "Critical policy violations"}
            ]
        },
        {
            "name": "Actionability",
            "weight": 0.20,
            "criteria": [
                {"score": 5, "description": "Clear next steps, specific instructions"},
                {"score": 4, "description": "Good guidance, minor ambiguity"},
                {"score": 3, "description": "Adequate but could be clearer"},
                {"score": 2, "description": "Vague or confusing recommendations"},
                {"score": 1, "description": "No clear action items"}
            ]
        }
    ],

    "aggregate_method": "weighted_average",
    "passing_threshold": 3.5
}

# Use in evaluation
evaluation = client.evaluations.create(
    name="Support Response Evaluation",
    dataset="ds_support_responses",
    evaluators=[{
        "type": "rubric",
        "config": {"rubric": rubric}
    }]
)

Custom metrics

Define metrics specific to your domain and use cases.

custom_metrics.py

from akhara import Metric, MetricResult

class TopicCoverageMetric(Metric):
    """
    Measures what percentage of the user's raised topics
    were addressed in the AI's response.
    """

    name = "topic_coverage"
    display_name = "Topic Coverage"
    description = "Percentage of raised topics addressed in the response"

    def calculate(self, sample) -> MetricResult:
        raised_topics = set(sample.input.get("topics", []))
        addressed_topics = set(sample.ai_output.get("addressed_topics", []))

        if not raised_topics:
            return MetricResult(
                value=1.0,
                details={"note": "No topics raised"}
            )

        coverage = len(addressed_topics & raised_topics) / len(raised_topics)
        missed = raised_topics - addressed_topics

        return MetricResult(
            value=coverage,
            details={
                "total_topics": len(raised_topics),
                "addressed": len(addressed_topics & raised_topics),
                "missed_topics": list(missed)
            },
            flags=["incomplete_response"] if coverage < 0.8 else []
        )


class TimeToResolutionMetric(Metric):
    """Measures efficiency of the agent's decision-making."""

    name = "time_to_resolution"
    display_name = "Time to Resolution Decision"
    description = "Conversation turns before a resolution decision"

    def calculate(self, sample) -> MetricResult:
        turns = sample.ai_output.get("conversation_turns", 0)
        resolution_turn = sample.ai_output.get("resolution_decision_turn", turns)

        # Benchmark: resolve within 5 turns for most cases
        efficiency_score = max(0, 1 - (resolution_turn - 3) / 10)

        return MetricResult(
            value=efficiency_score,
            details={
                "total_turns": turns,
                "resolution_turn": resolution_turn,
                "benchmark": 5
            }
        )

# Register custom metrics
client.metrics.register(TopicCoverageMetric)
client.metrics.register(TimeToResolutionMetric)

Confidence intervals

All metrics include confidence intervals to quantify uncertainty, especially important for small sample sizes.

results = client.evaluations.get("eval_abc123")

print(f"""
Classification Accuracy: {results.classification_accuracy.value:.1%}
  95% CI: [{results.classification_accuracy.ci_lower:.1%}, {results.classification_accuracy.ci_upper:.1%}]
  Sample Size: {results.classification_accuracy.n}

Sensitivity (Account Compromise): {results.sensitivity_account_compromise.value:.1%}
  95% CI: [{results.sensitivity_account_compromise.ci_lower:.1%}, {results.sensitivity_account_compromise.ci_upper:.1%}]
""")

Sample size matters: For rare cases, you may need larger datasets to achieve narrow confidence intervals. Akhara warns when sample sizes are too small for reliable conclusions.

Metric aggregation

Combine multiple metrics into composite scores for overall model assessment.

Method	Description	Use Case
Weighted Average	Sum of (weight × metric)	General performance score
Minimum	Lowest individual metric	Ensure no weak spots
Geometric Mean	∏(metric)^(1/n)	Balance across metrics
Threshold Gate	Pass only if all thresholds met	Safety-critical deployment

evaluation = client.evaluations.create(
    name="Production Readiness Check",
    dataset="ds_validation",
    evaluators=[...],

    aggregation={
        "method": "threshold_gate",
        "thresholds": {
            "classification_accuracy": {"min": 0.85},
            "sensitivity_critical": {"min": 0.95},
            "hallucination_rate": {"max": 0.01},
            "red_flag_detection": {"min": 0.98}
        },
        "require_all": True  # Must pass ALL thresholds
    }
)

Exporting metrics

Export metrics in formats suitable for compliance documentation, dashboards, or CI/CD pipelines.

# Export for compliance documentation
client.evaluations.export(
    "eval_abc123",
    format="compliance_report",
    output_path="./reports/quality_review.pdf",
    include=[
        "methodology",
        "dataset_description",
        "metric_definitions",
        "results_summary",
        "confidence_intervals",
        "failure_analysis"
    ]
)

# Export for CI/CD
results = client.evaluations.export("eval_abc123", format="json")

# Use in deployment decision
if results["composite_score"] >= 0.9 and results["safety_gate"] == "pass":
    deploy_model()
else:
    block_deployment(reason=results["failure_reasons"])