> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Human vs Automated Evaluation

> Understanding when to use automated evaluators versus human review, and how to combine them effectively.

## Overview

Healthcare AI evaluation requires both automated scoring and human clinical judgment. This guide explains when to use each approach and how to build effective hybrid workflows.

## Automated Evaluation

Automated evaluators run code-based scoring on every sample.

### Strengths

<CardGroup cols={2}>
  <Card title="Speed" icon="bolt">
    Process thousands of samples in minutes
  </Card>

  <Card title="Consistency" icon="equals">
    Same criteria applied uniformly
  </Card>

  <Card title="Scale" icon="expand">
    Evaluate 100% of production traffic
  </Card>

  <Card title="Cost" icon="dollar-sign">
    Fraction of human review cost
  </Card>
</CardGroup>

### Best For

| Use Case               | Example                              |
| ---------------------- | ------------------------------------ |
| Structured comparisons | Predicted triage vs. expected triage |
| Pattern matching       | Red flag keyword detection           |
| Compliance checks      | Required questions asked             |
| Metric calculation     | Latency, token count, cost           |
| Regression detection   | Score changes between versions       |

### Example

```python theme={null}
# Automated evaluators for structured checks
evaluators = [
    {
        "type": "triage_accuracy",
        "config": {
            "levels": ["emergency", "urgent", "routine"],
            "require_exact_match": True
        }
    },
    {
        "type": "red_flag_detection",
        "config": {
            "keywords": ["chest pain", "difficulty breathing", "severe bleeding"]
        }
    },
    {
        "type": "response_latency",
        "config": {
            "threshold_ms": 5000
        }
    }
]
```

## Human Review

Human reviewers provide clinical judgment that automated systems cannot replicate.

### Strengths

<CardGroup cols={2}>
  <Card title="Clinical Judgment" icon="stethoscope">
    Nuanced medical reasoning
  </Card>

  <Card title="Context Understanding" icon="brain">
    Interpret ambiguous situations
  </Card>

  <Card title="Edge Cases" icon="diamond">
    Handle novel scenarios
  </Card>

  <Card title="Ground Truth" icon="check-double">
    Generate training labels
  </Card>
</CardGroup>

### Best For

| Use Case                 | Example                                         |
| ------------------------ | ----------------------------------------------- |
| Clinical appropriateness | "Was this triage decision safe?"                |
| Reasoning quality        | "Did the AI ask the right follow-up questions?" |
| Edge cases               | Unusual symptom combinations                    |
| Ambiguous scenarios      | When correct answer is debatable                |
| Ground truth creation    | Labeling data for future automation             |

### Example

```python theme={null}
# Configure human review for complex cases
client.projects.update(
    project="patient-triage",
    review_config={
        "rubric": {
            "dimensions": [
                {
                    "name": "clinical_appropriateness",
                    "description": "Was the triage decision clinically appropriate?",
                    "scale": ["inappropriate", "questionable", "appropriate", "excellent"]
                },
                {
                    "name": "safety",
                    "description": "Were all safety concerns addressed?",
                    "scale": ["unsafe", "partially_safe", "safe"]
                },
                {
                    "name": "communication",
                    "description": "Was the communication clear and empathetic?",
                    "scale": [1, 2, 3, 4, 5]
                }
            ],
            "require_notes_on_failure": True
        }
    }
)
```

## The Hybrid Approach

The most effective strategy combines automated evaluation with targeted human review.

### Workflow

```mermaid theme={null}
flowchart TB
    A[All Samples] --> B[Automated Evaluators]
    B --> C[100% Scored]
    
    C --> D[Flagged Cases<br/>~5-10%]
    C --> E[Random Sample<br/>~5%]
    C --> F[Pass-through<br/>~85%]
    
    D --> G[Human Review<br/>Priority]
    E --> H[Human Review<br/>QA]
```

### Configuration

```python theme={null}
# Configure hybrid workflow
client.projects.update(
    project="patient-triage",
    
    evaluation_config={
        # Automated evaluators run on everything
        "evaluators": [
            {"type": "triage_accuracy"},
            {"type": "red_flag_detection"},
            {"type": "guideline_compliance"}
        ],
        
        # Routing rules for human review
        "routing_rules": [
            # Always review emergency decisions
            {
                "condition": "output.triage_level == 'emergency'",
                "action": "route_to_review",
                "priority": "high",
                "reviewer_credential": "MD"
            },
            
            # Review when AI is uncertain
            {
                "condition": "output.confidence < 0.8",
                "action": "route_to_review",
                "priority": "medium"
            },
            
            # Review mismatches between evaluators
            {
                "condition": "scores.triage_accuracy < 1.0 AND scores.red_flag_detection == 1.0",
                "action": "route_to_review",
                "priority": "medium"
            },
            
            # Review detected safety issues
            {
                "condition": "scores.red_flag_detection < 1.0",
                "action": "route_to_review",
                "priority": "urgent",
                "reviewer_credential": "MD"
            }
        ],
        
        # Random sampling for QA
        "sampling": {
            "routine": 0.05,       # 5% of routine cases
            "urgent": 0.10,        # 10% of urgent
            "emergency": 1.0       # 100% of emergency (covered by rules above)
        }
    }
)
```

## Routing Rules

### Rule Syntax

```python theme={null}
{
    "condition": "<expression>",
    "action": "route_to_review",
    "priority": "low|medium|high|urgent",
    "reviewer_credential": "MD|NP|RN|...",
    "due_within_hours": 24
}
```

### Condition Expressions

| Expression                           | Description                 |
| ------------------------------------ | --------------------------- |
| `output.triage_level == 'emergency'` | Match specific output value |
| `output.confidence < 0.8`            | Numeric comparison          |
| `scores.evaluator_name < threshold`  | Check evaluator score       |
| `'keyword' in output.symptoms`       | Check list membership       |
| `metadata.model_version == 'v2'`     | Match metadata              |
| `condition1 AND condition2`          | Combine conditions          |
| `condition1 OR condition2`           | Either condition            |

### Priority Levels

| Priority | SLA      | Use Case                   |
| -------- | -------- | -------------------------- |
| `urgent` | 1 hour   | Safety-critical issues     |
| `high`   | 4 hours  | Emergency triage decisions |
| `medium` | 24 hours | Standard flagged cases     |
| `low`    | 72 hours | QA random sampling         |

## Assignment Strategies

### Round-Robin

Distribute tasks evenly across available reviewers:

```python theme={null}
"assignment_strategy": {
    "type": "round_robin",
    "respect_credentials": True,
    "max_daily_per_reviewer": 50
}
```

### Load-Balanced

Assign based on current workload:

```python theme={null}
"assignment_strategy": {
    "type": "load_balanced",
    "factors": ["queue_size", "avg_review_time"],
    "max_queue_per_reviewer": 20
}
```

### Expertise-Based

Route to specialists by topic:

```python theme={null}
"assignment_strategy": {
    "type": "expertise",
    "routing": {
        "cardiology_symptoms": ["dr_chen", "dr_patel"],
        "pediatric": ["dr_wilson"],
        "mental_health": ["dr_garcia", "np_thompson"]
    }
}
```

### Dual Review

Require multiple reviewers for critical cases:

```python theme={null}
"assignment_strategy": {
    "type": "dual_review",
    "conditions": {
        "emergency_decisions": 2,
        "safety_flags": 2,
        "default": 1
    },
    "require_agreement": True,
    "tie_breaker": "senior_reviewer"
}
```

## Feedback Loop

Human reviews improve automated evaluation over time.

### Collecting Feedback

```python theme={null}
# When reviewer disagrees with AI
review = client.reviews.create(
    task="task_abc123",
    scores={
        "clinical_appropriateness": "inappropriate",
        "correct_triage": "emergency",  # AI said "urgent"
    },
    notes="Patient described classic ACS symptoms. Should have been triaged as emergency."
)
```

### Training Data Generation

```python theme={null}
# Export reviewed samples for model training
training_data = client.exports.create(
    project="patient-triage",
    filters={
        "has_human_review": True,
        "review_score.clinical_appropriateness": ["appropriate", "excellent"],
        "created_after": "2025-01-01"
    },
    format="jsonl"
)
```

### Evaluator Refinement

```python theme={null}
# Analyze disagreements to improve evaluators
disagreements = client.analytics.get_disagreements(
    project="patient-triage",
    automated_evaluator="triage_accuracy",
    min_count=10
)

for pattern in disagreements:
    print(f"Pattern: {pattern.description}")
    print(f"Count: {pattern.count}")
    print(f"Human usually says: {pattern.human_consensus}")
    print(f"Automated says: {pattern.automated_score}")
```

## Calibration & Agreement

### Inter-Rater Reliability

Monitor agreement between reviewers:

```python theme={null}
# Get agreement metrics
agreement = client.analytics.get_reviewer_agreement(
    project="patient-triage",
    dimension="clinical_appropriateness"
)

print(f"Cohen's Kappa: {agreement.cohens_kappa}")
print(f"Percent Agreement: {agreement.percent_agreement}")
print(f"Fleiss' Kappa: {agreement.fleiss_kappa}")
```

### Calibration Sessions

Run calibration exercises with your review team:

```python theme={null}
# Create calibration set
calibration = client.calibration.create(
    project="patient-triage",
    samples=["smp_1", "smp_2", "smp_3"],  # Carefully selected cases
    reviewers=["rev_a", "rev_b", "rev_c", "rev_d"],
    gold_standard={
        "smp_1": {"triage": "emergency", "rationale": "..."},
        "smp_2": {"triage": "urgent", "rationale": "..."},
        "smp_3": {"triage": "routine", "rationale": "..."}
    }
)

# View calibration results
results = client.calibration.get_results(calibration.id)
for reviewer in results.reviewers:
    print(f"{reviewer.name}: {reviewer.accuracy}% accurate")
```

## Best Practices

<AccordionGroup>
  <Accordion title="Start with More Human Review" icon="users">
    When launching a new AI system, route more cases to human review. As you build confidence, reduce the percentage.

    ```python theme={null}
    # Initial launch: review 30% of cases
    "sampling": {"all": 0.30}

    # After validation: reduce to 10%
    "sampling": {"all": 0.10}

    # Mature system: 5% + flagged only
    "sampling": {"routine": 0.05}
    ```
  </Accordion>

  <Accordion title="Always Review Safety Flags" icon="shield">
    Never skip human review for potential safety issues:

    ```python theme={null}
    # Non-negotiable rule
    {
        "condition": "scores.red_flag_detection < 1.0",
        "action": "route_to_review",
        "priority": "urgent",
        "reviewer_credential": "MD",
        "override_sampling": True  # Always review, ignore sampling rate
    }
    ```
  </Accordion>

  <Accordion title="Use Review Data to Improve" icon="graduation-cap">
    Systematically use human reviews to improve your AI:

    1. Export high-quality reviews as training data
    2. Analyze disagreement patterns
    3. Refine automated evaluator thresholds
    4. Update routing rules based on findings
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Evaluation framework" icon="scale-balanced" href="/evaluation/docs/evaluation-framework/overview">
    Evaluation types, metrics, and rubric design
  </Card>

  <Card title="Human Review Design" icon="clipboard-list" href="/evaluation/docs/evaluation-framework/human-review-design">
    Design effective expert review workflows
  </Card>
</CardGroup>