> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# AI / ML Metrics Definitions

> Definitions of machine learning metrics, evaluation measures, and performance indicators used in healthcare AI assessment

# AI / ML Metrics Definitions

Comprehensive reference for machine learning and evaluation metrics used in Akhara's healthcare AI assessment platform.

## Classification Metrics

<ResponseField name="Accuracy" type="metric">
  The proportion of correct predictions among the total number of cases examined.

  ```
  Accuracy = (TP + TN) / (TP + TN + FP + FN)
  ```

  **Healthcare Context**: Overall correctness of triage classifications. Note: Can be misleading with imbalanced classes (e.g., rare conditions).

  **Typical Range**: 0.0 - 1.0 (or 0% - 100%)
</ResponseField>

<ResponseField name="Precision" type="metric">
  The proportion of positive identifications that were actually correct. Also called Positive Predictive Value (PPV).

  ```
  Precision = TP / (TP + FP)
  ```

  **Healthcare Context**: When the AI flags a case as urgent, how often is it actually urgent? High precision reduces unnecessary escalations.

  **Typical Range**: 0.0 - 1.0
</ResponseField>

<ResponseField name="Recall (Sensitivity)" type="metric">
  The proportion of actual positives that were identified correctly. Also called True Positive Rate (TPR).

  ```
  Recall = TP / (TP + FN)
  ```

  **Healthcare Context**: Of all truly urgent cases, how many did the AI correctly identify? Critical for patient safety, missing urgent cases (false negatives) can be dangerous.

  **Typical Range**: 0.0 - 1.0 (aim for >0.95 in safety-critical applications)
</ResponseField>

<ResponseField name="Specificity" type="metric">
  The proportion of actual negatives that were identified correctly. Also called True Negative Rate (TNR).

  ```
  Specificity = TN / (TN + FP)
  ```

  **Healthcare Context**: Of all non-urgent cases, how many were correctly identified as non-urgent? Helps assess over-triage rates.

  **Typical Range**: 0.0 - 1.0
</ResponseField>

<ResponseField name="F1 Score" type="metric">
  The harmonic mean of precision and recall, providing a single score that balances both metrics.

  ```
  F1 = 2 × (Precision × Recall) / (Precision + Recall)
  ```

  **Healthcare Context**: Useful when you need to balance finding urgent cases (recall) with not over-escalating (precision).

  **Typical Range**: 0.0 - 1.0
</ResponseField>

<ResponseField name="F-beta Score" type="metric">
  Generalization of F1 that allows weighting recall vs precision.

  ```
  Fβ = (1 + β²) × (Precision × Recall) / (β² × Precision + Recall)
  ```

  **Healthcare Context**: Use β > 1 (e.g., F2) when recall is more important (catching all urgent cases). Use β \< 1 (e.g., F0.5) when precision is prioritized.

  **Common Values**: F0.5, F1, F2
</ResponseField>

***

## Probabilistic Metrics

<ResponseField name="AUC-ROC" type="metric">
  Area Under the Receiver Operating Characteristic Curve. Measures the ability to distinguish between classes across all classification thresholds.

  **Healthcare Context**: How well can the model separate urgent from non-urgent cases, regardless of the specific threshold chosen?

  **Interpretation**:

  * 1.0: Perfect discrimination
  * 0.9-1.0: Excellent
  * 0.8-0.9: Good
  * 0.7-0.8: Fair
  * 0.5: No discrimination (random)
</ResponseField>

<ResponseField name="AUC-PR" type="metric">
  Area Under the Precision-Recall Curve. More informative than AUC-ROC for imbalanced datasets.

  **Healthcare Context**: Preferred for rare conditions or uncommon urgency levels where the positive class is much smaller than the negative class.

  **Typical Range**: 0.0 - 1.0 (baseline depends on class prevalence)
</ResponseField>

<ResponseField name="Calibration" type="metric">
  How well predicted probabilities match actual outcomes. A model predicting 70% confidence should be correct 70% of the time.

  **Healthcare Context**: Critical for clinical decision support, clinicians need to trust that confidence scores are meaningful.

  **Measurement**: Calibration curves, Expected Calibration Error (ECE), Brier Score
</ResponseField>

<ResponseField name="Brier Score" type="metric">
  Mean squared difference between predicted probabilities and actual outcomes.

  ```
  Brier Score = (1/N) × Σ(predicted_prob - actual_outcome)²
  ```

  **Healthcare Context**: Lower is better. Captures both calibration and discrimination.

  **Typical Range**: 0.0 (perfect) - 1.0 (worst)
</ResponseField>

<ResponseField name="Log Loss (Cross-Entropy)" type="metric">
  Measures the performance of a classification model where the prediction is a probability.

  ```
  Log Loss = -(1/N) × Σ[y×log(p) + (1-y)×log(1-p)]
  ```

  **Healthcare Context**: Penalizes confident wrong predictions heavily, important when AI expresses high certainty.

  **Typical Range**: 0.0 (perfect) - ∞
</ResponseField>

***

## Healthcare-Specific Metrics

<ResponseField name="Under-triage Rate" type="metric">
  Percentage of cases where the AI assigned a lower urgency than appropriate.

  ```
  Under-triage Rate = Under-triaged Cases / Total High-Acuity Cases
  ```

  **Healthcare Context**: Primary safety metric. Under-triage can delay critical care and lead to adverse outcomes.

  **Target**: \< 5% for high-acuity conditions, \< 2% for life-threatening conditions
</ResponseField>

<ResponseField name="Over-triage Rate" type="metric">
  Percentage of cases where the AI assigned a higher urgency than necessary.

  ```
  Over-triage Rate = Over-triaged Cases / Total Low-Acuity Cases
  ```

  **Healthcare Context**: Affects resource utilization and patient experience. Some over-triage is acceptable if it reduces under-triage.

  **Target**: \< 30% is generally acceptable; trade-off with under-triage
</ResponseField>

<ResponseField name="Red Flag Detection Rate" type="metric">
  Percentage of critical symptoms or warning signs correctly identified by the AI.

  ```
  Red Flag Detection = Detected Red Flags / Total Red Flags Present
  ```

  **Healthcare Context**: Essential for patient safety. Missing red flags like chest pain radiation or stroke symptoms can be fatal.

  **Target**: > 98% for critical red flags
</ResponseField>

<ResponseField name="Guideline Adherence Score" type="metric">
  Percentage of AI decisions that align with established clinical guidelines and protocols.

  **Healthcare Context**: Measures whether the AI follows evidence-based medicine principles and organizational protocols.

  **Target**: > 90% adherence to applicable guidelines
</ResponseField>

<ResponseField name="Clinical Concordance" type="metric">
  Agreement rate between AI decisions and expert clinician judgments.

  ```
  Concordance = Matching Decisions / Total Cases Reviewed
  ```

  **Healthcare Context**: Gold standard comparison using board-certified clinician review.

  **Target**: > 85% concordance with expert panel
</ResponseField>

<ResponseField name="Time to Correct Triage" type="metric">
  Average time for incorrect triage decisions to be identified and corrected.

  **Healthcare Context**: Measures the effectiveness of human-in-the-loop review processes.

  **Target**: \< 30 minutes for urgent cases, \< 4 hours for routine cases
</ResponseField>

***

## NLP & Text Metrics

<ResponseField name="Entity Extraction Accuracy" type="metric">
  Correctness of identified medical entities (symptoms, medications, conditions) from clinical text.

  **Healthcare Context**: Foundation for downstream clinical reasoning. Includes:

  * **Exact Match**: Entity boundaries perfectly aligned
  * **Partial Match**: Overlapping but not exact
  * **Type Accuracy**: Correct entity category

  **Target**: > 90% F1 for critical entities
</ResponseField>

<ResponseField name="Negation Detection Accuracy" type="metric">
  Ability to correctly identify negated medical concepts (e.g., "no chest pain" vs "chest pain").

  **Healthcare Context**: Critical for accurate clinical understanding. False positives from missed negations can lead to incorrect triage.

  **Target**: > 95% accuracy on negation scope
</ResponseField>

<ResponseField name="Temporal Expression Accuracy" type="metric">
  Correctness of extracted time references and durations from clinical narratives.

  **Healthcare Context**: "Chest pain for 2 days" vs "chest pain 2 years ago" have vastly different clinical implications.

  **Target**: > 90% accuracy on temporal extraction
</ResponseField>

<ResponseField name="Hallucination Rate" type="metric">
  Percentage of AI-generated content that contains fabricated or unsupported clinical information.

  ```
  Hallucination Rate = Hallucinated Claims / Total Claims Made
  ```

  **Healthcare Context**: AI must not invent symptoms, medications, or findings not present in the source data.

  **Target**: \< 1% for clinical documentation
</ResponseField>

<ResponseField name="Completeness Score" type="metric">
  Percentage of clinically relevant information captured from source content.

  **Healthcare Context**: For clinical note generation, measures whether all important findings, symptoms, and plans are documented.

  **Target**: > 95% for critical clinical elements
</ResponseField>

***

## Imaging Metrics

<ResponseField name="Detection Sensitivity" type="metric">
  Percentage of true abnormalities detected by the imaging AI.

  **Healthcare Context**: For radiology AI, measures ability to find nodules, fractures, or other findings.

  **Target**: > 95% for critical findings (e.g., pneumothorax)
</ResponseField>

<ResponseField name="Localization Accuracy (IoU)" type="metric">
  Intersection over Union measures overlap between predicted and actual abnormality locations.

  ```
  IoU = Area of Overlap / Area of Union
  ```

  **Healthcare Context**: Important for surgical planning and treatment targeting.

  **Typical Range**: 0.0 - 1.0 (> 0.5 generally considered acceptable)
</ResponseField>

<ResponseField name="False Positive Rate (per scan)" type="metric">
  Average number of incorrect positive findings per imaging study.

  **Healthcare Context**: High FP rates lead to unnecessary follow-up procedures and patient anxiety.

  **Target**: \< 1 FP per scan for screening applications
</ResponseField>

<ResponseField name="FROC (Free-Response ROC)" type="metric">
  Extension of ROC analysis for lesion detection, plotting sensitivity vs false positives per image.

  **Healthcare Context**: Standard metric for medical imaging AI competitions and FDA submissions.
</ResponseField>

***

## Operational Metrics

<ResponseField name="Latency" type="metric">
  Time from input submission to AI response generation.

  **Healthcare Context**: Critical for real-time triage. Long latencies impact clinical workflow.

  **Target**: \< 2 seconds for synchronous triage, \< 30 seconds for complex analysis
</ResponseField>

<ResponseField name="Throughput" type="metric">
  Number of evaluations processed per unit time.

  **Healthcare Context**: Must handle peak volumes during high-census periods.

  **Typical**: Evaluations per minute/hour
</ResponseField>

<ResponseField name="Availability" type="metric">
  Percentage of time the system is operational and accessible.

  **Healthcare Context**: Healthcare AI must maintain high availability for patient safety.

  **Target**: > 99.9% uptime (\< 8.76 hours downtime/year)
</ResponseField>

<ResponseField name="Human Override Rate" type="metric">
  Percentage of AI decisions modified by human reviewers.

  ```
  Override Rate = Overridden Decisions / Total Decisions Reviewed
  ```

  **Healthcare Context**: Indicates AI-human agreement and areas needing model improvement.

  **Typical Range**: 5-15% (very low may indicate rubber-stamping)
</ResponseField>

***

## Statistical Concepts

<ResponseField name="Confidence Interval (CI)" type="concept">
  Range of values within which the true metric value likely falls, given sampling variability.

  **Healthcare Context**: Report metrics with 95% CIs, especially for safety-critical measures.

  **Example**: "Sensitivity: 0.94 (95% CI: 0.91-0.97)"
</ResponseField>

<ResponseField name="Statistical Significance" type="concept">
  The probability that observed differences are not due to random chance.

  **Healthcare Context**: When comparing model versions, ensure improvements are statistically significant (p \< 0.05).
</ResponseField>

<ResponseField name="Effect Size" type="concept">
  Magnitude of difference between groups, independent of sample size.

  **Healthcare Context**: A statistically significant but tiny improvement may not be clinically meaningful.

  **Common Measures**: Cohen's d, Odds Ratio, Risk Ratio
</ResponseField>

<ResponseField name="Inter-rater Reliability (IRR)" type="concept">
  Agreement between multiple human reviewers on the same cases.

  **Healthcare Context**: Important for establishing ground truth quality. Use Cohen's Kappa or Fleiss' Kappa.

  **Interpretation**: > 0.8 (excellent), 0.6-0.8 (good), 0.4-0.6 (moderate)
</ResponseField>