Harness Run · so_extraction 20260515T033428Z  ·  36 runs  ·  Generated 2026-05-15 03:35 UTC

Structured data extraction shows solid speed but field accuracy needs attention

All 36 runs completed successfully in about six seconds each, yet only 39% of fields matched expected values.

OVERALL PERFORMANCE
100% success rate
All 36 extraction runs completed without errors across four Claude models (Opus 4-5, Opus 4-6, Sonnet 4-5, Sonnet 4-6). Average runtime was 6.2 seconds per run, showing consistent speed regardless of model choice.
ACCURACY GAP
39% field match rate
Only 39% of extracted fields matched expected values, with an average of 13.4 mismatches per run. The variation is significant: standard deviation of 4.1 mismatches suggests some runs performed notably better or worse than others.
MODEL COMPARISON
Opus 4-6 leads accuracy
Opus 4-6 achieved the best field match rate at 43% (12.2 mismatches per run), slightly outperforming Opus 4-5 at 42% (12.4 mismatches). Sonnet models lagged: Sonnet 4-6 matched 38% of fields while Sonnet 4-5 matched just 35%.
Sec. 01

Results by dataset

This benchmark tested structured data extraction against a single dataset ('downloaded') containing purchase order information.

Accuracy below expectations: With only 39% of fields matching ground truth, the extraction agent may need prompt refinement, additional examples, or schema validation to reduce the 13+ mismatches per document.
downloaded
39.37%
Field match
36 runs13.39 mismatch
Sec. 02

Quality vs. speed

All four models delivered similar speed (6.0 to 6.3 seconds) but differed in accuracy. Opus models extracted fields more reliably than Sonnet variants.

Model frontier
Each bubble is one row in the table below (model × few-shot). Bubble size reflects mismatch load.
Sec. 03

Leaderboard

Compare model and few-shot configurations below. Each row represents 9 runs with a specific model tested at three different few-shot levels.

0 rows
Model FS Runs Avg s Mismatch Field match
Sec. 04

Few-shot sweep

Each model was tested with three few-shot example counts to assess learning from context.

FS 3
39.37%

Few-shot variation did not produce dramatic accuracy differences within models; the gap between Opus and Sonnet architectures was more significant than example count.

Sec. 05

What to check next

Based on these results, prioritize the following steps to improve extraction quality while maintaining speed.

01
Investigate mismatch patterns
Review the 13+ average mismatches per run to identify which fields fail most often (dates, amounts, nested structures). This will pinpoint whether errors are systematic or random.
02
Standardize on Opus 4-6
Since Opus 4-6 delivered the best field match rate (43%) at comparable speed, use it as the default model for production extraction until accuracy improves.
03
Refine extraction prompts
The 61% mismatch rate suggests prompt instructions may be ambiguous or incomplete. Test clearer field definitions, format examples, and error-handling guidance.
04
Add output validation
Implement schema checks or post-processing rules to catch common extraction errors before results reach downstream systems.
05
Expand few-shot testing
Current few-shot counts (3 levels tested) showed modest impact. Experiment with higher example counts or domain-specific samples to see if accuracy improves.
06
Benchmark against baseline
Compare these results to a previous extraction method or human performance to confirm whether 39% accuracy is acceptable for the use case or requires urgent improvement.
Run configuration (JSON) 
{
  "agent": "so_extraction",
  "pipeline": null,
  "models": [
    "sonnet-4-6",
    "opus-4-5",
    "opus-4-6",
    "sonnet-4-5"
  ],
  "datasets": [
    "downloaded"
  ],
  "chat": null,
  "chats_glob": null,
  "bulk": false,
  "runs_per_chat": 1,
  "max_workers": 25,
  "few_shot_explicit": [
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/multiple_product_multiple_shipment_complex.json",
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/single_product_multiple_shipment_medium.json",
    "/Users/tripathipranav/Documents/code/harness_agents/raw_data/chats/single_product_single_shipment_simple.json"
  ],
  "few_shot_walk": [],
  "few_shot_sweep": [],
  "few_shot_pool_argv": [],
  "few_shot_seed": 42,
  "db_few_shot_limit": 0,
  "skip_without_expected": false,
  "results_dir": "/Users/tripathipranav/Documents/code/harness_agents/results/20260515T033428Z",
  "config_file": "configs/agents.json",
  "few_shot_mode": "explicit",
  "few_shot_pool_size": 68,
  "few_shot_default_pool_size": 68,
  "few_shot_pool_override": null,
  "few_shot_variants": [
    {
      "label": "explicit",
      "count": 3,
      "paths": [
        "raw_data/chats/multiple_product_multiple_shipment_complex.json",
        "raw_data/chats/single_product_multiple_shipment_medium.json",
        "raw_data/chats/single_product_single_shipment_simple.json"
      ]
    }
  ],
  "allow_self_fewshot": false
}
How to read these numbers

SUCCESS RATE — Share of runs that finished without a harness or HTTP error. High success means the run was stable; it does not prove the answers matched the reference.

AVG ELAPSED (S) — Average wall time per run in that bucket. Useful for latency comparisons.

AVG MISMATCH / EXPECTED RUN — Average count of fields that differed from the golden JSON when a reference existed. Lower is better.

FIELD MATCH — Fraction of compared fields that matched the golden output across runs in that bucket. Higher is better.

Sample mismatches (up to 80 rows)
AgentChatModelFSMismatchesSample
so_extraction08__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonsonnet-4-5319
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 46000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction03__2026-01-30__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonopus-4-5317
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 3
  },
  {
    "path": "data[0].items[0].description",
    "expected": "BergaPur",
    "actual": "Bergapur"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 6300.0,
    "actual": 10.5
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3050.0,
    "actual": 3.05
  }
]
so_extraction03__2026-01-30__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonsonnet-4-5317
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 3
  },
  {
    "path": "data[0].items[0].description",
    "expected": "BergaPur",
    "actual": "Bergapur"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 6300.0,
    "actual": 10.5
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3050.0,
    "actual": 3.05
  }
]
so_extraction07__2025-12-23__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonsonnet-4-6317
[
  {
    "path": "data[0].items[0].description",
    "expected": "BergaPur",
    "actual": ""
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 10500.0,
    "actual": 10.5
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3100.0,
    "actual": null
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": ""
  }
]
so_extraction09__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonsonnet-4-5317
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 23000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction03__2026-01-30__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonopus-4-6316
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 2
  },
  {
    "path": "data[0].items[0].description",
    "expected": "BergaPur",
    "actual": "Bergapur"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 6300.0,
    "actual": 6.3
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3050.0,
    "actual": 3.05
  }
]
so_extraction03__2026-01-30__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonsonnet-4-6316
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 2
  },
  {
    "path": "data[0].items[0].description",
    "expected": "BergaPur",
    "actual": "Bergapur"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 6300.0,
    "actual": 6.3
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3050.0,
    "actual": 3.05
  }
]
so_extraction04__2026-01-29__120363408498669191_g_us__4b9c2faa-94dd-4236-abcc-398807051f21.jsonsonnet-4-6316
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 9000.0,
    "actual": 9.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Jakarta",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Jakarta"
  }
]
so_extraction01__2026-02-24__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonopus-4-6315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3250.0,
    "actual": 3.25
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction01__2026-02-24__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonsonnet-4-5315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3250.0,
    "actual": 3.25
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction01__2026-02-24__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonopus-4-5315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3250.0,
    "actual": 3.25
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction02__2026-02-09__120363426578757754_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonsonnet-4-6315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "Soya Lecithin Powder"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 39000.0,
    "actual": 39.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Nhava Sheva",
    "actual": "CIF"
  }
]
so_extraction04__2026-01-29__120363408498669191_g_us__4b9c2faa-94dd-4236-abcc-398807051f21.jsonsonnet-4-5315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 9000.0,
    "actual": 9.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Jakarta",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Jakarta"
  }
]
so_extraction04__2026-01-29__120363408498669191_g_us__4b9c2faa-94dd-4236-abcc-398807051f21.jsonopus-4-6315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 9000.0,
    "actual": 9.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Jakarta",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Jakarta"
  }
]
so_extraction04__2026-01-29__120363408498669191_g_us__4b9c2faa-94dd-4236-abcc-398807051f21.jsonopus-4-5315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 9000.0,
    "actual": 9.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Jakarta",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Jakarta"
  }
]
so_extraction06__2026-01-06__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonsonnet-4-6315
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3500.0,
    "actual": 3.5
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction08__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonsonnet-4-6315
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 46000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction01__2026-02-24__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonsonnet-4-6314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "EXW"
  }
]
so_extraction02__2026-02-09__120363426578757754_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonsonnet-4-5314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "Soya Lecithin Powder"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 39000.0,
    "actual": 39.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Nhava Sheva",
    "actual": "CIF"
  }
]
so_extraction02__2026-02-09__120363426578757754_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonopus-4-5314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "Soya Lecithin Powder"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 39000.0,
    "actual": 39.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Nhava Sheva",
    "actual": "CIF"
  }
]
so_extraction02__2026-02-09__120363426578757754_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonopus-4-6314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "Soya Lecithin Powder"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 39000.0,
    "actual": 39.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF Nhava Sheva",
    "actual": "CIF"
  }
]
so_extraction05__2026-01-20__120363407382355715_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonsonnet-4-6314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOIFNE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 26000.0,
    "actual": 26.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF NHAVA SHEVA",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Nhava Sheva"
  }
]
so_extraction05__2026-01-20__120363407382355715_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonopus-4-5314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOIFNE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 26000.0,
    "actual": 26.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF NHAVA SHEVA",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Nhava Sheva"
  }
]
so_extraction06__2026-01-06__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonsonnet-4-5314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3500.0,
    "actual": 3.5
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction06__2026-01-06__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonopus-4-6314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3500.0,
    "actual": 3.5
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction06__2026-01-06__120363421131250401_g_us__e05574ec-b110-4554-9fc3-3abb4f9011a8.jsonopus-4-5314
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOFINE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 1800.0,
    "actual": 1.8
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].unit_price",
    "expected": 3500.0,
    "actual": 3.5
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/KG"
  }
]
so_extraction05__2026-01-20__120363407382355715_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonsonnet-4-5313
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOIFNE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 26000.0,
    "actual": 26.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF NHAVA SHEVA",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Nhava Sheva"
  }
]
so_extraction05__2026-01-20__120363407382355715_g_us__12a4f3a7-d506-4d32-ae06-3f76508c6abd.jsonopus-4-6313
[
  {
    "path": "data[0].items[0].description",
    "expected": "GIIOFINE - P - S",
    "actual": "GIIOIFNE-P-S"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 26000.0,
    "actual": 26.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "KG",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].ship_term",
    "expected": "CIF NHAVA SHEVA",
    "actual": "CIF"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Nhava Sheva"
  }
]
so_extraction09__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonsonnet-4-6313
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 23000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction08__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonopus-4-6312
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 46000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction08__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonopus-4-5312
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 46000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction09__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonopus-4-5310
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 23000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction09__2025-09-29__120363403592950429_g_us__d586d853-694c-42f9-93be-bc7ba5b2110c.jsonopus-4-6310
[
  {
    "path": "data[0].items[0].description",
    "expected": "TG - BP102",
    "actual": "BP102"
  },
  {
    "path": "data[0].items[0].quantity",
    "expected": 23000.0,
    "actual": 23.0
  },
  {
    "path": "data[0].items[0].quantity_unit",
    "expected": "kg",
    "actual": "MT"
  },
  {
    "path": "data[0].items[0].pricing_unit",
    "expected": "usd/mt",
    "actual": "USD/MT"
  },
  {
    "path": "data[0].items[0].delivery_terms",
    "expected": "",
    "actual": "CIF Busan"
  }
]
so_extraction07__2025-12-23__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonopus-4-531
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 0
  }
]
so_extraction07__2025-12-23__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonopus-4-631
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 0
  }
]
so_extraction07__2025-12-23__120363403074656566_g_us__8f477a8f-2a60-4e0a-bf0e-8cc3cdf1dc9f.jsonsonnet-4-531
[
  {
    "path": "data",
    "expected_len": 1,
    "actual_len": 0
  }
]