Overall, this was a successful run with no agent execution failures across 252 attempts. However, the quality of the JSON output was highly sensitive to the number of few-shot examples provided. While the zero-shot configuration performed exceptionally well, the inclusion of 1 or 4 examples dramatically increased the error rate, indicating a brittle relationship between the prompt and the model's output quality for this task.
Success Rate is the percentage of runs where the agent and model completed without an HTTP error. Avg Runtime is the average wall-clock time for a run to complete. Avg mismatch/expected run is the average number of fields that did not match the golden ("expected") JSON output for a given run. Field match rate is the inverse: the percentage of fields that did match the golden JSON across all runs in that combination. Mismatch stdev measures the consistency of mismatches across runs; a high value means the number of errors per run is very inconsistent.
The sonnet-4-6 model with the so_extraction agent showed a dramatic variation in performance based on the number of few-shot examples (fs_count). The zero-shot (fs_count: 0) configuration was the strongest, achieving a 99.1% field match rate. Performance was significantly degraded with 1 and 4 few-shot examples, which were the weakest configurations, dropping the match rate to ~90% and ~89.3% respectively. Configurations with 2 and 5 examples also performed very well, nearly matching the top zero-shot performance.
{
"agent": "so_extraction",
"pipeline": null,
"models": [
"sonnet-4-6"
],
"datasets": [
"acme_foods",
"nova_exports"
],
"chat": null,
"chats_glob": null,
"bulk": false,
"runs_per_chat": 1,
"max_workers": 20,
"few_shot_explicit": [],
"few_shot_sweep": [
0,
1,
2,
3,
4,
5
],
"few_shot_seed": 42,
"db_few_shot_limit": 0,
"skip_without_expected": true,
"results_dir": "/Users/tripathipranav/Documents/code/harness_agents/results/20260511T193240Z",
"config_file": "configs/agents.json"
}
| Agent | Runs | Success | Avg attempts | Avg elapsed (s) | Avg mismatch/expected | Field match |
|---|---|---|---|---|---|---|
| so_extraction | 252 | 1.0000 | 1.0000 | 4.6449 | 1.0913 | 0.9556 |
| Agent | Model | FS count | Runs | Success | Avg attempts | Avg elapsed (s) | Avg mismatch/expected | Field match |
|---|---|---|---|---|---|---|---|---|
| so_extraction | sonnet-4-6 | 0 | 42 | 1.0000 | 1.0000 | 4.6833 | 0.2143 | 0.9913 |
| so_extraction | sonnet-4-6 | 1 | 42 | 1.0000 | 1.0000 | 4.8282 | 2.4762 | 0.8997 |
| so_extraction | sonnet-4-6 | 2 | 42 | 1.0000 | 1.0000 | 4.7019 | 0.2619 | 0.9894 |
| so_extraction | sonnet-4-6 | 3 | 42 | 1.0000 | 1.0000 | 4.5849 | 0.7381 | 0.9701 |
| so_extraction | sonnet-4-6 | 4 | 42 | 1.0000 | 1.0000 | 4.6151 | 2.5714 | 0.8932 |
| so_extraction | sonnet-4-6 | 5 | 42 | 1.0000 | 1.0000 | 4.4563 | 0.2857 | 0.9884 |
| Agent | FS count | Runs | Success | Avg mismatch/expected | Field match |
|---|---|---|---|---|---|
| so_extraction | 0 | 42 | 1.0000 | 0.2143 | 0.9913 |
| so_extraction | 1 | 42 | 1.0000 | 2.4762 | 0.8997 |
| so_extraction | 2 | 42 | 1.0000 | 0.2619 | 0.9894 |
| so_extraction | 3 | 42 | 1.0000 | 0.7381 | 0.9701 |
| so_extraction | 4 | 42 | 1.0000 | 2.5714 | 0.8932 |
| so_extraction | 5 | 42 | 1.0000 | 0.2857 | 0.9884 |
| Agent | Dataset | Runs | Success | Avg elapsed (s) | Avg mismatch/expected | Field match |
|---|---|---|---|---|---|---|
| so_extraction | acme_foods | 126 | 1.0000 | 4.8456 | 0.9841 | 0.9608 |
| so_extraction | nova_exports | 126 | 1.0000 | 4.4443 | 1.1984 | 0.9501 |
| Agent | Chat | Model | FS count | Runs | Success | Avg elapsed (s) | Avg mismatch/expected |
|---|---|---|---|---|---|---|---|
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 0 | 1 | 1.0000 | 6.5045 | 0.0000 |
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 1 | 1 | 1.0000 | 6.1190 | 0.0000 |
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.4393 | 0.0000 |
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 3 | 1 | 1.0000 | 6.3798 | 0.0000 |
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 4 | 1 | 1.0000 | 6.1929 | 0.0000 |
| so_extraction | fs_acme_simple.json | sonnet-4-6 | 5 | 1 | 1.0000 | 6.3001 | 0.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0979 | 1.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 1 | 1 | 1.0000 | 2.5555 | 1.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 2 | 1 | 1.0000 | 2.7664 | 1.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 3 | 1 | 1.0000 | 6.9110 | 1.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 4 | 1 | 1.0000 | 2.2091 | 1.0000 |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 5 | 1 | 1.0000 | 2.9203 | 1.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 0 | 1 | 1.0000 | 6.5087 | 0.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 1 | 1 | 1.0000 | 6.1940 | 3.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.1811 | 0.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 3 | 1 | 1.0000 | 6.4465 | 0.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 4 | 1 | 1.0000 | 7.9795 | 3.0000 |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 5 | 1 | 1.0000 | 6.4044 | 0.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 0 | 1 | 1.0000 | 6.8595 | 0.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 1 | 1 | 1.0000 | 7.0397 | 4.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.9937 | 0.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 3 | 1 | 1.0000 | 6.9958 | 0.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 4 | 1 | 1.0000 | 7.5384 | 5.0000 |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 5 | 1 | 1.0000 | 7.0501 | 0.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 0 | 1 | 1.0000 | 6.1636 | 0.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 1 | 1 | 1.0000 | 6.4572 | 2.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.9609 | 0.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9674 | 2.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.7122 | 2.0000 |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1896 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9247 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3743 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.7481 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.8682 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2008 | 0.0000 |
| so_extraction | generated_acme_foods_004.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.8497 | 0.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 0 | 1 | 1.0000 | 8.1184 | 0.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3701 | 2.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0707 | 0.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.0695 | 0.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.3110 | 2.0000 |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.4317 | 0.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.9082 | 0.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.5114 | 2.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.0075 | 0.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.2472 | 0.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 4 | 1 | 1.0000 | 3.9925 | 2.0000 |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1272 | 0.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0762 | 0.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.1700 | 3.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.7585 | 0.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.8652 | 0.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.3647 | 3.0000 |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.8555 | 0.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.1127 | 0.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.2281 | 2.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 2 | 1 | 1.0000 | 5.5940 | 0.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 3 | 1 | 1.0000 | 7.2992 | 0.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 4 | 1 | 1.0000 | 3.8850 | 2.0000 |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.9242 | 0.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0902 | 0.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.2811 | 2.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.3149 | 0.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9988 | 0.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 4 | 1 | 1.0000 | 3.9139 | 2.0000 |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1056 | 0.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.8754 | 0.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.1035 | 4.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.8213 | 0.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.6624 | 0.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.1989 | 4.0000 |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.7066 | 0.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.2116 | 0.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.4098 | 3.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0322 | 0.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.2934 | 0.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.6590 | 3.0000 |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.8561 | 0.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 0 | 1 | 1.0000 | 6.9514 | 0.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.1693 | 4.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 2 | 1 | 1.0000 | 5.8399 | 0.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.9262 | 0.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.7331 | 4.0000 |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.8628 | 0.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.1845 | 0.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.0661 | 2.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 2 | 1 | 1.0000 | 5.1067 | 0.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.6377 | 0.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 4 | 1 | 1.0000 | 3.9649 | 2.0000 |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0556 | 0.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.1746 | 0.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.0279 | 0.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0811 | 2.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.2229 | 0.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.1738 | 0.0000 |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 5 | 1 | 1.0000 | 5.1241 | 2.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0661 | 0.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.0986 | 0.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0115 | 0.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9400 | 0.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2271 | 2.0000 |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1164 | 0.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.5832 | 0.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.1671 | 2.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0920 | 0.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.3648 | 0.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.8760 | 2.0000 |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0048 | 0.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9088 | 0.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3426 | 2.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.8764 | 0.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.0122 | 1.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.0126 | 3.0000 |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0862 | 0.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0058 | 0.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 1 | 1 | 1.0000 | 3.8765 | 2.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0574 | 0.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9370 | 0.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.0516 | 2.0000 |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1496 | 0.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0722 | 0.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.4163 | 2.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.6096 | 0.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.5068 | 0.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.0526 | 2.0000 |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0004 | 0.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 0 | 1 | 1.0000 | 5.4759 | 0.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.0298 | 4.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.8740 | 0.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 3 | 1 | 1.0000 | 5.0512 | 0.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.0276 | 4.0000 |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 5 | 1 | 1.0000 | 5.9107 | 0.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9809 | 0.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.2274 | 4.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.2479 | 0.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.8310 | 2.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2525 | 4.0000 |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1506 | 0.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0736 | 0.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.5586 | 3.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.8437 | 0.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.1785 | 0.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.5716 | 3.0000 |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.5195 | 1.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 0 | 1 | 1.0000 | 5.3446 | 0.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3311 | 2.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.9119 | 0.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.3232 | 0.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.3782 | 2.0000 |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.2286 | 0.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9695 | 0.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 1 | 1 | 1.0000 | 6.2744 | 4.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.3446 | 0.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.4996 | 3.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2629 | 4.0000 |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.8977 | 0.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.8746 | 1.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.2370 | 4.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0173 | 0.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.2997 | 1.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.4567 | 4.0000 |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.2857 | 1.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.6913 | 0.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.4810 | 2.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.8202 | 0.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.1029 | 0.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2282 | 2.0000 |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1783 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.3760 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.2042 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.1827 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.3296 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.0895 | 0.0000 |
| so_extraction | realistic_acme_foods_007.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.2146 | 0.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.2246 | 2.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.4412 | 2.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.6196 | 2.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.5315 | 3.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.4468 | 2.0000 |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.6581 | 2.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.1537 | 0.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.2123 | 4.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.9470 | 0.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.3427 | 1.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.4760 | 4.0000 |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0972 | 0.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.3318 | 0.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3990 | 2.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 2 | 1 | 1.0000 | 6.1796 | 0.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.0778 | 0.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2520 | 2.0000 |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.9687 | 0.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9123 | 2.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.2387 | 2.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.3570 | 1.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.0797 | 3.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.7954 | 2.0000 |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1926 | 2.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.3790 | 0.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 1 | 1 | 1.0000 | 9.7674 | 4.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 2 | 1 | 1.0000 | 5.0918 | 1.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.1041 | 1.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.3283 | 5.0000 |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.0263 | 0.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0338 | 0.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.0031 | 6.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.1488 | 1.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.1308 | 3.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.6740 | 6.0000 |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.2689 | 0.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.1249 | 3.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.1720 | 3.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0719 | 3.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 3 | 1 | 1.0000 | 5.7056 | 3.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.1284 | 3.0000 |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.9162 | 3.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.2988 | 0.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 1 | 1 | 1.0000 | 3.9051 | 0.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.1017 | 0.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.7955 | 0.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 4 | 1 | 1.0000 | 3.9726 | 0.0000 |
| so_extraction | realistic_nova_exports_005.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.2473 | 0.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.3020 | 0.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 1 | 1 | 1.0000 | 3.9215 | 3.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.9426 | 0.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.1511 | 1.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.5360 | 3.0000 |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.3799 | 0.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.4647 | 0.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.4172 | 1.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.2147 | 0.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9533 | 0.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.2832 | 1.0000 |
| so_extraction | realistic_nova_exports_007.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.9772 | 0.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 0 | 1 | 1.0000 | 5.2927 | 0.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 1 | 1 | 1.0000 | 4.3008 | 4.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 2 | 1 | 1.0000 | 4.0401 | 0.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.5359 | 3.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.3126 | 4.0000 |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 5 | 1 | 1.0000 | 3.9599 | 0.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 0 | 1 | 1.0000 | 3.9063 | 0.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.3503 | 5.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 2 | 1 | 1.0000 | 3.8229 | 0.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 3 | 1 | 1.0000 | 3.9820 | 0.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 4 | 1 | 1.0000 | 5.0970 | 5.0000 |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 5 | 1 | 1.0000 | 4.1600 | 0.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 0 | 1 | 1.0000 | 4.0878 | 0.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 1 | 1 | 1.0000 | 5.3358 | 3.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 2 | 1 | 1.0000 | 5.3356 | 0.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 3 | 1 | 1.0000 | 4.0080 | 3.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 4 | 1 | 1.0000 | 4.0155 | 2.0000 |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 5 | 1 | 1.0000 | 5.8063 | 0.0000 |
| Agent | Chat | Model | FS count | Mismatches | Sample |
|---|---|---|---|---|---|
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 1 | 6 | [
{
"path": "data[0].items",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items[0].description",
"expected": "Black tea cartons",
"actual": "Black tea"
},
{
"path": "data[0].items[0].quantity",
"expected": 18.0,
"actual": 9.0
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 4 | 6 | [
{
"path": "data",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items[0].description",
"expected": "Black tea cartons",
"actual": "Black tea"
},
{
"path": "data[0].items[0].quantity",
"expected": 18.0,
"actual": 9.0
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 4 | 5 | [
{
"path": "data",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items",
"expected_len": 2,
"actual_len": 1
},
{
"path": "data[0].items[0].description",
"expected": "Robusta coffee bags",
"actual": "Robusta coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 4 | 5 | [
{
"path": "data[0].items[0].description",
"expected": "Ceylon tea cartons",
"actual": "Ceylon tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 18.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 360.0
}
] |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 1 | 5 | [
{
"path": "data[0].items",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items[0].quantity",
"expected": 25.0,
"actual": 12.5
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": 350.0,
"actual": 175.0
}
] |
| so_extraction | realistic_nova_exports_009.json | sonnet-4-6 | 4 | 5 | [
{
"path": "data[0].items",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items[0].quantity",
"expected": 25.0,
"actual": 12.5
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": 350.0,
"actual": 175.0
}
] |
| so_extraction | generated_acme_foods_002.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_acme_foods_010.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_nova_exports_002.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data",
"expected_len": 1,
"actual_len": 2
},
{
"path": "data[0].items",
"expected_len": 2,
"actual_len": 1
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_nova_exports_010.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[1].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[1].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 19.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 285.0
}
] |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 19.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 285.0
}
] |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 19.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 285.0
}
] |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 19.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 285.0
}
] |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "SACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 16.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/SACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 1280.0
}
] |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "SACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 16.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/SACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 1280.0
}
] |
| so_extraction | realistic_nova_exports_002.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 18.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/CARTON"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 360.0
}
] |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 1 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 4 | 4 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD/PACK"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_acme_foods_001.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea cartons",
"actual": "Assam tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_acme_foods_007.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea cartons",
"actual": "Assam tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_nova_exports_001.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea cartons",
"actual": "Assam tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Arabica coffee bags",
"actual": "Arabica coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | realistic_acme_foods_004.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Black tea",
"actual": "Black tea cartons"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea",
"actual": "Assam tea cartons"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD"
},
{
"path": "data[0].shipping_address",
"expected": "101 Market Street, Customer Hub",
"actual": ""
}
] |
| so_extraction | realistic_nova_exports_003.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Black tea cartons",
"actual": "Black tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 0 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 2 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 5 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_004.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 55.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 165.0
}
] |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea cartons",
"actual": "Assam tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_006.json | sonnet-4-6 | 4 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea cartons",
"actual": "Assam tea"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_008.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 6.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 60.0
}
] |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 3 | 3 | [
{
"path": "data[0].items[0].unit_price",
"expected": null,
"actual": 21.0
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "",
"actual": "USD"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 252.0
}
] |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 1 | 3 | [
{
"path": "data[0].items[0].description",
"expected": "Robusta coffee bags",
"actual": "Robusta coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].total",
"expected": null,
"actual": 252.0
}
] |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 3 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_acme_foods_003.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "KITS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/KIT"
}
] |
| so_extraction | generated_acme_foods_005.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "KITS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/KIT"
}
] |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_acme_foods_006.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "JARS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/JAR"
}
] |
| so_extraction | generated_acme_foods_008.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "JARS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/JAR"
}
] |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BOXES"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BOX"
}
] |
| so_extraction | generated_acme_foods_009.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BOXES"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BOX"
}
] |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_nova_exports_003.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BAG"
}
] |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 2 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "SACKS",
"actual": ""
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/SACK",
"actual": "USD"
}
] |
| so_extraction | generated_nova_exports_004.json | sonnet-4-6 | 5 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "SACKS",
"actual": ""
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/SACK",
"actual": "USD"
}
] |
| so_extraction | generated_nova_exports_005.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "KITS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/KIT"
}
] |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_nova_exports_006.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/CARTON"
}
] |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "JARS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/JAR"
}
] |
| so_extraction | generated_nova_exports_008.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "JARS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/JAR"
}
] |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BOXES"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BOX"
}
] |
| so_extraction | generated_nova_exports_009.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BOXES"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/BOX"
}
] |
| so_extraction | realistic_acme_foods_001.json | sonnet-4-6 | 3 | 2 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea",
"actual": "Assam tea cartons"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": ""
}
] |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | realistic_acme_foods_003.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "KITS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/KIT"
}
] |
| so_extraction | realistic_acme_foods_006.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "KITS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/KIT"
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 0 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 2 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_008.json | sonnet-4-6 | 5 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | realistic_acme_foods_010.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "PACKS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD",
"actual": "USD/PACK"
}
] |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 0 | 2 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea",
"actual": "Assam tea cartons"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD"
}
] |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 1 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 5 | 2 | [
{
"path": "data[0].items[0].description",
"expected": "Assam tea",
"actual": "Assam tea cartons"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD"
}
] |
| so_extraction | realistic_nova_exports_001.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
},
{
"path": "data[0].items[0].pricing_unit",
"expected": "USD/carton",
"actual": "USD/CARTON"
}
] |
| so_extraction | realistic_nova_exports_010.json | sonnet-4-6 | 4 | 2 | [
{
"path": "data[0].items[0].description",
"expected": "Robusta coffee bags",
"actual": "Robusta coffee"
},
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "BAGS"
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 1 | 1 | [
{
"path": "data",
"expected": [],
"actual": []
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 2 | 1 | [
{
"path": "data",
"expected": [],
"actual": []
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 4 | 1 | [
{
"path": "data",
"expected": [],
"actual": []
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 0 | 1 | [
{
"path": "data",
"expected": [],
"actual": []
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 5 | 1 | [
{
"path": "data",
"expected": [],
"actual": []
}
] |
| so_extraction | fs_nova_simple.json | sonnet-4-6 | 3 | 1 | [
{
"path": "data",
"expected": [],
"actual": [
{
"items": [
{
"sr_no": 1,
"description": "Ceylon tea",
"quantity": 15.0,
"quantity_unit": "CARTONS",
"unit_price": 18.0,
"pricing_unit": "USD/CARTON",
"ship_term": "",
"delivery_terms": "",
"shipment_date": "",
"shipping_address": "",
"packing": "",
"loading": "",
"total": null
}
],
"do_date": "",
"po_date": "",
"po_ref_no": "",
"vendor_name": "Van Beethoven",
"payment_date": "",
"delivery_terms": "",
"billing_address": "",
"shipping_method": "",
"shipping_address": ""
}
]
}
] |
| so_extraction | generated_nova_exports_007.json | sonnet-4-6 | 3 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "cartons"
}
] |
| so_extraction | realistic_acme_foods_002.json | sonnet-4-6 | 5 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "bags"
}
] |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 0 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
}
] |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 3 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
}
] |
| so_extraction | realistic_acme_foods_005.json | sonnet-4-6 | 5 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "cartons",
"actual": "CARTONS"
}
] |
| so_extraction | realistic_acme_foods_009.json | sonnet-4-6 | 3 | 1 | [
{
"path": "data[0].items[0].quantity_unit",
"expected": "",
"actual": "SACKS"
}
] |