Which Decision Models Sort Tax Transactions Best?
I gave seven models the following job: read bank transactions (just the payee and a rounded amount), then sort each one into the correct US personal tax category. I already use a system that sorts my expenses into tax categories, and I wanted to know which model should run it. Jev and d1 tied for the top at about 95.5% accuracy. d1 was the cheapest model in the test, and Jev was the fastest.
The task
Each model saw two things: a payee name, and the amount rounded to the nearest $5. It then picked one of eleven answers: ten tax categories, or “none of these.” The categories were wage income, interest income, medical, charitable, state and local tax, federal tax, education, student loan interest, transfer, and personal.
Take a $50 payment to a food bank (an illustrative row, not one from the set). The right answer is charitable. A model that answers personal costs you a deduction. A model that answers “none of these” costs nothing yet, but leaves the row for a person to sort. Five of the categories are deductible: medical, charitable, state and local tax, education, and student loan interest. A miss in any of those can cost money.
The test set had 1,000 payees. 655 are real names from my own transaction history, written the way a bank exports them. The other 345 are well-known US organizations I added to fill categories that had too few rows. The correct category for each row was mapped from the categories already in the source data; no person has reviewed those labels yet.
All seven models ran through OpenRouter’s Decisions API on October 5, 2026. Each model ran the full set three times, and every number below is the mean of those three passes.
Scoring
Accuracy is correct answers divided by 1,000. A “none of these” counts as a miss. I also report accuracy without abstentions, which is correct answers divided by correct plus wrong: how often a model is right when it commits to an answer. Wrong and abstained are rows per pass of 1,000.
| Model | Accuracy | Without abstentions | Wrong | Abstained | Cost per 100 rows | Time per 1,000 rows |
|---|---|---|---|---|---|---|
| Jev | 95.6% | 98.7% | 13 | 31 | $0.0039 | 20 s |
| d1 | 95.3% | 98.8% | 11 | 36 | $0.0022 | 35 s |
| Clef | 92.3% | 95.0% | 49 | 28 | $0.0183 | 69 s |
| Clef Flash | 91.8% | 95.2% | 46 | 36 | $0.0069 | 44 s |
| pplx-decider | 91.6% | 99.0% | 9 | 75 | $0.0024 | 336 s |
| kev-4b | 88.1% | 94.8% | 48 | 71 | $0.0024 | 192 s |
| Solar Decide | 81.4% | 97.3% | 23 | 163 | $0.0044 | 469 s |
Cost is input and output tokens times OpenRouter’s per-token rate when each run started. Time is the summed latency of the calls for 1,000 rows, sent as two batches of 500.
Where each model falls short
A model can miss a row in two ways: pick the wrong category, or decline to pick. The first is worse, because nothing flags it. This chart splits each model’s misses per 1,000 rows.
Jev and d1 miss the fewest rows, and most of their misses are abstentions. Their three passes overlapped (Jev scored 95.4% to 95.8%, d1 95.0% to 95.5%), so I count them as tied.
pplx-decider made the fewest wrong picks, 9 per pass, and was right 99% of the time it answered. It abstained on 75 rows to get there. Perplexity rate-limited the 500-row batches, so about 6% of its rows failed and were retried in smaller batches. Those retries are included in its time, which explains part of its 336 seconds, and in its cost.
Clef and Clef Flash make three to four times as many wrong picks as Jev. Solar Decide is right 97% of the time it answers, but it chose “none of these” on 163 rows per pass, mostly everyday spending, and took almost 8 minutes per pass.
Accuracy against cost
At these prices the whole test is cheap: 1,000 rows on d1 cost about two cents. The spread still matters at volume. Clef costs about eight times as much as d1 and scores three points lower. Clef Flash costs about a third as much as Clef and scores nearly the same.
By category
Recall is the share of a category’s rows a model got right. The table shows the six categories where the models differed most. Every model scored 100% on federal tax and above 90% on charitable and education. Medical ranged from 82.5% (Solar Decide) to 96.3% (Clef). The number under each heading is how many rows the category has.
| Model | Wages (38) | Interest (41) | State and local tax (45) | Student loan (30) | Transfer (46) | Personal (594) |
|---|---|---|---|---|---|---|
| Jev | 97.4% | 100% | 100% | 91.1% | 71.0% | 96.4% |
| d1 | 94.7% | 100% | 100% | 93.3% | 70.3% | 96.1% |
| Clef | 97.4% | 100% | 100% | 83.3% | 87.0% | 89.9% |
| Clef Flash | 97.4% | 100% | 100% | 86.7% | 67.4% | 90.7% |
| pplx-decider | 97.4% | 100% | 100% | 76.7% | 82.6% | 90.4% |
| kev-4b | 53.5% | 48.8% | 73.3% | 83.3% | 31.2% | 96.1% |
| Solar Decide | 97.4% | 100% | 99.3% | 94.4% | 59.4% | 75.5% |
Transfers were the hardest category. Only Clef (87%) and pplx-decider (83%) got more than 75% of them right; the rest scored between 31% and 71%. kev-4b missed about half of the wage and interest rows. Small categories move fast: with 25 to 46 rows, a row missed in all three passes changes recall by 2 to 4 points, and a row missed in one pass changes it by about 1.
Real names against added names
The real names are raw strings from bank exports, which are messier than an organization’s official name: think SQ *BLUE BOTTLE 0423 rather than “Blue Bottle Coffee” (an illustrative example, not a row from the set). Splitting the results by source shows which models handle them:
| Model | Real names (655) | Added names (345) |
|---|---|---|
| Jev | 94.2% | 98.3% |
| d1 | 93.9% | 97.9% |
| Clef | 89.8% | 97.1% |
| Clef Flash | 89.6% | 95.9% |
| pplx-decider | 89.2% | 96.2% |
| kev-4b | 93.4% | 77.9% |
| Solar Decide | 73.9% | 95.7% |
Most models did better on the clean added names. kev-4b went the other way: it read raw bank strings nearly as well as the leaders but dropped to 77.9% on the added organizations. Solar Decide did the reverse and fell to 73.9% on real names.
What this does and doesn’t show
The labels came from mapping existing categories and haven’t been reviewed by a person, so some “wrong” answers may be label errors. A third of the set is names I added, which a real transaction history wouldn’t contain. And each model saw only the payee and a rounded amount; with a date or a memo line, the rankings could change.
For this task as tested, I’d use Jev or d1: d1 if cost matters most, Jev if speed does. Either one still leaves a few rows a person should check, and I’d rather a model tell me it isn’t sure than guess wrong.