Which Decision Models Sort Tax Transactions Best?

I gave seven models the following job: read bank transactions (just the payee and a rounded amount), then sort each one into the correct US personal tax category. I already use a system that sorts my expenses into tax categories, and I wanted to know which model should run it. Jev and d1 tied for the top at about 95.5% accuracy. d1 was the cheapest model in the test, and Jev was the fastest.

The task

Each model saw two things: a payee name, and the amount rounded to the nearest $5. It then picked one of eleven answers: ten tax categories, or “none of these.” The categories were wage income, interest income, medical, charitable, state and local tax, federal tax, education, student loan interest, transfer, and personal.

Take a $50 payment to a food bank (an illustrative row, not one from the set). The right answer is charitable. A model that answers personal costs you a deduction. A model that answers “none of these” costs nothing yet, but leaves the row for a person to sort. Five of the categories are deductible: medical, charitable, state and local tax, education, and student loan interest. A miss in any of those can cost money.

The test set had 1,000 payees. 655 are real names from my own transaction history, written the way a bank exports them. The other 345 are well-known US organizations I added to fill categories that had too few rows. The correct category for each row was mapped from the categories already in the source data; no person has reviewed those labels yet.

All seven models ran through OpenRouter’s Decisions API on October 5, 2026. Each model ran the full set three times, and every number below is the mean of those three passes.

Scoring

Accuracy is correct answers divided by 1,000. A “none of these” counts as a miss. I also report accuracy without abstentions, which is correct answers divided by correct plus wrong: how often a model is right when it commits to an answer. Wrong and abstained are rows per pass of 1,000.

Model Accuracy Without abstentions Wrong Abstained Cost per 100 rows Time per 1,000 rows
Jev 95.6% 98.7% 13 31 $0.0039 20 s
d1 95.3% 98.8% 11 36 $0.0022 35 s
Clef 92.3% 95.0% 49 28 $0.0183 69 s
Clef Flash 91.8% 95.2% 46 36 $0.0069 44 s
pplx-decider 91.6% 99.0% 9 75 $0.0024 336 s
kev-4b 88.1% 94.8% 48 71 $0.0024 192 s
Solar Decide 81.4% 97.3% 23 163 $0.0044 469 s

Cost is input and output tokens times OpenRouter’s per-token rate when each run started. Time is the summed latency of the calls for 1,000 rows, sent as two batches of 500.

Where each model falls short

A model can miss a row in two ways: pick the wrong category, or decline to pick. The first is worse, because nothing flags it. This chart splits each model’s misses per 1,000 rows.

WrongAbstained050100150200JevJev: 13 wrongJev: 31 abstained44d1d1: 11 wrongd1: 36 abstained47ClefClef: 49 wrongClef: 28 abstained77Clef FlashClef Flash: 46 wrongClef Flash: 36 abstained82pplx-deciderpplx-decider: 9 wrongpplx-decider: 75 abstained84kev-4bkev-4b: 48 wrongkev-4b: 71 abstained119Solar DecideSolar Decide: 23 wrongSolar Decide: 163 abstained186
Rows missed per pass of 1,000, ranked by accuracy. The number at the end of each bar is the total.

Jev and d1 miss the fewest rows, and most of their misses are abstentions. Their three passes overlapped (Jev scored 95.4% to 95.8%, d1 95.0% to 95.5%), so I count them as tied.

pplx-decider made the fewest wrong picks, 9 per pass, and was right 99% of the time it answered. It abstained on 75 rows to get there. Perplexity rate-limited the 500-row batches, so about 6% of its rows failed and were retried in smaller batches. Those retries are included in its time, which explains part of its 336 seconds, and in its cost.

Clef and Clef Flash make three to four times as many wrong picks as Jev. Solar Decide is right 97% of the time it answers, but it chose “none of these” on 163 rows per pass, mostly everyday spending, and took almost 8 minutes per pass.

Accuracy against cost

80%85%90%95%100%$0.002$0.003$0.005$0.01$0.02Cost per 100 rows (log scale)Jev: 95.6% at $0.0039 per 100 rowsJevd1: 95.3% at $0.0022 per 100 rowsd1Clef: 92.3% at $0.0183 per 100 rowsClefClef Flash: 91.8% at $0.0069 per 100 rowsClef Flashpplx-decider: 91.6% at $0.0024 per 100 rowspplx-deciderkev-4b: 88.1% at $0.0024 per 100 rowskev-4bSolar Decide: 81.4% at $0.0044 per 100 rowsSolar Decide
Accuracy against cost per 100 rows. Up and to the left is better. The cost axis is logarithmic: each gridline step is a multiple, not a fixed amount.

At these prices the whole test is cheap: 1,000 rows on d1 cost about two cents. The spread still matters at volume. Clef costs about eight times as much as d1 and scores three points lower. Clef Flash costs about a third as much as Clef and scores nearly the same.

By category

Recall is the share of a category’s rows a model got right. The table shows the six categories where the models differed most. Every model scored 100% on federal tax and above 90% on charitable and education. Medical ranged from 82.5% (Solar Decide) to 96.3% (Clef). The number under each heading is how many rows the category has.

Model Wages (38) Interest (41) State and local tax (45) Student loan (30) Transfer (46) Personal (594)
Jev 97.4% 100% 100% 91.1% 71.0% 96.4%
d1 94.7% 100% 100% 93.3% 70.3% 96.1%
Clef 97.4% 100% 100% 83.3% 87.0% 89.9%
Clef Flash 97.4% 100% 100% 86.7% 67.4% 90.7%
pplx-decider 97.4% 100% 100% 76.7% 82.6% 90.4%
kev-4b 53.5% 48.8% 73.3% 83.3% 31.2% 96.1%
Solar Decide 97.4% 100% 99.3% 94.4% 59.4% 75.5%

Transfers were the hardest category. Only Clef (87%) and pplx-decider (83%) got more than 75% of them right; the rest scored between 31% and 71%. kev-4b missed about half of the wage and interest rows. Small categories move fast: with 25 to 46 rows, a row missed in all three passes changes recall by 2 to 4 points, and a row missed in one pass changes it by about 1.

Real names against added names

The real names are raw strings from bank exports, which are messier than an organization’s official name: think SQ *BLUE BOTTLE 0423 rather than “Blue Bottle Coffee” (an illustrative example, not a row from the set). Splitting the results by source shows which models handle them:

Model Real names (655) Added names (345)
Jev 94.2% 98.3%
d1 93.9% 97.9%
Clef 89.8% 97.1%
Clef Flash 89.6% 95.9%
pplx-decider 89.2% 96.2%
kev-4b 93.4% 77.9%
Solar Decide 73.9% 95.7%

Most models did better on the clean added names. kev-4b went the other way: it read raw bank strings nearly as well as the leaders but dropped to 77.9% on the added organizations. Solar Decide did the reverse and fell to 73.9% on real names.

What this does and doesn’t show

The labels came from mapping existing categories and haven’t been reviewed by a person, so some “wrong” answers may be label errors. A third of the set is names I added, which a real transaction history wouldn’t contain. And each model saw only the payee and a rounded amount; with a date or a memo line, the rankings could change.

For this task as tested, I’d use Jev or d1: d1 if cost matters most, Jev if speed does. Either one still leaves a few rows a person should check, and I’d rather a model tell me it isn’t sure than guess wrong.