These generative models are wonderful because they can automate everything. The only problem is they're not 100% accurate — and they never will be. Now, there are ways you can build tests and evaluation pipelines around models to get things closer to 100%, but not everybody's doing that. A lot of people who are new to generative AI are like, oh boy, this is amazing — I threw all my financial figures in and it came out with calculations that look correct. That's great. But if they take a magnifying glass to those numbers, what they're going to realize is that every now and then — it's rare, but it happens — the model just makes up a number. And I've got to tell you, as soon as accounting finds out, they're going to ask you to stop using it. Or if it's accounting that's using the AI, they'll probably just find a way to quietly change some of the numbers.
I did a study of a calculation recently — a calculation that can be done procedurally but can also be done with an inference engine. It was related to a date calculation. I created a test and ran it a hundred times without an LLM, just using a procedural process, and it was right a hundred out of a hundred. Then I built a skill to do the same calculation — not a prompt, a skill. I had variations of it: one that contained code for the calculation, one that just contained natural language describing the process, and a few other approaches to explore the accuracy of skills versus procedural functions. I ran each variation a hundred times with Opus, which at the time was one of the most expensive models available. The version with code was right 96% of the time — which is an important number to call out, because even for a high-priced model, we're still looking at a 4% error rate, and if you're talking about financial numbers, 4% is a pretty large number. When I dialed it down to Sonnet, the error rate went up to about 9%. And when I ran it on Haiku, it was about 12%.
I can make that error rate go down further if I increase the reasoning level, but that also increases the tokens it has to spend, the number of cycles, and the latency. The original procedural solution took less than a millisecond to run and the cost was negligible. If I wanted to get up to 99 or 100% accuracy with Opus, I had to run it at high reasoning with multiple iterations. The ultimate cost was about 10 cents per run, and each run took about 9 seconds. Now, 10 cents might not seem like a lot of money, but running something a million times at 10 cents — that's $100,000. And the time? Nine million seconds. That's over 104 days of compute, running nonstop. Now let's turn back to the procedural solution — the one that was right a hundred out of a hundred. It took about 200 microseconds to run, which is 0.2 milliseconds. Run that a million times and you get 200 seconds — about 3 minutes. What's the cost? Let's say your laptop costs $4,000 and you amortize that over three years of use. Three minutes of compute on that laptop, including the electricity, costs you less than a penny. So let me put that side by side: for accuracy with Opus, you can spend $100,000 and wait 104 days. With the procedural function, you spend less than a cent and wait 3 minutes. If you really want accuracy, you're going to have to spend a fantastic amount of time and money — or you can just write the function. So which one do you want? Do you want to wait 104 days, or do you want to wait 3 minutes? Do you have a check you can write me right now for $100,000?
The real choice when it comes to solving the Probability Pixie is knowing when to use the model and when not to. The Pixie isn't just messing up numbers — it's messing up decisions throughout your system. Unless you've invested the time to make AI correct itself, you could be looking at a real problem. The Probability Pixie loves to fly in at the most unexpected moment and screw up your calculations.