You can measure whether an AI tool is paying off with one number tied to one task. Write down the baseline today, check it again in 90 days, and cancel the tool if it has not saved time or made money. Most large companies skip this step: a16z's State of Markets II found about 69% of S&P 500 companies have deployed AI and about 30% report some impact, but only about 2% disclose a metric they track over time.
A retirement account with no balance statement
The episode opens with an analogy. Imagine putting money into a retirement account every month and never receiving a statement. You would never accept that from your bank, yet most of us accept it from our AI tools. The a16z figures make the same point at scale. Deployment is common, claimed impact is much rarer, and a tracked metric disclosed over time is rarer still. The 2% figure is about disclosure, so it does not prove those companies measure nothing internally. It does show how little most of them can point to.
Source: a16z State of Markets II, with the 2% figure reported by The Neuron.
This gap is 16 years old
The measurement problem did not start with generative AI. Rexer Analytics' data scientist surveys, run between 2007 and 2023, found the same pattern across every tool generation: only about 32% of models were usually deployed, and only about 41% of teams measured their return. The research summary behind the episode traces the cause to business alignment rather than technology. Teams skipped baselines, lacked buy-in from the people who would use the output, or never tied the tool to a specific outcome.
Source: Rexer Analytics Data Science Survey.
Where the time savings go
Glean's Work AI Index describes a pattern it calls botsitting: workers report saving hours with AI, then spend a large share of that time supervising, fixing and reformatting the output. The episode's example is a small business owner who pays for a tool to write weekly client reports. The tool hallucinates or breaks the formatting, the owner spends an hour cleaning it up, and the saving disappears. If you only count the minutes the tool took to draft, the tool looks like a win. If you count the cleanup, it may not be.
Source: Glean on AI time savings.
Is a 5% win still a win?
The episode pushes back on its own advice here. Say a ten-step workflow takes ten minutes per step and AI cuts one step from ten minutes to five. That is a 5% saving on paper. In practice, the friction of logging in, prompting, checking the work and moving data between human and AI steps can eat the five minutes. We reach for isolated tasks because redesigning a whole workflow takes uncomfortable strategic thinking. The fix is to measure the whole task end to end, not the step the tool touched.
Treat AI as leverage
Leverage means doing the heavy lifting once, such as data wrangling or formatting, and getting paid for it repeatedly, while you keep the human decisions: pricing, hiring, which projects deserve your taste. AI handles repetitive execution. You handle the strategic thinking.
The one-number test
Pick one number tied to one task. Hours saved drafting emails, or cost per marketing asset.
Baseline it with today's date. Write the current value down before you automate anything.
Calendar a remeasure in 90 days. Compare against the baseline, not against how the tool feels.
Decide. If it is not saving you time or making you money, cancel the subscription.
Web guides on small-business AI ROI converge on a simple formula: hours saved times your true hourly rate, minus the subscription, minus the hours you spent learning the tool, tracked for about 90 days. If the tool touches customers, add a customer metric such as retention or resolution, because time saved means little if it costs you customers. Treat survey statistics quoted in social posts as secondhand until you can trace them to a source.
FAQ
How do I measure AI ROI as a solo creator?
Pick one task, record its current time or cost, run the AI workflow for 90 days, and compare. Subtract the subscription and your learning time from the value of the hours saved.
What does it mean that about 2% of companies disclose an AI metric?
In a16z's State of Markets II, about 2% of S&P 500 companies disclosed a metric they track over time, compared with about 30% reporting some impact. It is a disclosure figure, not proof that the rest measure nothing.
What is botsitting?
Botsitting is the time spent supervising and cleaning up AI output, which can cancel out the hours the tool appeared to save.
When should I cancel an AI tool?
When your 90-day remeasure shows no gain in time saved or money made against the baseline you recorded on day one.
Companion episode:
Want more practical AI breakdowns for small teams? Get the free playbooks and automations at joebuildsai.com.