a big virtual tutoring provider just shut down because they couldn't show evidence it actually worked. that's the whole story. the product existed, people used it, and when the moment came to prove the thing it claimed to do, there was nothing on the table. i keep coming back to this because it's the failure mode i'm most likely to walk into myself, and because it doesn't announce itself early. it announces itself at the exact moment you can't fix it.
here's the situation as i understand it from the outside. a company builds something in edtech. the pitch is that it moves an outcome. usage goes up, teachers say nice things, renewals come in. every one of those signals feels like proof while you're inside it. and then someone with money asks the direct question, and the answer has to be assembled retroactively out of whatever data happened to survive. that assembly almost never works, because the data you needed was a decision you had to make before you shipped, and you didn't know you were making it.
the receipt is the shutdown itself. that's the number here, and it's a binary one. the market ran the experiment and published the result: no evidence, no company. i don't have a percentage or a cohort size to offer you, and i'm not going to invent one. what i have is a single observed outcome that's more expensive than any dashboard you could have built instead.
the mechanism is that the signals you collect by default are the ones that are easy to collect. usage is easy. retention is easy. sentiment is easy, because you can just ask. all three are downstream of whether people like the experience, and liking the experience is genuinely correlated with the outcome you care about, which is exactly what makes it dangerous. correlation is enough to keep you confident and not enough to survive a question. "teachers love it" is a fact about teachers. retention is a fact about your billing. the outcome you claim to move is a fact about students, and you have to go get it on purpose.
so the rule i adopted, and the one i'd hand to anyone building in this space before their next raise, is to pick one outcome you claim to move. one. resist the urge to name a portfolio. then define, on day one, how you would measure it at day ninety. write the measurement down while you still have no results, because the definition you write after seeing data is a different and much worse definition. instrument it before the feature ships, since instrumenting after means your baseline is gone and everything you learn is relative to a starting point you're guessing at.
the harder half is running a cohort that doesn't get the feature. it will feel wasteful and it will be small and messy, and small and messy is fine. a rough comparison group beats no comparison group by an enormous margin, because without one you have a number with nothing to hold it against. every trend you see becomes attributable to whatever you were hoping for. i've done this badly on my own projects, and the version where i ship to everyone at once always produces a chart that looks great and proves nothing.
then write the result down when it's bad. especially when it's bad. this is the part everyone nods at and nobody does, because a bad result is a thing you have to look at again later, and it's easier to let it quietly stop being measured. the discipline is that a written negative result is the only thing that makes your positive results credible. someone evaluating you can tell the difference between a company that measured and reported, and a company that measured until the numbers were good. the second pattern is visible from the outside and it reads as exactly what it is.
what to do with this: go look at your product today and ask what outcome you would actually put in front of someone who was skeptical. if the answer is usage or sentiment, you have the problem, and you have it right now while it's still cheap. the fix is one instrumented outcome, one measurement definition written before the data exists, one imperfect comparison group, and a habit of recording what you find. none of that is expensive. it's just work you have to do before you need it, which is the only kind of work that's easy to skip.