Big Data Was the Question. AI Is Sometimes the Wrong Answer.
For a decade the recipe was: collect everything, store it cheaply, query it later. Big Data infrastructure — Hadoop, Spark, the cloud lakehouse generation — made that practical at a price point that would have been absurd in 2010. The implicit assumption now is that AI will mine value from those lakes automatically. It usually does not.
Three failure modes recur. First, model accuracy is bottlenecked by label quality, not data volume — the petabyte of unlabelled clickstream is consistently less useful than a hand-curated 50,000-row dataset built by people who know the domain. Second, the "more data equals better model" intuition is a 2018 idea; modern transformer scaling laws favour data quality and diversity over raw size, especially at the fine-tuning stage where most enterprises operate. Third, most "Big Data" turns out to be long-tail Small Data once you decompose it — 200 separate dimensions of customer behaviour, none with enough signal individually to beat a competent rules engine.
So the real question shifts: when does AI actually add value over a thirty-line SQL query plus a domain expert? The honest answer: when the patterns are non-linear, the problem is recurring, and the cost of a wrong call exceeds the cost of an analyst-hour by an order of magnitude. For everything else — most reporting, most segmentation, most one-off investigations — BI wins on interpretability, audit-trail and operating cost. Pretending otherwise is how teams burn six months building an ML pipeline to replace a query that ran fine on Monday.
