Market Risk vs Product Risk
To follow up on why standard Agile practices don't quite work for AI, one observation is that Agile is good at managing market risk, but AI also has a lot of product risk.
Market Risk: Will anyone buy the damn thing?
Product Risk: Can we even build the damn thing?
Pure Market Risk
Much of the software built over the last few decades had pure market risk. The pressing question was never whether it was technically feasible—some interesting engineering challenges at scale notwithstanding—to build the thing but whether anyone would care enough to use it. This is where Agile methodologies and the Lean Startup movement shine, with their Minimum Viable Products and incremental iterations.
In short, If you only have market risk, iterate away as fast as you can.
Pure Product Risk
Let's imagine the other extreme. If you're a biotech company working on a cure for just about everything, you don't need focus groups, usability studies, or small-scale pilots. You have a tough road ahead of you, but you know there'll be an eager market once you get there. Defining intermediate goals and milestones still makes sense, and you can have successful releases of these intermediate products. Still, the micro-iterations of the Lean and Agile camps are a distraction.
In short: If you only have product risk, don't bother with quick customer-facing iterations.
Why not both?
And that brings us to AI products, which combine both risks.
Can you even build the thing? Are current models sufficiently capable, or can you train your own and achieve the required accuracy?
Once done, will people care about the AI-enabled product?
In such a situation, we're prevented from iterating rapidly. However, we can still use the general ideas from Agile and Lean and make deliberate bets and experiments to validate or invalidate our hypothesis. We just have to accept that the risk with each bet is higher. The increased product risk forces us to move slower, which, in turn, limits how quickly we can eliminate market risk.
Iteration with a Long-Term View
So, rather than iterating and pivoting mindlessly, the way forward with AI products is to have a compelling long-term view (a vision!) of what you want to establish. If it is sufficiently bold, this vision can help mitigate some market risks. It should be possible to have an educated opinion about whether people want to buy the result of this bigger vision. In the meantime, the vision helps guide the—now somewhat slower—product iterations as we move the product from one intermediate milestone to the next.
In short, If you have both product and market risk, slow down the cycle of iterations so you can make meaningful progress before each customer-facing release.
