Agile when AI is involved
One of the reasons why it's so important to have good intuition about what AI approach is correct for your problem is that they have vastly different complexity and timescale:
ChatGPT with the right prompt is good enough? You can be done in a week or two.
Need to fine-tune a model on hard-to-get data in a messy format and integrate into a custom internal solution? We’re looking at several months.
The agile principles caution us to move forward in small, incremental steps. That’s fine and good. But it’s still preferable to not go into this agile discovery mode flying blind. The point of agility is to be able to respond to unknown unknowns, the surprises and curveballs, not to waste time rediscovering the wheel.
Even then, of course, there’s uncertainty involved. With AI, we’re shifting more towards science rather than engineering. That means running lots of experiments. Here, the plan is simple: Treat “approach selection” as its own, experimental phase in the project and run the cheapest, fastest experiments first. Expect that a lot of this early work will end up getting tossed out. That’s fine. We’re invalidating hypotheses, Lean Startup style.
That leads us to the first important principle: Fast experiments require fast feedback. And that means building out a robust evaluation framework before even starting work on the actual problem: What are the success criteria, and how do we tell whether solution A or solution B works better, in a matter of minutes instead of days?
The next idea is to start with the simplest thing that could conceivably work, and then get a lot of automated feedback on it. If we’re lucky, the simple approach is already good enough. If not, at least we’ll know exactly where it breaks down. And that means we can go to the next, more complex, step with a good idea of what to pay attention to.
Finally, we need to know when it’s time to stop experimenting and start shipping. That requires intuition, because we have to stop experimenting before the final version of the AI tool is done. We need to trust that hitting “almost good enough” in the experiment phase will let us get to “definitely good enough” in the next phase.
Getting to that final, complex, solution might still take several months. But with the above way, as long as we aren’t just blindly thrashing around, we will have delivered value at every step along the way.
