Agile doesn’t work for AI
Oh, them's fighting words. Or maybe you're in the camp that would say, "Well, duh, Agile doesn't work at all."
But not so fast. The higher principles behind Agile very much apply. Avoid waste. Take small, safe steps. Have tight feedback loops to make sure you're building the right thing.
Where things break down is when rigidly codified "Agile" practices—which might make total sense when applied to standard software development—are applied without modification to AI projects.
Some examples:
AI development is much closer to research than development, so everything's a spike in Scrum terms. You might as well not bother, then.
A large feature does not intuitively break down into smaller pieces. Case in point: to build an image recognition model, you don't start with a model that can recognize one category and then slowly add additional categories. In fact, it's often the opposite: You start with a model that can recognize hundreds of categories, then you throw those all out and replace and fine-tune for those you particularly care about.
There are non-intuitive discontinuities in mapping a user story to the required effort. Minor changes to the requirements turn "sure thing, give me an afternoon" to "uhm, give me millions in VC and five years" (Hat tip to XCKD, whose predicted timeline for image recognition turned out just about right).
Test-driven development (TDD), a fantastic practice for software development, does not productively apply to ML. Sure, you can TDD that your outward plumbing around the machine learning system is correct, but you can't TDD your way to, say, better performance on a relevant model eval.
So what are we going to do about it? I'd say the community at large is still figuring that out. I'm hoping to add my own thoughts to the discussion over time. It would be a shame if the solid principles were to get thrown out due to frustration with the concrete practices.
