Evolutionary Design for AI Systems
In an earlier post, we saw how complex systems come from simple systems. This practice of evolutionary design is a cornerstone of agile technical practices. Start so simple that it's almost embarrassing and add complexity slowly, one step at a time.
In typical software, that means ditching features and particular execution paths early on, but in AI, that doesn't quite work. If you're building a plant identification app, there's not much sense in deciding to start with support for only two types of plants. It doesn't simplify the tasks of model training and deployment. Plus, chances are you'd have to significantly change the model architecture and training parameters at each step, throwing away the previous work.
So, what do we do instead? We apply evolutionary design to our machine learning infrastructure and don't initially worry about releasable "MVPs." This approach is recommended in Google's Rules of Machine Learning and Andrej Karpathy's great Recipe for Training Neural Networks.
The first step, the evolutionary seed, is setting up the end-to-end training and evaluation skeleton, plus picking a "dumb" baseline
We run a number of sanity checks to make sure things are set up correctly
We gradually increase the model's complexity until we hit diminishing returns
The mental picture I have of this process is that, in traditional evolutionary design, you're assembling small Lego pieces into a complex whole, whereas for designing an AI system, you're slowly bringing a blurry image into sharp focus.
