Prompt Engineering - Not here to last
In the early days of ChatGPT, social media was full of people sharing their tips and tricks for adequately prompting it to get the desired outcome. Some of those proved genuinely useful like Chain-of-Thought prompting (now baked directly into various "reasoning" models.) Others bordered on the superstitious, like an alchemist's magical incantations. Ideas like offering the LLM a tip for a job well done or threatening it with punishment.
These prompt techniques appeared so esoteric and beyond mortal comprehension that soon, there was news of Prompt Engineers earning USD 400,000/year. Now, I wonder if much actual engineering was involved, or if it was more a case of splatterprompting, throwing prompts at the model to see what sticks.
There are two issues with prompt engineering:
It is inherently brittle. The longer and more convoluted the prompt, the less likely it will survive a model upgrade and the more likely it will sidetrack the model.
Consumers, in particular, don't want to bother with it. They want to tell the AI to answer a question, not obsess over the intricacies and inner workings of the underlying model. As long as we need to prompt the AI just right, we can't call it natural language understanding.
As it turns out, newer and better models do much better with actual natural language. Now, for internal use of LLMs via an API, there'll always be a need to engineer the prompts to get the desired output with near-100% probability. The emphasis here is on engineering, with all the important principles we talk about on this newsletter:
Start simple. The shortest prompt that expresses the problem.
Add complexity one small step at a time.
Test and evaluate after each step.
Happy (non-)prompting!
