Last year, during an online conference, I met Alan Nichol, CTO of Rasa Technologies. I got fascinated by his Agile approach to developing Conversational AI, called CDD - Conversational Driven Development, so I started exchanging ideas with him.
Rasa.com published their method to develop Conversational AI applications, called Conversation Driven Development (CDD).
It is the process of listening to your users and using those insights to improve your AI assistant. CDD is composed of 6 actions:
- Share: Give your prototype to users to test as early as possible. People will always surprise you with what they say. Too many teams spend months designing conversations that will never happen.
- Review: Take time to read through the conversations people have with your assistant. It’s helpful at every stage of a project, from prototype to production. Too many teams get caught up looking exclusively at metrics (like “what % of users express intent X?”).
- Annotate: Improve your NLU model based on messages from real conversations. Coming up with examples yourself or generating synthetic samples with a paraphrasing approach can help you bootstrap. But when you’re going into production, less than 10% of your data should be artificial.
- Test: Use whole conversations as end-to-end tests of your assistant. Professional teams don’t ship applications without rigorous testing. When you go into production, you should have dozens of end-to-end tests covering the most critical conversations. Use continuous integration and deployment to ship updates reliably.
- Track: Come up with a way to identify successful conversations. For example, a user taking action (like signing up for your service) or not taking action (like not getting back in touch with support within 24 hours). Use that data to tag and filter conversations to understand what’s working and what’s not working.
- Fix: Study conversations that went smoothly and ones that failed. Successful conversations can become tests right away. Unsuccessful talks show you where you need more training data or where you need to fix your code. Track the different ways your assistant fails, so you know you’re reducing failures over time.
According to Alan, «the shift to CDD is analogous to switching from waterfall to agile development. In both cases, you cannot anticipate what's ahead, so it's better to lean into that uncertainty and build an iterative process. Agile is necessary because you can't plan ahead for how long every feature will take to build. CDD is necessary because you can't anticipate all the things users will say to your assistant.»
I love the formulation of CDD, with “Share” as the first step. One could argue that the first step should be “Build,” but my learning is that, instead, in traditional software development, you develop the software (and you test it) to put it in production finally.
With Conversational AI, you put it in production to develop it (and test it). Kudos to Alan and the people at Rasa to capture this crucial nuance so clearly in the CDD formalization.