Alan is the co-founder and CTO of Rasa. Rasa builds the standard infrastructure layer for AI, providing the infrastructure and tools necessary for high-performing, resilient, proprietary contextual assistants that work. With Rasa, all developers can create better text- and voice-based assistants. The company created Rasa Open Source, a free open-source machine learning framework to automate text- and voice-based assistants. It’s now used by millions of developers globally.
You can take a look at Alan’s talk in detail including the Q&A session, below
Key themes at Rasa:
- Champion open source tools
- Nuture a community of makers
- Ship applied research to solve real problems.
Alan notes “We invest very heavily in the community, making sure that people can learn from each other. And we do a lot of applied research because there are a lot of unsolved problems in this space. And we care a lot about fixing those things and bringing those things into production, to make sure that everyone can use the latest research and benefit from the latest developments in the field.”
Rasa has developers in almost every single country in the world, and over eight million downloads, and three 3 million developers that love and use Rasa.
Conversation driven development (CDD)
Why:
- Building conversational AI is hard.
- Getting a prototype up and running is not hard. That’s not the hard part, right? It’s easy
- That hard parts show up when you want to go from a prototype to something that could be shipped.
Conversation driven development is to have a way so that people who are new to the fields don’t have to learn the lessons of those that came before them the hard way, so that everyone can kind of skip ahead and start Working on the interesting problems, and not kind of keep making the same mistakes every time new people come to the field.
CDD is made up of 6 actions:
Share: As early as possible, as soon as you have the most basic things working, go and give that prototype to people and get them to test it out.
Review: it doesn’t matter if you’re just in the prototyping stage or you’ve been in production for a month. It’s always valuable to spend time looking at these conversations and reading them, like look at what people are saying to your assistant.
Annotate: look at what people are saying, collect real training data from real users, and add labels to it.
Test: End to end tests: Send a sequence of user inputs, messages or audio clips into your system and check that, you know, the whole conversation is handled the way that you expect it to. And you should run those on continuous integration, the same as you would do any other software testing, check your coverage, all the rest of it.
Track: “This can be quite tricky. But it’s important to come up with some kind of proxy measures, and kind of approximate ways to understand which conversations are being successful, and which ones are not being successful. Negative signals are useful too, e.g. users not getting back in touch with support.
Fix: “It doesn’t matter how advanced your project is, it will always fail. There will always be cases where your conversation doesn’t go the way that you hoped it would. And so when you’re reviewing these conversations turn successful conversations into a test. When you encounter things that aren’t working, because there’s a bug in your code, because you don’t have enough data for training your model, go and fix those things, right? And then maybe add a new test as well if you can, and ensure that you’re continuously updating your system in fixing all the places where things break.
It’s normal and good to be jumping between these tests!
Alan notes that all of this is is often a team effort because it really requires a mix of skills. “So you need to go and work together between software engineering, and design and product and all these folks kind of working together on improving the AI.”
Rasa X layers on top of Rasa Open Source
Rasa Open Source is a framework for natural language understanding, dialogue management, and integrations. Rasa X is a free toolset used to improve contextual assistants built using Rasa Open Source. Together, they include all the features to create great text- and voice-based assistants and chatbots.
The first simple but important feature of Rasa X is just that you can share your assistant or your bot or what have you built, or you can just share it with people with just a link All of those conversations, whether they’re from the sharing link, or rather than coming in from Facebook, or SMS show up inside Rasa X and you can see them. You can filter by the length of the conversation, by the day, by the channel, by particular intents which were predicted to get actions which were taken by the amount of confidence that the model had,
NLU inbox: all the messages that came in where the language understanding model made some predictions are available for the annotate step. And all of these messages that are being sent to the assistant can become your training data.
Rasa integrates with git meaning you can push new data to git and trigger your CI pipeline. “So in a test I take all the data that I now have, and I train a model, I run through my end to end tests to make sure that they all work correctly, that I haven’t broken anything, that all the most important conversations will still work. I do cross-validation of the NLU model, which is a technique for trying to estimate how well my model generalises outside of the data that I’ve seen, and then it posts those results as a comment onto the pull request.
Then, before I merge these new annotations in or whatever changes I’ve made, I can check that I’m actually improving my system, I haven’t broken anything, I’ve maybe improved performance and have added some more data to some of these examples. And its a strong QA process for kind of going through and making sure that I keep improving my system.”
Track failures and track successes
When you’re reviewing conversations, you can add tags to individual conversations both manually and by the API. This is a really important piece because if something is happening outside of the conversation you can bring that information back in via Rasa X.
With a single click, you’re can save any conversation that anybody had and turn it into an end to end and test right which means that it gets added to your CI/CD pipeline.
This is just some of Alan’s presentation, take a look at the video above to enjoy all the whole session including the Q&A session where he answers questions from the audience.
