How NLP Text-Based Chatbots Work

As we wade our way carefully into the new decade, there’s one term you can expect to be hearing a lot more of: chatbots.

What you’ll learn from this article:

  • Buggy chatbots that have no recollection of previous conversations are called rules-based chatbots.
  • Natural Language Understanding is a branch of NLP that’s all about machine reading comprehension.
  • Natural Language Generation and Natural Language Interaction are more subsets of NLP.
  • The most important elements of an NLP-powered chatbot include dialogue management, human hand-off, business logic integration, rapid iteration, training and iteration, NLP, and simplicity.
  • Intelligent chatbots function with 5 major steps: tokenising, normalising, recognising entities, dependency parsing, and generation.

Skip Ahead:

  • Rules-based chatbots
  • Difference between NLP, NLU, NLG, and NLI
  • Next-generation chatbots powered with NLP
  • How intelligent chatbots work
  • Keeping an eye on your chatbot

  • And if the word “chatbots” conjures cringe-worthy recollections of stilted, disjointed, and altogether unnatural conversations you’ve been forced into with chatbots in times past, worry not. The chatbots of today are modern, sleek, sophisticated, and, with a bit of machine learning technology woven into them, can feel altogether, well, human.

    It’s true: today’s chatbots can hold a conversation that looks and feels like you’re having a conversation with a real, live human. They’re not out to trick anyone, of course. These chatbots 2.0 are, instead, helping human agents by augmenting their work and stepping in on the tedious, repetitive communications that traditionally took up a huge portion of the customer service rep’s time. This, in turn, frees up the human agent to concentrate on those more complex cases that do require human input. But how does this all work? What is the machine learning that’s taking place behind the scenes, and what other elements are at play when building an intelligent text-based chatbot?

    Understanding how your NLP text-based chatbot works will help you not only use it but also keep an eye on the technology (which is a must-do) to ensure it stays on track as it goes to work for your company.

    >> Learn everything you need to know about chatbots in Ultimate Knowledge.

    Rules-based chatbots

    The stilted, buggy chatbots of yesteryear are what’s known in the industry as “rules-based chatbots”. These bots are often compared to goldfish because they don’t remember the previous interactions with your users. This is because they are powered by an ultra-simple machine learning technique called “pattern matching”. So, take this question, for example: “What is the price of your membership?”

    This question is an example of a pattern that can be matched together with similar questions that will be asked by customers in the future. The rules-based chatbot is taught how to respond to questions -- but the wording must be an exact match. That means it’s necessary to manually program all the different ways to ask how much a membership costs -- and do this for every possible question a customer may ask. That’s a grand, time-consuming task, to say the least. That’s not to say this type of chatbot can’t be useful: if your company tends to get only a certain number of questions that are usually asked in just a few ways, then a simple rules-based chatbot is probably the way to go. But for most companies these days, this technology is not powerful enough to keep up with their customer queries.

    Difference between NLP, NLU, NLG, and NLI

    There are a whole lot of acronyms in the world of automation and AI, and the technology behind chatbots is no different. Look at four key terms that will make your chatbot an unstoppable customer service tool.

    NLP: Natural Language Processing is a branch of AI that helps computers read and understand natural human language. Its main goal is to improve human-machine communication.

    >> Learn more about NLP in Ultimate Knowledge.

    NLU: While NLP is a branch of AI, NLU is a branch of NLP. Natural Language Understanding is all about machine reading comprehension, and making sure the machine understands what the text it’s processing really means. In more scientific terms, NLU takes place when the machine converts the user’s inputted data (what they’re saying) into a logical form that’s understandable to computer algorithms.

    >> Learn more about NLU and the customer experience here.

    NLG: Natural Language Generation is another subset of NLP, which is essentially NLU in reverse: the machine generates a logical response which it then converts to a natural language response so that it’s easily understood by the human reader.

    NLI: Natural Language Interaction -- another branch of NLP -- refers to the above two steps in combination, or in other words, the communication that takes place between humans and machines that is translated from programming language to human language, and back again, like a very complex Google Translate for machines.

    Next-generation chatbots powered with NLP

    Enter the next generation of chatbots. These NLP-powered assistants are pretty smart -- and they become more so by the day. If rules-based chatbots are like goldfish, then these intelligent chatbots are like elephants. They carry information from one conversation to the next and learn as they go. Here are some of the most important elements of an NLP-powered chatbot:

    • Dialogue management: This tracks the state of the conversation. The core components of dialogue management in chatbots include a context (saving and sharing data exchanged in the conversation) and session (one conversation from start to finish).
    • Human handoff: This refers to the seamless communication and execution of a handoff from the chatbot to a human agent.
    • Business logic integration: It’s important that your chatbot has been programmed with your company’s unique business logic.
    • Rapid iteration: You want your bot to be slick and easily programmable, which the bots of today generally are. Rapid iteration refers to the fastest route to the right solution.
    • Training and iteration: To ensure your chatbot doesn’t go awry, it’s necessary to systematically train and send feedback to improve its understanding of customer intents. This refers to the real-world conversation data being leveraged across channels to do so.
    • Natural Language Processing: Your chatbot’s NLP works off the following keys: utterances (ways the user refers to a specific intent), intent (the meaning behind the words a user types), entity (details that are important to the intent like dates and locations), context (which helps to save and share parameters across a session), and session (one conversation from start to finish, even if interrupted).
    • Simplicity: To get the most out of your bot, you’ll want it to be set up as simply as possible, with all the functionality that you need -- but no more than that. There is, of course, always the potential to upgrade or add new features as you need later on.

    How intelligent chatbots work

    Now it’s time to really get into the nitty-gritty of how today’s intelligent chatbots work. There are five major steps involved -- tokenising, normalising, recognising entities, dependency parsing, and generation -- for the chatbot to read, interpret, understand, and formulate and send a response. Let’s take a closer look.

      Keeping an eye on your chatbot

      One of the most impressive (and useful) things about intelligent chatbots is that they learn from and get smarter with each interaction. That said, it must be remembered that machine learning chatbots are still in primary school, and should be monitored carefully. NLP technology is prone to bias and error and can learn to speak in an offensive manner. Now that you understand the inner workings of NLP, machine learning, and chatbots, you’re ready to build and deploy your new chatbot baby to the world. Allow her to stand at the front line of your customer service team, as your polite and clever representative, who handles your customer queries not only efficiently, but -- very importantly -- naturally.

      1) Tokenising: The chatbot starts by chopping up text into pieces (also called ‘tokens’) and tosses away punctuation.

      2) Normalising: Next, the bot finds common misspellings, slang spellings, or typos in the text and amends these to its “normal” version.

      3) Recognising entities: Now that the words are all normalised, the chatbot seeks to identify which type of thing is being referred to. For example, it would identify North America as a location, 67% as a percentage, and Google as an organisation.

      4) Dependency parsing: For the next step, the bot splits the sentence into nouns, verbs, objects, punctuation, and common phrases.

      5) Generation: Finally, the chatbot generates a number of responses using the information determined in all the other steps, and the bot selects the most appropriate one (also based on the information received in the previous steps) and sends it to the user.




          Similar Articles

          Discover the power of

          Schedule a demo and one of our specialists will be in touch to show you how can transform your business.

          Chat with an Expert