How NLP Text-Based Chatbots Work

Not all chatbots are created equal. Here’s your crash course on chatbot types and lingo.

What you’ll learn from this article:

  • Natural Language Processing versus Artificial Intelligence
  • The different types of chatbots: rule based chatbots versus intelligent chatbots
  • The five steps that make Intelligent chatbots function

If the word “chatbot” conjures memories of frustrating and unnatural conversations, worry not. The chatbots of today are sleek and sophisticated. In fact, with machine learning technology, they can even feel human.

These chatbots 2.0 not only provide a better user experience, they also help human agents by taking over repetitive and time consuming communications. This frees up the human agent to concentrate on those more complex cases that require human input. But how does this all work? What is this machine learning that’s taking place behind the scenes?
Understanding how your NLP text-based chatbot works will help you ensure it stays on track as it goes to work for your company.
>> Learn everything you need to know about chatbots in Ultimate Knowledge.

The Difference Between NLP, NLU, NLG, and NLI

There are a whole lot of acronyms in the world of automation and AI, and the technology behind chatbots is no different. Here are four key terms that you need to know.

NLP, or Natural Language Processing, is a branch of AI that helps computers read and understand natural human language. Its main goal is to improve human-machine communication.
>> Learn more about NLP in Ultimate Knowledge.

NLU, or Natural Language Understanding, is a branch of NLP. It is all about machine reading comprehension, and making sure the machine understands the text’s actual meaning. In more scientific terms, NLU takes place when the machine converts the user’s inputted data (what they’re saying) into a logical form that’s understandable to computer algorithms.
>> Learn more about NLU and the customer experience here.

NLG, or Natural Language Generation, is another subset of NLP, which is essentially NLU in reverse: the machine generates a logical response which it then converts to a natural language response so that it’s easily understood by the human reader.

NLI, or Natural Language Interaction, another branch of NLP, refers to the above two steps in combination, or in other words, the communication that takes place between humans and machines that is translated from programming language to human language, and back again, like a very complex Google Translate for machines.

Rules-based chatbots: The Old School

The stilted, buggy chatbots of yesteryear are called rules-based chatbots. These bots don’t remember the previous interactions with your users because they are powered by an ultra-simple machine learning technique called pattern matching. So, take this question, for example: “What is the price of your membership?”

This question is an example of a pattern that can be matched together with similar questions that will be asked by customers in the future. The rules-based chatbot is taught how to respond to questions—but the wording must be an exact match. That means it’s necessary to manually program all the different ways to ask how much a membership costs, for every possible question a customer may ask. That’s an incredibly time-consuming task, to say the least.

That’s not to say this type of chatbot can’t be useful: if your company tends to get only a certain number of questions that are usually asked in just a few ways, then a simple rules-based chatbot is probably the way to go. But for most companies these days, this technology is not powerful enough to keep up with their customer queries.

NLP Powered Chatbots: The Next Generation

The new generation of chatbots are NLP-powered agents that get smarter each day. They carry information from one conversation to the next and learn as they go. Here are some of the most important elements of an NLP-powered chatbot:

  • Dialogue Management: This tracks the state of the conversation. The core components of dialogue management in chatbots include a context —saving and sharing data exchanged in the conversation—and session —one conversation from start to finish.
  • Human Handoff: This refers to the seamless communication and execution of a handoff from the chatbot to a human agent.
  • Business Logic Integration: It’s important that your chatbot has been programmed with your company’s unique business logic.
  • Rapid Iteration: You want your bot to be slick and easily programmable, which the bots of today generally are. Rapid iteration refers to the fastest route to the right solution.
  • Training and Iteration: To ensure your chatbot doesn’t go awry, it’s necessary to systematically train and send feedback to improve its understanding of customer intents. This refers to the real-world conversation data being leveraged across channels to do so.
  • Natural Language Processing: Your chatbot’s NLP works off the following keys: utterances (ways the user refers to a specific intent), intent (the meaning behind the words a user types), entity (details that are important to the intent like dates and locations), context (which helps to save and share parameters across a session), and session (one conversation from start to finish, even if interrupted).
  • Simplicity: To get the most out of your bot, you’ll want it to be set up as simply as possible, with all the functionality that you need — but no more than that. There is, of course, always the potential to upgrade or add new features as you need later on.

How intelligent chatbots work

Now it’s time to really get into the nitty-gritty of how today’s intelligent chatbots work. There are five major steps involved—tokenizing, normalizing, recognizing entities, dependency parsing, and generation—for the chatbot to read, interpret, understand, and formulate and send a response. Let’s take a closer look.

  1. Tokenizing: The chatbot starts by chopping up text into pieces (also called ‘tokens’) and tosses away punctuation.
  2. Normalizing: Next, the bot finds common misspellings, slang spellings, or typos in the text and amends these to its “normal” version.
  3. Recognizing Entities: Now that the words are all normalized, the chatbot seeks to identify which type of thing is being referred to. For example, it would identify North America as a location, 67% as a percentage, and Google as an organization.
  4. Dependency Parsing: For the next step, the bot splits the sentence into nouns, verbs, objects, punctuation, and common phrases.
  5. Generation: Finally, the chatbot generates a number of responses using the information determined in all the other steps, and the bot selects the most appropriate one (also based on the information received in the previous steps) and sends it to the user.

Caring for your chatbot

One of the most impressive things about intelligent chatbots is that they learn from and get smarter with each interaction. However, in the beginning, machine learning chatbots are still in primary school and should be monitored carefully. NLP technology is prone to bias and error and can learn to speak in an offensive manner.

Now that you understand the inner workings of NLP, machine learning, and chatbots, you’re ready to build and deploy your new chatbot baby to the world. Allow her to stand at the front line of your customer service team, as your polite and clever representative.


Similar Articles

Discover the power of

Schedule a demo and one of our specialists will be in touch to show you how can transform your business.

Get Started