Natural Language Processing
The technology that allows computers to understand, interpret, and generate human languages in a meaningful way is called natural language processing (NLP). You use NLP in everyday life when chatting with automated customer service agents, chatbots, and virtual assistants.
In this lesson:
- What is natural language processing?
- How is NLP used?
- How does NLP process text?
What is natural language processing?
NLP is a sub-field of artificial intelligence that focuses on understanding and processing natural language, enabling AI to generate human-like responses in conversations. NLP is critical to conversational AI.
Natural language processing uses machine learning methods such as data extraction and model training to create a “foundation model.” NLP then applies specialized language tasks to create a “large language model” like ChatGPT.
Large language model
A large language model (LLM) is trained on vast amounts of text data to understand and generate human-like language. An LLM can:
- Understand human language
- Interpret the meaning
- Generate a human-like response
LLMs have advanced language capabilities, including language generation, contextual meaning, summarization, translation, and adaptation.
How is NLP used?
Outside of conversational AI, NLP is used for spam email detection, recommendation systems, language translation, sentiment analysis, and text summarization.
Within conversational AI, NLP can process sentiment, slang, dialects, foreign languages, and computer code.
Example:
[User]: “The meeting today was a bear.”
[Chat]: “I’m sorry you had a difficult meeting.”
How does NLP process text?
While a full discussion of NLP is outside the scope of this training, let’s look at a few ways that NLP processes text.
- Tokenization
- Identifies words and characters (called “tokens”) within the text.
- N-grams
- Identifies commonly-used sequences of words (phrases).
- Part-of-speech tagging
- Assigns a grammatical category (noun, verb, adjective) to each word.
- Named entity recognition
- Identifies and categorizes named entities such as names of persons, organizations, locations, and dates.
- Sentiment analysis
- Determines sentiment expressed in text as positive, negative, or neutral.
These and other techniques enable NLP to receive, understand, and express text and speech in a natural, human-like way.
What’s next?
We’ve seen how artificial intelligence, machine learning, and natural language processing are used to build large language models like ChatGPT. This concludes the section on conversational AI fundamentals. In the next lesson, we’ll look at actual methods of AI chat, aka, “prompt engineering.”