What Is NLP?
In our research project DEEEP, funded by the Ministry of Economic Affairs of Baden-Württemberg, we are extending plusmeta with a Deep Learning component. In our blog series, we dive deeper with you into the topics we are working on in our research project. Today, you will learn how AI understands language.
Table of Contents
Natural Language Processing – Computers Learn to Speak
The field of artificial intelligence that deals with understanding, processing, and producing natural language is called Natural Language Processing (NLP). Another common term is "computational linguistics." This area of AI has experienced a genuine wave of innovation since 2012.
The reason for this is the increased availability of computing power, which makes it possible to train models on larger volumes of text. The amount of training data required for Deep Learning is very high — especially for a complex task like language comprehension. In addition, model architectures themselves have become increasingly better (and more complex). We explain which new approaches are particularly promising further below.
You may have already experienced this leap in innovation in your own everyday life. When you talk to Siri or Alexa, dictate messages in WhatsApp, or translate texts with DeepL — NLP is at work.
NLP is about processing human language. This processing can be broken down into sub-areas with different tasks. The AI methods, model architectures, and training paradigms used all depend on the task to be solved. The diagram below shows how broad the field of NLP is.
In plusmeta's use cases, the goal is for the AI to assign texts from Technical Documentation to an appropriate class. At document level, this could be a document type, for example. When looking at documentation modules (topics), we can determine an information type or the product lifecycle phase being described. This task is called "text classification." For texts to be correctly assigned to classes, the computer must first "understand" the text.
But how does an AI — or a computer — actually understand text? For this, a language model is required.
The Right Form: Language Models and Embeddings
In language models, algorithms transform unstructured, natural language into a machine-readable form. To do this, texts are segmented — or broken down into tokens. During this process, words are reduced to their base form, so that grammatical word modifications (e.g., inflections) no longer make a difference.
The information contained is then converted into numerical representations. There are different approaches to this transformation, which influence the complexity of the resulting model. In simpler models, each word is assigned a discrete value (bag-of-words). Synonyms and spelling variations must be merged through manual intervention. Text classification is realized here by matching the classes against the words contained in the index.
More complex models use embeddings to represent word meaning. This means that vectors are created which contain the semantic meaning of a word. The number of dimensions depends on the complexity of the model. Similar words like "big" and "small" are closer to each other in the vector space than the words "small" and "table." Word relationships can also be represented, so that "Berlin" relates to "Germany" in a similar way as "Madrid" to "Spain."
In addition, modern language models also capture the context of a word. This means they do not only look at the word itself, but also at the text before and after it, and take this into account when creating the word embedding. That context can be decisive becomes clear with ambiguous words, negations, irony, etc. A "bank," for example, can either be a place to withdraw money or a bench in the park, depending on context.
Embeddings are learned by training the language model on large quantities of text. In these texts, individual words are masked. The model then trains itself — adjusting the weights in the neural network so that it correctly predicts the masked words. After such training, the model knows, for example, that the word "bloom" is more likely to follow the word "flower" than the word "communicate."
So the model can now handle language. But can we now classify any text with the model — or answer any question we like?
Pretraining and Fine-Tuning
The answer is "no." The model still lacks the knowledge needed to solve specific tasks. This knowledge can be built up through fine-tuning. There are various approaches to this.
In classic fine-tuning, correctly labeled examples are used to teach the model a specific task. The model is given a new output layer that answers the specific question. Within the model, the weights are adjusted as needed for the new task. We examined this training process in more detail in our first blog post in this series.
Classic fine-tuning requires very large amounts of data. In our research project, we did have data from different companies. However, to meaningfully adapt a model with millions of neurons, thousands to tens of thousands of examples are needed.
Fortunately, modern AI research also knows methods that can get by with less training data.
Transfer Learning
Humans learn very quickly by applying existing knowledge to new experiences. We recognize patterns, form associations, and draw conclusions. Transfer Learning in fine-tuning follows this same idea: using already acquired, existing knowledge to solve new tasks.
Classic fine-tuning requires very large amounts of data. In our research project, we did have data from different companies. However, to meaningfully adapt a model with millions of neurons, thousands to tens of thousands of examples are needed.
Fortunately, modern AI research also knows methods that can get by with less training data.
Zero-Shot Learning
Zero-Shot Learning takes the transfer learning approach to the extreme. Instead of using its own data for training, the zero-shot approach applies an already trained model (pretrained and fine-tuned) directly to the specific use case.
Through pretraining, the model has learned the fundamental structure of language. In addition, it was fine-tuned for the specific task of Natural Language Inference (NLI). Models fine-tuned in this way recognize whether two pieces of content convey a similar semantic message — or are logically related to each other.
Such a model can determine, for any pair of sentences, whether they:
- have the same meaning (Entailment),
- are neutral toward each other (Neutral), or
- are contradictory (Contradiction).
The classification task is converted into a comparison task in Zero-Shot Learning. For each assignable value from the class, a comparison is made. To do this, the respective label is embedded in a sentence and then compared against the text to be classified. This example sentence is called a hypothesis. There are further approaches for creating hypotheses — for example, a label definition, a paraphrase, or a prototype can also be used as a counterpart.
The pair with the highest similarity is output as the correct label. In this way, models fine-tuned on general texts can be used without the need for further fine-tuning, since the learned task remains the same.
If the results from Zero-Shot Learning are not yet accurate enough, Few-Shot Learning can build on it very effectively.
Few-Shot Learning
Few-Shot Learning is based on the same principle as the zero-shot approach. In addition, a small amount of data is used to train the model on the specific data.
As in the zero-shot approach, the text to be classified is compared against a sentence. All pairs are taken into training with labels. The training dataset is further expanded by also using the negative examples for training. With 3 labels to be assigned, there is one matching label and 2 negative examples. This means there are a total of 3 training examples per text to be classified — making the training dataset three times larger. This artificial augmentation of the dataset promises better results.
We will reveal the results we were able to achieve with these approaches in classifying Technical Documentation in the final part of our blog series.
To briefly summarize the key points: For a computer to process language, it is transformed into a structured numerical form that represents the meaning of a word in a context-dependent way (embeddings). There are extensive language models that have learned this fundamental language comprehension from vast amounts of data. Through fine-tuning, models learn to solve specific tasks. Zero-Shot and Few-Shot Learning are two modern approaches that, by transferring already learned knowledge and reframing the task, manage with little or no training data.
Would you like to learn more about how you can classify Technical Documentation with plusmeta? We would be happy to arrange a demo.
In our blog posts, we try to explain complex technologies in a simple and understandable way — which sometimes calls for significant simplification. If you want to explore the subject matter in a more technical and in-depth way, we recommend the publications of our AI experts Alina Cartus and Maximilian Both.
Sources:
- Jurafsky, D. and Martin, J. H.: Speech and Language Processing. An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall series in artificial intelligence. Upper Saddle River, NJ: Prentice Hall Pearson Education International, 2009
- Xian, Y., Lampert, C., Schiele, B., Akata, Z. [2020]: Zero-Shot Learning — A Comprehensive Evaluation of the Good, the Bad and the Ugly
- Peng, B., Zhu, C., Li, C., Li, X., Li, J., Zeng, M. and Gao, J. (2020): Few-shot Natural Language Generation for Task-Oriented Dialog
- Barthel, S. (2020): Grundsätze moderner Textklassifizierung für Machine Learning: Word Embeddings. Einstieg in das Machine Learning: Teil 1 https://entwickler.de/python/grundsatze-moderner-textklassifizierung-fur-machine-learning-word-embeddings
- Natural Language Processing, blog post by Maximilian Both https://www.tecislava.com/categories/nlp
- Cartus, Alina (2021): Die Analyse der Einbindung von Metadaten in ein Language Model zum verbesserten Sprachverständnis von Task Oriented Dialogue Systems für TGA-Komponenten. Bachelor thesis
Your question about NLP wasn't answered? Feel free to contact us.
