What Is Deep Learning?
In our research project DEEEP, funded by the Ministry of Economic Affairs of Baden-Württemberg, we are extending the plusmeta platform with a Deep Learning component. In this blog post, we take a deep dive and explain what Deep Learning is and how it works. To understand what Deep Learning is, let us first take a step back and look at the bigger picture: What is Artificial Intelligence and how does it relate to Deep Learning?
Table of Contents
What Is Artificial Intelligence? – An Overview
Artificial Intelligence (AI) is a subfield of computer science that attempts to imitate human thinking and behavior. Taking a pragmatic view, one could also say that computer programs are intelligent when they produce a result that appears intelligent.
From this perspective, AI can be divided into two subfields: Machine Learning and Rule-Based Methods.
Rule-Based Methods – Complexly Linked Rule Sets
Rule-based methods are complex algorithms. They solve tasks based on predefined solution paths following patterns such as "if x occurs, do y." These rules can be combined and nested so intricately that the end result appears intelligent. Rule-based methods include "label matching," "evaluation of knowledge graphs," and "pattern matching."
In plusmeta, many metadata assignments are made using rule-based methods, as these only require lists of values to be assigned (label matching), a knowledge graph with dependencies and relationships, or a pattern for identifying specific formats (pattern matching).
Machine Learning – When Computers Learn to Learn
In Machine Learning, the computer learns independently. The solution path is not explicitly pre-programmed. Machine Learning applications learn from "experience" — or rather, from historical, statistical data. The foundation is a model that is trained for a specific task. Further below, in the section on "training," we will take a closer look at these models and how AI learns.
Machine Learning, in turn, has its own subfields — one of which is Deep Learning. Other Machine Learning methods include self-learning decision trees, where AI is guided through the tree, and linear regression, where the AI extrapolates past results to make predictions.
Another Machine Learning method is based on a vector space model. This approach underlies the Machine Learning currently used in plusmeta. Here, many classified example contents are used to form one representative vector per value. New, unclassified content is assigned the value whose vector is closest to its own.
Deep Learning – Brain-Inspired Models
And here we finally arrive at Deep Learning. Deep Learning is a special form of Machine Learning. What makes it distinctive is the architecture of its models — they are modeled on the human brain. Artificial Neural Networks are built from many artificial neurons that are interconnected, forming layers. At each node, a simple decision is made. Through the chaining of these decisions, a complex system emerges.
In the first layer, all input information is processed — for example, all pixels of an image. In the subsequent layers, increasingly abstract tasks are performed, such as recognizing lines and shapes. Later layers handle more concrete and complex tasks. These layers are also called "hidden layers," because what exactly happens in each layer is a black box. The AI itself finds the relevant features and decisions to be made. Which features are considered in which neuron or layer remains the model's secret.
At the very end is an output layer. It answers the original question — for example, "Does the image show a squirrel?" Simplified, one can imagine the decisions at all neurons as binary yes/no answers. In reality, however, probabilities are output — for example, there is an 87% probability that the image shows a squirrel.
The model then checks its own prediction against classified training data. If it is correct — great! If not, the information is fed back into the model in the form of a weighting. The algorithm thus learns from its own mistakes and corrects itself with every pass.
But how does such a model come into being? It has to be trained!
Training – Nothing Comes from Nothing
Different paradigms come into play for training. The training strategy is selected depending on the task and the desired model architecture. The three most important approaches are supervised learning, unsupervised learning, and reinforcement learning.
Supervised Learning: In supervised training, the AI is shown manually labeled examples. From these examples and the classifications assigned to them, the AI derives the decisive features for each class. In the case of "classic" text classification, these can be, for example, typical word groups. This type of training is already used in the plusmeta platform today.
Reinforcement Learning: In reinforcement learning, the AI is given an objective and receives a reward when it achieves good results. The AI tries to optimize itself to receive as many rewards as possible — for example, in the form of points.
Unsupervised Learning: In unsupervised training, no classified training data is required, because the task itself is different. The AI is supposed to group the data. We specify how many groups there should be, and the AI decides which features are suitable for grouping the data into those groups. Afterwards, one can explore the criteria by which the AI likely sorted.
An important takeaway on the topic of "training": AI can only do what it has been trained to do. In the case of supervised learning, for example, this means the AI cannot recognize rabbits in images if it has never seen example images of rabbits.
Layer by Layer Toward the Goal
We speak of Deep Learning when there are multiple hidden layers. For such hidden layers to exist, very large amounts of training data are required. Common language models for the German language, for example, were trained on all German-language Wikipedia articles.
Most projects do not have access to that much training data. Therefore, in Deep Learning, one typically works with pre-trained models and simply teaches them the knowledge needed for the new task.
Task-specific training creates new layers in the model. For example, this allows the model to recognize typical word groups for the product lifecycle phase "operation," or to identify the features by which a circuit diagram is recognized in an image.
The many layers enable complex tasks such as image recognition and, in the case of language understanding, the consideration of not just individual words but also the context in which they appear.
All a Matter of Structure - Model Architecture
In addition to the training strategy, the model architecture is also crucial in determining what types of tasks an AI can fundamentally solve. The example described above refers to a Convolutional Neural Network (CNN). Such models can be used for classification tasks. They can suggest metadata assignments, recognize images, or understand language. For translating texts, sequence models are used instead. Texts can be generated by Generative Adversarial Networks.
Even from the different models alone, it becomes clear how broad the range of tasks is for which AI can be used. You just need the right model and thorough training.
Conventional Machine Learning vs. Deep Learning
Feature extraction is the process of creating a more easily processable representation of the data.
In conventional Machine Learning, feature extraction is influenced by humans or measurement parameters. For text classification tasks, for example, the text is broken down into fragments. How many words such a fragment — also called an n-gram — contains can be configured. "n" determines the number of words per fragment. Furthermore, the weighting of the n-grams can be adjusted so that particularly meaningful words, for instance, are weighted more heavily. When it comes to weighting, it makes sense to adapt it to the type of text being processed. In our case, the weighting is tailored to the specifics of Technical Documentation (weighting developed in the dissertation of Dr. Jan Oevermann).
In Deep Learning, by contrast, feature extraction is part of the algorithm and is handled by the AI itself. However, this also means that we have no influence over what the AI extracts from the training data as relevant features and learns.
And what conclusion can we draw from this examination?
Deep Learning is a vast field with many different application possibilities. The task at hand influences the model type and the training approach. Sufficient training data and substantial computing power are required.
A closing note
The models and methods are in some cases presented in a highly simplified manner. In this post, our primary aim is to explain the principles in an understandable way for our users, who come mainly from the field of Technical Writing.
Sources:
- Aporras (2019): What is the difference between Deep Learning and Machine Learning? (https://quantdare.com/what-is-the-difference-between-deep-learning-and-machine-learning/)
- Heller, Martin (2022): Was ist Deep Learning? (https://www.computerwoche.de/a/was-ist-deep-learning,3549921)
- Manning, Christopher (2020): Artificial Intelligence Definitions. https://hai.stanford.edu/sites/default/files/2022-06/HAI_Terms_6_22.pdf
- Norvig / Russell (2021): Artificial Intelligence: A Modern Approach, Global Edition. 4th edition
- Oevermann, Jan (2019): Dissertation: Optimierung des semantischen Informationszugriffs auf Technische Dokumentation.
- Dirik, Iskender (2018): Was ist den bitte Deep Learning? Whitepaper (https://news.microsoft.com/de-de/deep-learning-whitepaper/)
- Hötter, Johannes; Warmuth, Christian (2020): OpenHPI course "Künstliche Intelligenz und maschinelles Lernen für Einsteiger" (https://open.hpi.de/courses/kieinstieg2020)
