What is the difference between AI training data and learning data? A clear explanation!

2. What is teacher data?

There are three learning methods in machine learning: "supervised learning," "unsupervised learning," and "reinforcement learning." All learning methods require a vast amount of data, but the data prepared for each method differs. In "supervised learning," data known as training data is required.

In supervised learning, for AI to recognize specific objects from data through learning, it is necessary to indicate the objects within the data. For example, to recognize Mount Fuji from an image, it means marking the data with "This image is Mount Fuji." The process of marking data in this way is called annotation. By providing the AI with these annotated images, it learns for the first time that "This image is Mount Fuji." The annotated data is referred to as training data.

3. Difference Between Teacher Data and Learning Data

In machine learning, data referred to as training data is used in addition to labeled data. While the two may seem similar at first glance, there are differences.

3-1. What is training data?

Training data is a dataset used for AI to learn recognition targets. In "supervised learning," the training data includes labeled data. In "unsupervised learning" and "reinforcement learning," it does not include labeled data. In other words, any data used for learning in machine learning is referred to as training data.

3-2. Differences Between Learning Data and Training Data

As explained earlier, the training data is the data that has been annotated. It is essential in "supervised learning," but is not used in other methods. This is the difference between training data and labeled data.

4. Flow of AI Development

Here, we will focus on the part of the AI development process that involves learning from data.

: Data Collection
We collect data for learning purposes. We gather a large amount of data that includes the subjects we want to recognize. By preparing data under as many different conditions as possible, we prevent the occurrence of "overfitting," where recognition accuracy improves only under specific conditions. Various conditions, in the case of Mount Fuji mentioned earlier, refer to images with various variations such as different seasons, different weather, different angles, and different sizes.

: Annotation
Once the data collected is selected for creating the training data, a specification document will be created, and annotation work will be carried out based on it. Since it is necessary to create a large amount of data, we will assign the necessary number of annotators to manage quality and productivity, ensuring that there are no delays in the development schedule or impacts on AI learning due to quality issues.

: Learning
The completed training data is used to train the AI. The AI that has learned the correct patterns is called an inference model or a trained model.

: Model Evaluation
We evaluate whether the results output by the inference model, given learning data other than the training data, reach the desired accuracy.

: Implementation
Once it is confirmed that the AI can achieve sufficient accuracy, it will be implemented and operated on various devices and software. We will monitor to ensure that recognition accuracy does not decline, and perform additional training and maintenance as necessary.

5. Points to Consider When Collecting Training Data

When collecting data, it is necessary to be mindful of copyright and personal information. According to the document "About the Relationship Between AI and Copyright" published by the Ministry of Internal Affairs and Communications, under Article 30-4 of the current Copyright Act, it is stated that "in the case of information analysis such as AI development, acts of use that do not aim to enjoy the thoughts or feelings expressed in the work can generally be used without the permission of the copyright holder."

Citation:Regarding the relationship between AI and copyright

However, with the recent advancements in generative AI, there is a possibility that the output may closely resemble the original copyrighted work, which could unjustly infringe on the rights of the author. Even when using commercially available datasets, it is necessary to comply with the terms of use. Additionally, data that includes personal information, such as facial photographs obtained from street cameras, may infringe on privacy. Even if data is collected with the consent or permission of the individual, there have been cases where objections to its use arise later, leading to the discontinuation of development. Therefore, it is essential to exercise utmost caution in the collection and use of data.