What is the difference between AI training data and learning data? Explained clearly!

2. What is Teacher Data?

There are three learning methods in machine learning: "supervised learning", "unsupervised learning", and "reinforcement learning". All of these methods require a large amount of data, but the type of data needed differs for each method. In "supervised learning", data known as "training data" is required.

In "supervised learning," AI needs to recognize specific objects from data through learning. For example, in order for AI to recognize Mount Fuji from an image of Mount Fuji, it is necessary to mark the object in the data as "this is Mount Fuji." This process of marking data is called data annotation. By providing annotated images to AI, it can finally learn that "this image is Mount Fuji." The annotated data is referred to as training data.

3. Differences between Teacher Data and Learning Data

In machine learning, there is something called training data, which is different from teacher data. Although they may seem similar at first glance, there are differences between the two.

3-1. What is Learning Data?

Learning data is a dataset used for AI to learn the target of recognition. In "supervised learning", the learning data includes teacher data. In "unsupervised learning" and "reinforcement learning", there is no teacher data included. In other words, all data used for learning in machine learning is called learning data.

3-2. Differences between Training Data and Teacher Data

As mentioned earlier, teacher data is annotated data. It is essential for "supervised learning", but not used in other methods. This is the difference between training data and teacher data.

4. Flow of AI Development

Here, we will focus on the part of learning data within the flow of AI development.

: Data Collection
We collect data for learning purposes. We gather a large amount of data that includes the target we want to recognize. By preparing data with various conditions as much as possible, we prevent the occurrence of "overfitting", where the recognition accuracy only increases for specific conditions. These various conditions include different seasons, different weather, different angles, and different sizes of images, as in the example of Mt. Fuji mentioned earlier.

: Data Annotation
Once the data to be used for creating training data is selected, a specification document is created and annotation work is carried out based on it. In order to create a large amount of data, it is necessary to assign the required number of annotators and manage the quality and productivity to avoid delays in the development schedule and any impact on AI learning due to quality.

: Learning
The completed teacher data is trained on AI. AI that has learned the correct patterns is called an inference model or a trained model.

: Model Evaluation
We evaluate whether the results output by the inference model reach the desired accuracy by giving the model learning data other than the teacher data.

: Implementation
Once it is confirmed that AI can achieve sufficient accuracy, it will be implemented in various devices and software for operation. We will monitor to ensure that recognition accuracy does not decrease, and perform additional training and maintenance as needed.

5. Points to Note When Collecting Learning Data

When collecting data, it is necessary to pay attention to the copyright and personal information. According to the document "AI and Copyright Relations" issued by the Ministry of Internal Affairs and Communications, under Article 30-4 of the current Copyright Act, "In information analysis such as AI development, the use of copyrighted works for the purpose of enjoying thoughts or emotions is generally possible without the permission of the copyright holder."

Quote:<font id="4">About the Relationship between AI and Copyright

However, with recent AI generation, there is a possibility that the output may closely resemble the original work, which could potentially infringe on the rights of the author. Even when using commercially available datasets, it is important to comply with the terms of use. Additionally, data that includes personal information, such as facial photos obtained from street cameras, may violate privacy. Even if the data was collected with the consent or permission of the individual, there have been cases where objections were raised later and development was discontinued. Therefore, it is important to exercise caution when collecting and using data.