What Are the Results of Optimizing Prompts and Role Allocation? Blog Creation Case Study Using Generative AI

Hello! This is K, a consultant. I usually handle projects for creating and improving manuals for companies in the manufacturing and pharmaceutical industries.

In the previous blog, “The Journey of Blog Creation Using Generative AI: The Beginning of Considering Its Use in Business,” I introduced the challenge of trying to utilize generative AI for creating our company blog. In this sequel, I would like to talk about the obstacles we encountered while advancing the consideration of generative AI utilization and how we overcame them.

1. Summary of Last Time: The Challenges We Faced When Trying to Write a Blog Using Generative AI

To consider the use of generative AI in manual creation tasks, we first decided to try using generative AI in our own company’s blog writing tasks. We researched prompts and learned the basics of how to use generative AI, and developed a "writing assistance tool" that organizes prompts into a menu. While we found generative AI to be convenient through these efforts, we also encountered the following issues.

・During the writing process, keywords necessary for SEO measures considered by the planner may be omitted

・The quality of the output is inconsistent, and in some cases, humans have to make significant adjustments
(for example, it was expected to start with raising an issue and end with presenting a solution, but instead, the discussion continued on the problems until the end)

This time, I would like to write about how we addressed such challenges and how we proceeded with our subsequent efforts.

2. Measure ① Improve the prompts

As introduced in the previous blog, let me once again explain our company's blog writing workflow.

1. Outline Confirmation and Adjustment:
In our company, the planner who creates the blog outline (blog title, summary, reference information) and the writer who drafts the main text are different people. In this process, the writer reviews the outline created by the planner and roughly considers what kind of manuscript to create and how to structure the flow of the story. In some cases, the outline itself may be adjusted.

2. Writing:
This is the process of drafting the blog article based on the outline. While incorporating the writer’s experiences and impressions, the content is expanded with the aim of creating an engaging blog that can be easily read during spare moments. Since the writing style is completely different from manuals, even experts in writing require considerable time.

3. Review:
This is the process of sharing the draft with internal stakeholders and having them review it. Reviewers check from perspectives such as whether there are any incorrect contents in the draft, whether the flow feels off, and whether it is written in an easy-to-read manner. Since the reviewers are also members involved daily in the manual production project, there are especially many comments regarding clarity and readability.

4. Adjustment:
This is the process of reflecting the review results in the draft and making adjustments. If necessary, a second review may also be conducted.

5. Submission:
The manuscript is submitted to the internal web page administrator.

The first thing we tackled was improving the prompts used in the "Writing" process. In the previous blog, I wrote about creating a writing assistance tool that organized prompts into a menu, but perhaps because the prompts set in the tool were abstract phrases like "write" and "expand the text," there weren't many outputs that we found satisfactory. Sometimes the targeted SEO keywords were missing, or the text came out stiff like a textbook despite being a blog, or upon closer reading, it was nonsensical... In any case, simply letting the generative AI write freely was insufficient.

Therefore, we proceeded with researching prompts and learned how to write prompts when having the AI generate text, as well as which key points to focus on. Then, we rewrote the prompts implemented in the writing assistance tool. Also, for the "Outline Confirmation to Adjustment" phase, instead of using the writing assistance tool, we used ChatGPT alone, and prepared a prompt template for use there. The reason for creating the template was to prevent variations in prompt skill levels among different people.

Improving and templating these prompts turned out to be surprisingly difficult, leading to a continuous process of trial and error. Just a slight change in the instructions to the generative AI could drastically alter the output, or even if the content was good, the format might not follow the instructions... Carefully considering the prompts also means clarifying "what and how much to have the generative AI do," so while thinking about the prompts, I struggled with questions like, "How far should I let the generative AI handle this?" and "Wouldn't it be faster and more reliable for a human to adjust this rather than including it in the prompt?".

Comparing the prompts and output results repeatedly and adjusting the prompts was a painstaking task, but through this trial and error, I realized that it is important to clearly decide what and how much to have the generative AI do, and to instruct the AI with clear words—that is, prompts.

In manual production, the process of clearly defining the target quality of the manual and creating the table of contents structure is called "design" and is positioned as a very important process. In the business use of generative AI, creating prompts is precisely the "design" process, and this is a crucial step that determines the quality of the output.
However, even when using fixed-form prompts, it was not possible to handle all cases successfully. Ultimately, it was necessary to adjust the prompts according to the situation each time and to have additional interactions with the generative AI.

3. Measure ② Clarify "How far humans should make adjustments"

Although I tried writing a blog by devising prompts, having a good prompt did not always guarantee a perfect blog.

While interacting with generative AI by changing prompts, I realized that it is important not to expect a perfect score output from the AI. If you expect the generative AI to write flawless text, you tend to conclude, "It's not as good as I thought... generative AI really can't be used for work." Instead, I thought we should consider a division of roles where "this is as far as we leave it to the generative AI, and from there humans will make adjustments," and formalize that into a workflow.

However, if the part that humans adjust is left to the discretion of the person in charge, the extent to which the AI-generated output is edited will vary from person to person, resulting in inconsistencies in the quality of the final deliverables. Therefore, we created a checklist for the process after the AI generates the blog draft to check and revise it.

The checklist included items to improve the readability and clarity of the blog. For example, the following:

・Is the ratio of kanji to hiragana approximately 3:7?

・Are there any phrases that are difficult to understand?

・Whether the main point is conveyed concisely

The text generated by AI may look clean at first glance, but upon closer reading, there can be parts that don't make sense or are verbose and hard to understand. At least at that time, those parts could be improved more quickly and reliably by humans using a checklist like the one above.

In this way, by improving the prompts, clearly defining the roles between generative AI and humans, and ensuring consistency regardless of the person, we were able to improve the efficiency and standardize the quality of blog writing compared to the previous workflow.

Also, at the regular meetings where we discussed the use of generative AI, blog writing team members shared the characteristics of the text output by the AI and discussed the parts where human intervention was particularly beneficial, gaining significant insights.

4. Presented at the TC Symposium!

We presented the efforts to utilize generative AI in blog creation introduced so far at the "JTCA Symposium" hosted by the Technical Communicators Association in 2023. We presented both online and offline, and in both settings, we were able to share information with many people interested in the use of generative AI, making it a very meaningful opportunity.

At that time, what particularly interested the attendees was the demonstration of the "writing assistance tool" introduced in the previous blog. Seeing the tool actually in operation made many feel that "it could also be applied to our work." During the Q&A session, some people even asked, "Do you plan to sell the writing assistance tool?"

We received many positive comments in the post-session survey as well. Comments such as "I felt the potential to reduce writing workload," "I was able to organize the strengths and weaknesses of generative AI and humans," "I want to try using (the writing assistance tool) as support for manual creation," and "I also want to know about applications in the translation field" conveyed that everyone’s interest is growing.

On the other hand, we also received feedback from people who had concerns about using generative AI in their work. In particular, many commented on how to deal with generative AI producing false information, such as "fact-checking seems difficult" and "how to verify the basis of the generated information." These survey results also revealed that generative AI is no longer a matter of the future, but has reached the stage where the question is "how to use it" at the field level.

5. Summary: Let's Design Workflows Where Generative AI and Humans Collaborate

Not only for manuals and blogs, but the profession of writers is sometimes said to disappear with the advent of generative AI. However, what we should consider here is not how to confront and defeat new technology, but how writers (humans) can take the lead while leveraging new technology to organize workflows.

Through this attempt, I have come to hold this idea even more strongly. In order to continue writing work in the world, it is important to decide "how much to entrust to generative AI," carefully consider the division of roles between generative AI and humans, and systematize the workflow including "how much humans intervene."
The key point is this "systematization of the workflow." Even if you focus solely on the scope entrusted to generative AI, in the end, there will be people who adjust the AI's output a lot and those who do not, causing the final product's quality to depend on individuals, meaning the work tends to become personalized. Therefore, workflow systematization is necessary to avoid this.

This is not limited to generative AI, but when considering the use of new technologies in business operations, it is easy to fall into the trap where introducing the technology itself becomes the goal, and the original objectives of improving efficiency and eliminating dependency on specific individuals are not achieved. In this initiative, to avoid such situations, we have kept in mind the idea that "generative AI is merely a means," and have been examining improvements aimed at efficiency and preventing dependency on individuals.

In that sense, the workflow we reviewed this time still had plenty of room for improvement. Since the work was being carried out by multiple people, there were ambiguous points in the division of tasks, and there were still aspects of the prompts that could be revised. In the next blog post, I would like to write about further reviewing the workflow and bringing it to a point of provisional completion. For those who follow along until the end, please be sure to check out the final installment titled "Utilizing Multiple LLMs! Workflow Improvement for Blog Creation Using Generative AI."

Now, around this time, new LLMs other than GPT, such as Claude and Gemini, have become a hot topic. In fact, alongside this initiative, we have also been conducting an experiment where a current manual production director compares texts obtained by giving the same instructions to multiple types of LLMs. I plan to write about that in another blog, so please look forward to it.