After taking on the role of ‘DevRel’, I developed a small habit. It is a habit of giving reasons for even the smallest details. ‘Just’ was difficult to understand. There had to be a reason and basis. The in-house hackathon ‘9oormthon RAID WINTER’, which concluded in 2023, was no exception. The theme of “Increasing work efficiency with generative AI” did not only consider organizational direction. I had a strong hope that this would be the beginning of a ‘change’ in which we would improve the inefficiencies in various parts of our country that had been neglected, failed to take action, and had been put off, and that would lead to interest in ‘inefficiency’ and continuous improvement.
Now, three months later, I suddenly became curious about the story behind it. I was curious whether, as intended, we would have become more interested in the inefficiencies around us, what changes would have occurred to each member, and what help it would have had in our growth. So, we asked three members of ‘Russian Roulette’, one of the hackathon participating teams, to contribute. We asked you to talk about the results of the hackathon and the changes that followed.
by snow, jade, jhin, philip
Editor's note_ Contribution re-edited in interview format I would like to inform you of this.
Russian Roulette is created by GEN AI SQD’s AI engineer Jin, EDU full-stack engineer Jade, and EXP full-stack engineer Philip. It is a team formed. Jin was in charge of planning and prompt engineering, and Jade and Philip were in charge of back-end development. Among the 7 participating teams, we were the only team to raise a customer service issue, which was especially memorable.
goorm’s representative services include five types. CS channels also vary, including wired, email, and channel talk. The range of customers is wide, from corporate customers to general customers. It is not easy for a CXCustomer eXperience manager to take charge of all of this. Therefore, each organization appoints a CS person among its members on a daily or weekly basis. Instead of a CX manager, the product developer is in charge of CS response. There is a realistic reason that it is difficult for a small number of CX managers to take charge of all CS, but rather, it was to develop into a ‘product-centered’ service in which developers directly listen to customers’ inconveniences and demands and reflect these opinions in product development.
Contrary to intention, there were no problems. Since there was no dedicated person in charge, I had to find out who was on duty for CS today. Because the product and business are complexly connected, it is often difficult to know which team CS should be delivered to. Each SQD had different CS operation systems and methods, making it difficult to know whether the inquiry was processed, and how to check past history was also different. The ‘Russian Roulette’ team set out to solve the problem that we knew was a problem but had been trying to ignore.
Russian Roulette Team of 3
Gen AI SQDsquad 1 I work as an AI Engineer. As a planner, I participated in this hackathon and was in charge of overall planning of the project and creating prompts to use ChatGPT. So far, I have mainly collaborated with DEVEL SQD people. I participated in the hackathon because I wanted to take this opportunity to work with other SQD colleagues. AI services are usually carried out over a long period of time, but I also wanted to experience creating results within a limited time like a hackathon.
I am working as a full-stack engineer at EXP SQD, which develops ‘goormEXP, which manages work like a game.’ I am very interested in sharing what I know with everyone and improving efficiency. It hasn’t even been a year since I joined goormee, and I participated in the hackathon because I wanted to work with and get to know other goormee.
I am working as a full-stack developer at EDU SQD. EDU SQD has a lot of CS inflow due to the nature of its business. I always thought it would be nice to have a bot to unify CS, and it was a golden opportunity to implement it, so I participated in the hackathon. I prepared for the hackathon by talking about CS with SQD planners and PMs and researching what difficulties there were.
Q. The name is creepy. Why Russian Roulette?
Jhin_ Russian roulette is a game in which one bullet is put into a revolver with six cartridges, and participants take turns putting the muzzle of the gun to their own heads and pulling the trigger. It is said that soldiers and criminals did it to show their courage. For this scary game to end, someone has to get hit by a bullet. In other words, the game only ends when you die.
We named the team Russian Roulette because we thought that something like Cloud’s ‘If CS continues like this, a revolver bullet will fire one day’ might happen. Our goal was to improve the CS processing and management process, which everyone was indifferent to and had delayed solving because they were different products and services, all within one cloud.
Q. Who first came up with the CS idea?
Jade_ In fact, our EDU SQD receives a lot of CS from outside. Even before participating in the hackathon, CS was one of my worries. I used to talk about CS issues with the planner and PM of the same SQD. The reason I decided to participate in the hackathon was because I wanted to improve CS. I came up with the idea, but we all thought about and developed the idea together. Until around 10 p.m., when other teams were finishing planning and implementing it, we were still planning and thinking through the plan.
Q. What difficulties did you have in responding to CS?
Jade_ It shouldn’t have reflected only one side’s position. The priority was to listen to everyone’s opinions. So we conducted a survey. We listened to the opinions of not only CS managers but also members who have experienced CS about what difficulties there are in CS processing and how they would like to be improved.
The biggest problem was that the CX manager in charge of CS had a lot of difficulty writing the VoCVoice of Customer report. It took a considerable amount of time to collect information such as where CS records came from and how they were processed. On the other hand, developers handling CS had great difficulty finding and responding to previous history in Slack. In the end, the key was to build up the history of CS inflow path, inquiry content, response method, and processing status in the DB.
Jhin_ To summarize. The CS process was different for each management entity, such as SQD and team. Everyone had different CS processing management and recording methods.
So, developers do what they do,
- It is difficult to find CS history.
- I don’t know who the other squad CS person is.
- It is urgent, but it is burdensome to request CS processing outside of business hours.
- I had some urgent work to do, so I put it off for a while and then forgot.
- I don’t know which squad to send it to.
CX managers follow CX managers,
- If it is not a product/business in charge, it is difficult to find a person in charge.
- I feel the need for history management and sharing is increasing.
- Issuance of VoC reports is often delayed because it takes a lot of time to process CS data.
I complained about these difficulties.
Q. How have you changed your CS process?
Jade_ We improved the CS process in three major aspects. CS distribution, processing, and history.
Regarding CS distribution, when CS comes in from an external channel, the CX manager mentions Slackbot in the CS channel and leaves an inquiry. ❶Slackbot automatically classifies keywords, finds and assigns the SQD in charge. Then, a mention will be sent to the CS person in charge of the relevant SQD. ❷The CS manager presses the [Start CS response] button on Slackbot and responds to CS. ❸If you want to view previous history during the response process, you can click [Analyze History] to see similar CS response methods. After completing the CS response, click [Complete Response] to complete the CS processing.
Of course, depending on the type of CS, the CX manager may handle it directly. We also added the ability to save as a CSV file. This was to relieve the difficulty of writing VoC reports issued by CX managers on a weekly or sprint basis.
Q. Were there any difficulties?
Philip_ It was my first time developing Slackbot, so it was a little difficult. It took some time to get used to the Slack Bolt SDK. This was disappointing. If I had done it a little faster and better, I would have been able to produce better results. When developing Slackbot, we especially thought a lot about user flow. It took a lot of time for CS Slackbot to decide how to distribute CS and assign it to the person in charge.
Jhin_ My original major(?) was in the field of natural language processing. Cloud also deals with ‘vision’, but natural language processing can be said to be its main business(?). This is my first time using the ChatGPT API. He’s smart, but he doesn’t give me the answers I want. It gave only an approximation, similar to the answer I wanted, but not the exact answer I wanted. To run a service, you need accurate answers. It was difficult to ensure consistency of answers.
Q. I also tried several LLMs and found that part difficult. How did you solve it?
Jhin_ As anyone who has used ChatGPT knows, it does not give the same results for the same request. Sometimes this is an advantage, but it becomes a problem in cases where a set answer framework is needed, like Slackbot. To ensure consistency of results, we designed the following two methods.
Select result format
First, I used the few shot learning method, which predetermines input and output examples. Thanks to this, when a certain input value is entered, the output value is displayed in a certain format.
{
role: 'user',
content: `Here's the input you need to classify. Input: ${targetMessage}category: `,
},
{
role: 'user',
content: `Here's the input you need to summarize. Question: ${question} \n Answer: ${sampleAnswer} \n Reference data: ${additionalData} \nSummary: `
}
Lower temperature value
I lowered the temperature value to increase the consistency of answers. The temperature parameter has a lot to do with how creative you want to be. A high value leads to more creative and diverse answers, while a low value leads to consistent and predictable answers.
I also took a few more things into consideration when writing the prompt.
Keyword
goorm operates multiple services, so its CS is diverse. There are cases where CS inquiries that seem unrelated need to be handled by the same person in charge, and conversely, there are times when similar CS inquiries need to be handled by different people in charge. Therefore, it was difficult to obtain results of sufficient quality with a simple few-shot learning method. (Of course, quality can be improved by building more data for few-shot learning, but it was difficult to do it in the limited time of a hackathon.)
So, I extracted the main keywords for each service in advance and added a thought process to the prompt that compared the keywords for each service with the keywords extracted from CS inquiries.
{
"role": "system",
"content": `
Task: Your task is to classify the given input into one of six categories. I'll provide descriptions and keywords mostly related to each category. Focus on whether the given input contains relevant keywords.
category: EXP
Description: A platform for evaluating and managing students or employees, or enhancing members' capabilities through gamification. Most inputs containing the keyword 'exp' are related to this category.
Keywords: store, exp, gifticon, learning plan, cloud fragment, campaign, quest, finishing, poker, organization chart, stand-up, attendance check, compliment, AI Gurumi, feedback, learning diary
category: IDE
Description: A platform focusing on operating cloud IDE services, with emphasis on container service operations rather than detailed code.
Keywords: container, IDE, ide, gpu, container, domain, refund, discount, always, keep on, wake up, loading, team plus
category: EDU
Description: A platform for student lectures and courses.
Keywords: edu, grading criteria, remark, grading, soldier, class, lecture, channel, course, certificate of completion, school, student, lecturer, teacher, invitation
category: DEVEL
Description: A platform where students, job seekers, and internal skill checkers can conduct tests and exams and practice programming.
Keywords: devel, depth, devth, level, devel, coding test, test, level, observer, recruitment, problem, exam, quiz, codepro, codepro, candidate, supervisor, exam room, obserview, coding, programming
category: SRE
Description: Maintenance of internal development and service servers.
Keywords: git, sre, log, proxy, image deletion, apm, deployment, pipeline, Nginx, aws, gcp, domain deletion, region, memory, rollback, block, ssh
category: NONE
Description: Cases not belonging to the above five categories.
Based on the descriptions and keywords of each category, classify the input carefully.
You should only answer with one of the six categories. Just provide the category name without explanation.
`
}
Fast, faster
One of the important things we considered in Slackbot development was ‘speed’. If it takes a long time to hear a response from Slackbot, you might think, “Wouldn’t it be better to use something else?” So I used GPT 3.5 instead of GPT 4. Performance is relatively low, but it is well optimized and has a fast response.
Embedding
ChatGPT does not do all the processing, such as extracting keywords. Tokens are too expensive to process all past CS history and current CS inquiries by sending them to ChatGPT. If CS continues to increase, there is a possibility that it will eventually exceed the maximum number of input tokens allowed by ChatGPT.
So, we designed it to store the embedding value of the CS in the DB every time the CS is processed. When searching for similar CS history, we used a method of searching based on the similarity of embeddings. I was able to get a faster and more accurate answer than when I used ChatGPT.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("jhgan/ko-sroberta-multitask")
def return_embedding(sentence):
return model.encode(sentence)
@app.post("/return_embedding/")
async def return_embedding_api(sentence: str):
processed_sentence = return_embedding(sentence)
return {"processed_sentence": processed_sentence.tolist()}Editor's note_ The embedding model used is SentenceTransformer's jhgan/ko-sroberta-multitask.
Q. Didn’t you do fine tuning?
Jhin_ There are many ways to use ChatGPT. There are methods such as fine-tuning ChatGPT’s model with data and using the improved model, or zero-shot and few-shot learning methods using only prompts. Normally, I would have thought about and tested various methods to find the most effective one. But it didn’t happen this time.
Higher performance can be achieved with fine tuning, but we decided it was not suitable for a hackathon where time was limited. This is because repeated testing is necessary for fine tuning.
Q. Are you still using CS Slackbot?
Jade_ That’s what I regret the most. CS Slackbot is the result of an idea we created during a short hackathon. It was not created with the consent or reflection of all opinions from CS-related people. In order to use this, it takes effort and time to persuade the relevant people and test it. Also, testing it on the CS channel may be confusing as there are a lot of CS inquiries and people involved, so I haven’t been able to apply it yet. Within this year, I really want to apply CS Slackbot at least to EDU SQD, even if not to the entire company.
Q. I’m curious about the story after participating in the hackathon. What changes have occurred?
Jhin_ I think I started thinking more broadly and bigger. Would you say your thoughts have broadened? After the hackathon, many different ideas came to mind. I think it was helpful to use it and experience it myself. For example, when processing some data with ChatGPT, I had never thought about embedding by first storing the data and quickly retrieving it, but it made me think about this processing method again.
Philip_ I am an introvert who is very shy, but I think I have become a little more comfortable approaching people after the hackathon. It was my first time working with Jhin, and I felt that AI engineers were different. When I wrote the prompt, I didn’t quite get the answer I wanted, but after seeing Jhin’s writing, I think LLMLarge Language Model is as good as it gets. Although it has nothing to do with cloud after the hackathon, I have used ChatGPT for a personal project. This is a service that selects interview questions based on your self-introduction, and Jin’s prompts were very helpful.
Jade_ The best thing was collaborating with people from other organizations that I normally don’t meet and talk to. I had no experience making Slackbots and had never seen related code, so I was a little lost, but Philip was good at it and helped me a lot. I also liked that I learned something new by using something new, like Slack Bolt, that I haven’t used before. More than anything, I liked that I was able to think deeply about what I normally felt uncomfortable about and come up with a solution.
Editor’s Note_ Growth is difficult to quantify or feel. This is because people do not grow at a constant slope, but in steps. That’s why it’s difficult to realize growth until you climb one step. Therefore, I need colleagues around me who will watch over me and tell me about my growth. In order for us to grow steadily, we must be ‘together’ by sharing knowledge, clashing with different opinions, and fighting against each other.
The in-house hackathon was started to promote the ‘growth’ of such members and change the ‘development culture’ for the better. It also serves as a venue for obtaining new business ideas. Of course, visible results are not immediately apparent. It also costs quite a bit. However, we grew through the process and experience of bringing together people from various positions who had never collaborated with each other and put their heads together to solve problems, and we held an in-house hackathon because we knew that our growth would lead to the growth of the company. It is being held twice.
Since the first in-house hackathon started in 2023, small changes have occurred in Cloud. The experience of using ChatGPT, which was discussed twice, has spread to organizations, and the use of generative AI has recently increased significantly in various fields. There are also members who ask, ‘When will we have another hackathon?’
Although the three Russian Roulette CS slackbots have not been applied to internal CS yet, this hackathon at least served as an opportunity for us to think together about our CS problems. As long as there are members with the will to improve these inefficiencies, various problems in CS will gradually improve. The changes brought about by the in-house hackathon are still continuing. ☁️
Editor's Note_ Good news just a few days after introducing CS Bang on the technology blog. I heard this. There was news that discussions had begun to officially introduce CS Bang Slackbot. We will cover the story behind CS Bang Slackbot later as it became established as a CS process and what changes it brought about.
- SQD is a form of organization and a representative purpose organization. SQD is a form of teamwork of related personnel to achieve a specific purpose. Developers, designers, marketers, PMs, etc. team up in one SQD. On the other hand, a team, which is a representative functional organization, is made up of designers among designers, and developers among developers. Cloud has a hybrid organizational system that takes advantage of both functional and purpose organizations. ↩︎
