This article was contributed to Amazon Web Services (AWS).

goorm is an AI·SW EduTech startup founded in 2013, working towards realizing the vision of ‘everyone becomes a developer’. To enable everyone to receive quality SW education and grow, it provides related services to solve the challenges of setting up development environments and utilize cloud resources to freely develop software capabilities.

One of goorm’s key services, goormDEVTH, was launched in 2016, providing a ‘non-face-to-face assessment environment’ where talents can be analyzed and competencies evaluated in an environment similar to actual industry development settings. This service is being utilized by many companies and organizations such as LG, HYUNDAI MOTOR COMPANY, KOREA Data Agency, and more.

written by curiduck.kim, adalyn.kim, loki.ki
edite by snow.cho

Challenge of Non-face-to-face Competency Assessment

Following the release of goormDEVTH in 2016, goorm faced the following key challenges.

1. Overcoming Market Resistance: Overcoming the market’s fear and resistance towards a new method of remote assessment.
2. Ensuring Fairness: Guaranteeing the integrity of assessments and preventing misconduct in an environment where exams can be conducted anywhere without on-site supervisors.
3. Real-time monitoring requirement: Addressing the increased demand for real-time monitoring due to COVID-19.
4. Large-scale concurrent access handling: Preparation for situations where a large number of users access the system simultaneously, such as during the job hunting season.
5. Cost efficiency: Establishing a cost structure that can flexibly respond according to usage.

Limitations of the existing solution

goormDEVTH initially used a post-capture analysis method to ensure the fairness of remote assessments. This method involved periodically capturing the screens of assessment applicants and analyzing them after the assessment to supervise any misconduct.

It used to be a post-sanction approach to identify suspicious behavioral patterns such as leaving the seat for a certain period of time or continuously looking in a specific direction and then taking action afterwards. However, this approach required a significant amount of time and manpower as supervisors had to manually review numerous capture images during the monitoring process. It was also challenging to determine the accuracy of misconduct solely based on capture images. In particular, the post-sanction method was controversial as it imposed disadvantages after the evaluation had already been completed.

To address this issue, real-time monitoring functionality needed to be implemented. In particular, with the shift to non-face-to-face assessments becoming commonplace due to COVID-19, the importance of real-time monitoring functionality has become even greater.

Background of Amazon Chime SDK

goorm has started reviewing the technology that can accommodate various customer demands for real-time monitoring features. The requirements at that time were as follows.

1. Each assessment is supervised by over 1,000 applicants simultaneously.
2. Supervisor room for monitoring 10 or more applicants.
3. Real-time control over applicants (such as assessment pause, resume, etc.)
4. Using a webcam, the supervision leaves no blind spots, monitoring the applicant’s face, test screen, and the surrounding environment with a mobile camera.
5. During the assessment period, ensure service stability without any issues.
6. Minimize the time and cost required for construction.

The key feature of real-time monitoring function is to receive video and related data from various input sources such as webcams, smartphone cameras, and screen sharing, and deliver them to supervisors in real-time. It is similar to video conferencing technology, but the main difference lies in its ability to accommodate large-scale assessments with over 1,000 applicants per evaluation.

Furthermore, most of the video conferencing technologies were limited to accommodating up to 10 participants in a single meeting room, but ‘real-time monitoring’ needed to support more attendees. In addition to these constraints, all of goorm’s services utilize AWS infrastructure, and after reviewing Slack’s case of transitioning both video and voice services to Amazon Chime SDK for its proven security, post-support from AWS, we ultimately decided to adopt Amazon Chime SDK.

Introduction to Amazon Chime SDK

Amazon Chime SDK is a Software Development Kit that combines various features necessary for video conferencing. To create code that runs on a specific platform, operating system, or programming language, a debugger compiler and libraries are required. Amazon Chime SDK simplifies and speeds up the process of creating video conferencing services by providing these components in one place. In addition to desktop, you can host or join video conferences, chat, and share screens on iOS and Android devices. Particularly, when configuring directly with WebRTC, you would need to manage server resource, scaling, load balancing, etc. However, Amazon Chime SDK offers the advantage of being a ‘serverless’ service, as AWS manages the infrastructure.

With the introduction of Amazon Chime SDK, we were able to complete goormDEVTH’s real-time monitoring prototype in just two weeks.

The key advantages of the Amazon Chime SDK that we experienced during the development process are as follows.

1. Convenient Monitoring: Easily monitor and manage service status through AWS Console and Amazon CloudWatch.
2. Cost efficiency: Beneficial for coding tests with significant usage fluctuations due to a usage-based billing system.
3. Technology stack compatibility: Excellent compatibility with existing technology stack through JavaScript SDK support.
4. Flexible Scalability: Easy scale in/out with just adjusting RPS (Requests Per Second).
5. Various language support: In addition to JavaScript, support for a variety of programming languages and frameworks such as Python, PHP, .NET, Ruby, Java, Go, and Node.js.
6. AWS service integration: Facilitating the development of an integrated solution that connects with other SaaS services of AWS.

Amazon Chime SDK Architecture

The architecture of Amazon Chime SDK is as follows.

The Customer Applications area in the client-side runs on mobile, web, and desktop environments, integrating the Amazon Chime SDK to provide real-time audio, video, chat functions, and handle tasks such as user participation in video meetings.

Customer Service is a server-side application that handles customer requests. The AWS SDK here is a library set that helps the application interact with various AWS services like Amazon CloudWatch. It assists the application in initiating or managing meetings and processing media streams by invoking the features of Amazon Chime. In this area, it also takes care of controlling access permissions for the application to access AWS resources using IAM (Identity and Access Management).

Finally, the Control Plane Region and Media Plane Region are the core areas of Amazon Chime located in the AWS Cloud. The Control Plane is responsible for all operations required for meeting management and control in Amazon Chime, while the Media Plane is the area that handles real-time media streams (audio and video). This is where actual data such as audio, video, and screen sharing between meeting participants is transmitted and processed.

Real-time monitoring feature ‘Obserview’ utilizing Amazon Chime SDK.

The real-time monitoring feature of goormDEVTH is named ‘Obserview.’ This name is a combination of ‘Observer,’ meaning a supervisor or observer, and ‘View,’ indicating superiority.

Obserview allows supervisors to monitor multiple applicants in real-time on one screen. It provides features such as webcam, smartphone camera, applicant screen sharing, real-time exam logs, and chat, enabling supervisors to monitor and assess applicants without blind spots during remote competency assessments.

The applicant’s actions, in addition to their frontal view, surroundings, and test screen, are recorded in logs. It is possible to check in the logs which problems they are solving, whether they answered correctly, and if any misconduct is suspected, a message can be sent to illuminate the surrounding environment with the smartphone camera. Furthermore, the supervisor can temporarily pause the assessment or extend the evaluation time.

Obserview architecture

The API server and webcam server in the service domain are running on Amazon EC2, with the API server handling communication with Amazon Chime SDK and API. The webcam server captures the applicant’s video feed, webcam, and smartphone camera footage, storing them in AWS Simple Storage Service (S3).

Description of Obserview’s real-time video sharing system

1. The applicant creates a meeting room to broadcast their Webcam/Mobile/Screen for connection

const observiewConnector = {};
// When an applicant connects to the waiting room, information about the chime meeting room is retrieved.

const setObserviewConnection = async () => {

	// Manage information about the meeting room
	let meetData = await getMeetData();
	
	// webcam = applicant's PC webcam
	// screen = applicant's PC screen
	// mobile = applicant's mobile screen
	const mediaDeviceTypes = ['webcam', 'screen', 'mobile']
	
	// If there is no meetData, create meeting data
	if (!meetData) {
		meetData = await setMeetData();
		...
	}

	// Create video dom based on the retrieved screen sharing data
	// Iterate through webcam and screen sharing types to connect video to Obserview
	mediaDeviceTypes.forEach((mediaDeviceType)=>{
		// Utilize the AWS Chime SDK to manage applicant device connections and receive events
		// Notify Obserview when the applicant's video status changes and connect the video
		observiewConnector[mediaDeviceType] = getObserviewConnector({});
		
	})

};

/** Get meeting information */
const getMeetData = async () => {
		const { data } =
			await axios.get(
				`/api/obserview/meetInfo?examId=<<EXAM_ID>>`
			);
		return data;
};

Code_WebRTCManager.js manages webcam and screen sharing streams using AWS Chime SDK, connecting them to the video status of applicants (users). The setObserviewConnector function connects videos based on media device data such as webcams and screen sharing, handling real-time streaming.

Code_WebRTCManager.js is the implementation part that manages webcam and screen sharing streams using AWS Chime SDK, and connects them with the video status of applicants (users). The setObserviewConnector function connects videos based on media device data such as webcams and screen sharing, and handles real-time streaming.

1. Import meetData:
   • Calling getMeetData() retrieves meetData, which contains the necessary information for the meeting.
2. Video DOM creation:
   • Iterates through each mediaDeviceType to manage applicants’ device connections, where various media devices such as webcams and screen sharing may exist.
3. Device connectivity using AWS Chime SDK:
   • The ObserviewConnector manages the video status of applicants using AWS Chime SDK through a connection object called ObserviewConnector. This class notifies Obserview when the video status of an applicant changes and appropriately connects the streaming.
4. Connection with media devices:
   • For each mediaDeviceType, create an ObserviewConnector object, retrieve the device data from meetData, and establish the connection.
   • In this process, additional data such as exam identification values is provided to manage each applicant’s status according to specific situations.

2. Implementation part for retrieving meeting room information

/** Retrieves the chime meeting room that the user belongs to. */
router.get(
	'/meetInfo',
	async (req, res) => {
		...

		// Check if there is a meeting room that the user can access based on the exam.
		const meetInfo = await ObserviewService.getMeetingInfo({
			examId,
			userId
		});

		// If a meeting room exists,
		// notify the supervisor via socket that the applicant has entered the exam room.
		if (meetInfo) {
			socketIo.send_room_message({
				room: `supervisor-room-id`,
				arguments: ['obserview-connection-start', userId],
			});
		}
		res.json(meetInfo);
	}
);

The Code_Server routes/api code processes incoming requests to a specific API endpoint using the router.get method. It is responsible for retrieving and responding with meetInfo (meeting information) for real-time connections.

1. Definition of API path:
   • The HTTP GET request coming to the /api/observiewConnection/meetInfo path is handled through router.get.
   • Using asynchronous functions (async) to process data asynchronously.
2. Meeting information retrieval:
   • Calling the observiewService.getMeetingInfo() function retrieves the meetInfo.
   • This function takes data such as examId and userId as parameters.
     • examId: An index that identifies a specific exam or meeting.
     • userId: Represents the ID of the user who sent the request.
3. If there is meeting information:
   • If meetInfo exists, call socketIo.send_room_message() to send a message to the corresponding room (supervisor-room-id).
     • This message delivers the userData.id along with an event called obserview-connection-start.
     • This is a message to inform the supervisor that the applicant is connected to the waiting room and the examination room.
4. Client Response:
   • Finally, respond to the client with the meetInfo data in JSON format using res.json(meetInfo).

/** Creates a meeting along with attendee invitations */
async createMeetingWithAttendees({ examId, userId, deviceTypes = [] }) {

	// Generate an ID for tracking the user in chime
	const userObserviewId = generateChimeTrackingId(userId)

	// Retrieve the list of supervisors for the exam
	const supervisorIds =
		await userModel.getSupervisors({ examId }) || [];
	
	// Generate trackable IDs for the supervisors
	const supervisorExternalIds =
		supervisorIds
			.map(supervisorId => generateChimeTrackingId(supervisorId))
			.map(supervisorId => ({ExternalUserId: supervisorId}))
	
	// Generate IDs for tracking multimedia devices (up to 3 per applicant)
	const userExternalIdsWithDevices = 
			deviceTypes.map((deviceType) => { ExternalUserId: `${userObserviewId}/${deviceType}`})

	// Create a meeting in chime
	const chimeMeetingAndAttendees = await chime
		.createMeetingWithAttendees({
			ClientRequestToken: `YOUR_ClientRequestToken`,
			MediaRegion: 'YOUR_MediaRegion',
			ExternalMeetingId: `YOUR_ExternalMeetingId`,
			Attendees: [
				...userExternalIdsWithDevices,
				...supervisorExternalIds,
			],
		})
		.promise();

	// Information of the created meeting
	const meeting = chimeMeetingAndAttendees.Meeting;

	// Information of the invited meeting attendees (applicants per media, supervisors)
	const attendees = chimeMeetingAndAttendees.Attendees || [];

	// Add the created meeting information to the database
	await observiewModel.addConnection({
		examId,
		userId,
		meetingId: meeting.MeetingId,
		info: meeting,
	});
	// Add the created meeting attendee information to the database
	// This information will be used to find the connection targets when accessing the supervisor's room, etc.
	await observiewModel.addAttendees(attendees)

	return {meeting, attendees};
}

The createMeetingWithAttendees function invites attendees to the meeting simultaneously with the meeting creation. It invites the applicant and the supervisor to the meeting, allowing the supervisor to view the applicant’s video.

5. User ID formatting:
   • The user ID is converted into an External ID format that can be identified within Chime.
6. Meeting creation:
   • Call the `createMeetingWithAttendees` function of chime to create a meeting and add attendees.
7. Save the generated meeting information to the database.
   • When the supervisor logs in, observiewModel.addConnection allows the supervisor to access the meeting associated with the applicant.

3. The supervisor can connect to the applicant’s screen for monitoring

The following codes retrieve meeting information of applicants connected to AWS Chime and involve the supervisor in the meeting. The code and operational flow for both client and server are as follows.

/** 
 * Component that displays the user's videos in a grid format 
 */ 
function UserVideoCardGrid({ gridColumnSize, users }) { 
	return ( 
		<CardGrid columnCount={gridColumnSize}> 
			{users.map((user, index) => ( 
				<UserVideoCard key={user.key} id={user.id}/> 
			))} 
		</CardGrid> 
	); 
} 

/** 
 * Component that displays the video streamed by the applicant in Obserview 
 */ 
function UserVideoCard({ id }) { 
	return ( 
		<div 
			className={classnames( 
				styles.LabelVideo 
			)} 
		> 
			<video id={id} /> 
		</div> 
	); 
}

1. Supervisor Room Screen:
   • When the supervisor enters the real-time supervisor room, they retrieve the applicant meeting information stored in the DB.
   • The UserVideoCard component creates a video element for each user, mapping it with an ID to connect with Chime’s tile.

/**
 * Function to connect the video stream in Chime
 */
const connectVideo = ({
	user
}) => {
	
	const videoDomInfoList = user.videos.map((video) => ({
		userId: user.id,
		mediaDeviceType: video.type,
		domId: video.domId,
	}));
	const meetingKey = `YOUR_MeetingKey`;
	const meetData = meetingInfoMap.get(meetingKey);
	
	const originMeetingId =
		ObserviewConnectionMap.get(meetData.meeting.ExternalMeetingId)?.meetingId;
	
	const observiewConnector = getObserviewConnector({
		videoDoms: videoDomInfoList,
	});
	
	ObserviewConnectionMap.set(meetData.meeting.ExternalMeetingId, {
		userId: user.id,
		meetingId: meetData.meeting.MeetingId,
	})
};

2. connectVideo.js
   • To receive the video feed of the attendees in the invited meeting, the supervisor connects the connector.
   • Pass the reference array of the generated video DOM to the Obserview connector.

/** 
 * Part of the getObserviewConnector function 
 */
import {
  DefaultDeviceController,
  DefaultMeetingSession,
  MeetingSessionConfiguration,
} from 'amazon-chime-sdk-js';

const API_ENDPOINT_URL = 'https://<<AWS_API_GW_ENDPOINT>>/api/meetingevents';

const qualitySet = {
  // Resolution
  'resolution': {
    width,
    height,
    frameRate,
    maxBandwidthKbps
  }
};

const getObserviewConnection = async ({
  meeting,
  attendee,
  logger,
  quality,
  videoStream,
  onChangeRemoteVideos,
  DEFAULT_INTERVAL_SEC
}) => {
  let meetingEvents = [];

  const deviceController = new DefaultDeviceController(logger);
  deviceController.chooseVideoInputQuality(
    qualitySet[quality].width,
    qualitySet[quality].height,
    qualitySet[quality].frameRate,
    qualitySet[quality].maxBandwidthKbps
  );
  const configuration = new MeetingSessionConfiguration(meeting, attendee);
  const meetingSession = new DefaultMeetingSession(
    configuration,
    logger,
    deviceController
  );

  const sendMeetingEvents = () => {
    fetch(API_ENDPOINT_URL, {
      method: 'POST',
      body: JSON.stringify(meetingEvents),
    });
  }

  if (videoStream) {
    await meetingSession.audioVideo.chooseVideoInputDevice(videoStream);
  }

  const observer = {
    audioVideoDidStop: (sessionStatus) => {
      const sessionStatusCode = sessionStatus.statusCode();
      // Handle exceptions based on statusCode
    },
    eventDidReceive: (name, attributes) => {
      // Logic to handle events when received
    },
  };

  if (onChangeRemoteVideos) {
    observer.remoteVideoSourcesDidChange = (remoteVideoSources) => {
      // Track remote video length to monitor device connections of the participant
      onChangeRemoteVideos(remoteVideoSources?.length || 0);
    };
  }

  if (Array.isArray(videoDoms) && videoDoms.length > 0) {
    observer.videoTileDidUpdate = (tileState) => {
      // Check if the user who joined the meeting has a corresponding DOM
      const videoDomInfo = videoDoms.find(
        (dom) =>
          tileState.boundExternalUserId === getUserExternalId(dom.userId, dom.mediaDeviceType)
      );

      const videoDom = document.getElementById(videoDomInfo.domId);
      if (videoDom) {
        meetingSession.audioVideo.bindVideoElement(
          tileState.tileId,
          videoDom
        );
      }

    };
  }

  meetingSession.audioVideo.addObserver(observer);
  meetingSession.audioVideo.start();
  meetingSession.audioVideo.startLocalVideoTile();

  const interval = setInterval(() => {
    sendMeetingEvents();
  }, DEFAULT_INTERVAL_SEC);

  return {
    meetingSession,
    interval,
  };
};

export default getObserviewConnection;
```

3. ObserviewConnector.js
   • Within the Obserview connector, it manages the actual chime connections.
   • The `bindVideoElement` function sets up the applicant’s video for the supervisor to view, and updates it when the applicant’s broadcasting status changes.

// When the server detects that the applicant has connected to Obserview, connect the applicant with the meeting.
socket.on('obserview-connection-start', (userId) => {
    fetchObserviewConnection({userId, examId, includedUser});
});


// Retrieve the AWS Chime meeting information for a specific applicant and add the supervisor as a participant.
export const fetchObserviewConnection = async ({ userId, examId }) => {
  const response = await axios.get('/api/manager/observiewConnection',
      { params: { examId, userId } });
  return {
    params: { userId },
    data: response.data,
  };
};

4. socketManager
   • If an applicant successfully connects to the waiting room, a notification will be sent to the supervisors in the supervisor room.
   • Upon receiving the notification, the supervisor updates the supervisor room information to acknowledge the new user has logged in.

Case of a large-scale coding test with 2,000 applicants

At goormDEVTH’s service, the most important aspect is conducting competency assessments without interruption in a timely and stable manner. Following that, providing the ‘smooth real-time monitoring’ service is crucial. These two elements are key factors determining the reliability of remote competency assessments.

In June 2024, goormDEVTH conducted a large-scale coding test to verify the reliability and efficiency of these services. The test, conducted by N Company, was a large-scale event with approximately 2,000 applicants and over 100 supervisors participating.

Considerations and Preparation Process for Large-Scale Events

When the competency assessment begins, applicants gather. Even in such worst-case scenarios, goormDEVTH must continue to operate without interruption. If a service disruption occurs, system errors may occur for specific applicants, or the system speed may slow down, compromising the fairness of the test. The damage extends beyond the applicants and can cause significant harm to the clients.

Therefore, goorm has prepared through the following process before competency assessments such as coding tests.

• [D-30] The business representative has signed a coding test contract. The schedule, number of participants, and test options have been shared. The initial expected number of applicants was approximately 3000.
• [D-14] We discussed the server scaling settings in relation to the applicant scale between the SRE Squad and Product Squad. We determined the deployment scale of the WAS and Code Build Server, and calculated the expected connection count for simultaneous access related to Chime.
• [D-12] Regarding Chime, we have requested AWS to relax the limitation on the expected RPS limit setting for the CreateMeetingWithAttendees feature.
• [D-7] We have checked whether the Chime API RPS Limit relaxation has been applied in the AWS Console.
• [D-2] We conducted Load Testing with server scale-out preparation and scenario setup for the exam.
• [D-day] Large-scale coding test examination was conducted.

Applicants enter the waiting room before the competency assessment and wait. They set up the necessary equipment for real-time supervision (webcam, smartphone, screen sharing) and conduct tests. When the time comes, they enter the examination room and take the test.

Applicants typically start entering the waiting room 30 minutes before the test. The challenge lies in users accessing the system simultaneously once the exam begins. The maximum user access trend to the waiting room observed in other competency assessment cases is as follows.

Based on these past patterns, we have estimated the maximum number of expected users, expected connection count, and traffic, taking into account the scaling of WAS and Code Build Server in advance at the beginning of the competency assessment.

When an applicant enters the waiting room, a meeting room that the applicant will connect to is created. Meeting rooms are limited to a maximum of 3,000, with only one room generated per person. The number of participants in the meeting is limited to 9,000, calculated as three media streams (camera, screen sharing, webcam) per 3,000 attendees required for Obserview.

On the day of the exam, we calculated and monitored the RPS using the Chime Event Log in Amazon CloudWatch Logs.

Errors that occurred during communication with the Chime server are collected in the service logs, and we monitored them by cross-checking with the Chime Event Log.

After thorough preparation, I successfully completed N company’s coding test. Without a single issue, I was able to complete a large-scale coding test with over 2,000 applicants participating.

Closure

The real-time monitoring feature of goormDEVTH Obserview, released in 2016, has been further enhanced by AI in January 2024. The previous Obserview required personnel to be present in the supervisor room for real-time monitoring, but the AI Obserview automatically detects misconduct post-examination. This means supervisors no longer need to monitor in real-time directly.

Even for large corporations, manpower and costs are never sufficient. While they could have chosen to develop their own services, goorm quickly met market demands and established a leading position in the remote assessment market by introducing the Amazon Chime SDK, a proven reference from their trusted partner AWS, and rapidly launching a prototype in just two weeks.

Moving forward, goorm will continue to swiftly respond to the ‘competency assessment’ market in collaboration with a trusted partner, AWS, and will persistently provide customer-centric remote competency assessment services.