Assessing adaptability using multiple speed assessments

By Jovina Ang

SMU Office of Research – Adaptability is a highly sought after skill for today’s VUCA (volatile, uncertain, complex and ambiguous) world.

It has been recognised as one of the key skills of the 21st century by Singapore’s Ministry of Education (MOE) alongside critical thinking, effective communication, and collaboration.

This explains why organisations are constantly seeking ways to successfully identify and assess the skill of adaptability in their applicants and existing employees.

“Before the introduction of this method of assessment, which we call multiple speed assessments (MSAs), adaptability is often assessed using self-reported trait-based surveys or cognitive learning tasks. These assessment methods are limited, though, as they are based on what people report instead of on their actual behavioural displays,” SMU Professor of Human Resources Filip Lievens told the Office of Research.

“It is for this reason we wanted to develop a more robust method of assessment. In MSAs, people’s behaviours and reactions are evaluated based on ‘thin and brief’ slices of observations. Participants of MSAs are asked to participate in a large set of short role plays that mimic the work situation. These role plays are either 1-minute or 3-minute long and are evaluated by different trained assessors,” he continued.

He added: “The MSA method is a method of assessment that combines elements of Situational Judgment Tests (SJTs) and Assessment Centres (ACs). Like SJTs, MSAs can cover a wide range of job-related situations. And like ACs, they focus on assessing interactive behaviours as a result of how people adapt to different work contexts.”

The research

The research project, which was helmed by Professor Lievens, Dr. Jan Corstjens, fellow at SMU Lee Kong Chian School of Business, and Dr. Jomel Ng, who is Assistant Professor at Zhejiang International Business School in China, was centred on three objectives:

1. To determine if MSAs are a reliable method of assessing adaptability;

2. To investigate whether MSAs could lead to improved predictions of adaptability; and

3. To determine whether MSAs are a fair and unbiased assessment method for minority groups such as foreigners and women.

This research was also funded by an MOE Academic Research Fund Tier 2 grant.

Dr. Jan Corstjens (left), fellow at SMU Lee Kong Chian School of Business, and Dr. Jomel Ng (right), Assistant Professor at Zhejiang International Business School in China

Before rolling out the research, the research team conducted a pilot to test and refine the role plays in the MSAs. A total of 18 role plays were created.

With the support from SMU Student Success Centre and the Centre for Teaching Excellence, the research project which became the ‘Agility Game’ was rolled out to a sample size of over 200 undergraduate Gen Z students during the freshmen onboarding period for this academic year.

“Students were told that the Agility Game would give them an opportunity to evaluate, learn and improve their adaptable leadership and communication skills as they would be given real-time feedback after each role play,” Dr. Corstjens explained.

30 undergraduate and graduate students were recruited and trained as role-playing assessors and raters for the students’ performance according to international assessment standards. These role-playing assessors’ scores were then aggregated to reflect a total score for each student participant.

Results

The results from the Agility Game were overwhelmingly promising.

High validities in predicting future performance were obtained from the study. This result implies that MSAs can be used to accurately forecast people’s adaptability.

MSAs were shown to provide a fair and unbiased method of assessment regardless of nationality or gender due to the aggregation of scores from the different assessors.

And based on further analyses using Machine Learning algorithms, it is foreseeable that future MSAs could adopt a combined Natural Language Processing (NLP)/ Machine Learning (ML) learning approach. The benefits of such an approach include costs and time savings as one or more assessors could be substituted with algorithms.

“The feedback from the Gen Z freshmen students was positive too. They appreciated the experiential learning that they obtained from participating in the MSA,” added Dr. Corstjens.

Implications of the research

Given that adaptability is such a sought after skill in today’s environment, one implication is for organisations to adopt MSAs as a reliable solution for assessing adaptability, which is an important indicator for future job performance.

MSAs have multiple advantages over SJTs and ACs. They provide both a time- and cost-efficient assessment method. And they are a fair and unbiased assessment method for organisations that hire large numbers of applications on a regular basis.

This research also has significant implications for the scientific community.

Commented Professor Lievens: “MSAs provide a dynamic and context-specific method to assess adaptability, rooted in the understanding on how individuals interact with their environment. The MSAs enrich our understanding of adaptability and have been proven to pave the way for future research delving deeper into the mechanisms of adaptability.”

Having successfully implemented MSAs in person, Professor Lievens and his team have plans to extend the project by incorporating virtual reality into MSAs and virtual coaches to provide feedback.

Back to Research@SMU November 2023 Issue