Situational Judgment

One of my areas of great interest involves the use of low fidelity simulations (i.e., Situational Judgment Tests, SJTs) in high stakes selection contexts. SJTs present candidates with a hypothetical situation and ask them what they would do in that situation. There exists a lot of interest to use situational judgment tests as supplements to more traditional cognitive ability tests in high-stakes testing situations (e.g., in the context of college admission).

My earlier research dealt with the criterion-related validity, fakability, and coachability of SJTs. I found that video-based SJTs that were constructed as part of the college admission to medical studies in Belgium became more valid over the years and had incremental validity over traditional cognitive ability tests.

My more recent studies tackle fundamental issues in SJTs such as the questions as to whether they can be designed to assess (a) general domain knowledge and (b) cross situational variability.

Overview Articles

The articles below are a good starting point to become familiar with the "ins and outs" of SJTs and with relevant research studies.

Lievens, F., Schäpers, P., & Herde, C.N. (2021). Situational Judgment Tests: From low-fidelity simulations to alternative measures of personality and the person-situation interplay. In D. Wood, P. Harms, S. Read, & A. Slaughter (Eds.) Emerging approaches to measuring and modeling the person and situation (pp. 285-311). Elsevier.

Lievens, F. (2017). Construct-driven SJTs: Towards an agenda for future research. International Journal of Testing, 29, 269-276.

Corstjens, J., Lievens, F., & Krumm, S. (2017). Situational Judgement Tests for Selection. In H. W. Goldstein, E. D. Pulakos, J. Passmore, & C. Semedo (Eds.), The Wiley Blackwell Handbook of the Psychology of Recruitment, Selection, and Employee Retention (pp. 226–246). Hoboken, NJ: Wiley-Blackwell.

Oostrom, J.K., De Soete, B., & Lievens, F. (2015). Situational Judgment Tests: A review and some new developments. In I. Nikolaou & J. K. Oostrom (Eds.), Employee recruitment, selection, and assessment: Contemporary issues for theory and practice (pp. 172-189). Sussex, UK: Psychology Press.

Lievens, F., De Soete, B. (2015). Situational Judgment Test. In: James D. Wright (Ed.), International Encyclopedia of the Social & Behavioral Sciences (pp. 13–19), 2nd edition, Vol 22. Oxford: Elsevier.

Lievens, F., & De Soete, B. (2012). Simulations (pp.383-410). In N. Schmitt (Ed.) Handbook of Assessment and Selection. Oxford University Press.

Sackett, P.R., & Lievens, F. (2008). Personnel selection. Annual Review of Psychology, 59, 419-45.

Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37, 426-441.

SJT Validity & Subgroup Differences

The following articles focus on establishing the validity of SJTs for predicting performance. In addition, articles are listed that examine to what extent SJTs lead to fewer subgroup differences than traditional cognitive ability tests, which is pivotal for increasing diversity. These studies also identify factors (e.g., an open-ended response format) that might contribute to SJTs being less prone to subgroup differences.

Herde, C. N., Lievens, F., Jackson, D. J. R., Shalfrooshan, A., & Roth, P. L. (2020). Subgroup differences in situational judgment test scores: Evidence from large applicant samples. International Journal of Selection and Assessment, 28, 45-54.

Lievens, F., Sackett, P. R., Dahlke, J., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority-majority differences and validity. Journal of Applied Psychology, 104, 715–726.

Lievens, F., Patterson, F., Corstjens, J., Martin, S., & Nicholson, S. (2016). Widening access in selection using Situational Judgment Tests: Evidence from UKCAT. Medical Education, 50, 624-636.

Slaughter, J.E., Christian, M.S., Podsakoff, N.P., Sinar, E.F., & Lievens, F. (2014). On the limitations of using Situational Judgment Tests to measure interpersonal skills: The moderating influence of employee anger. Personnel Psychology, 67, 847-885.

Patterson, F., Lievens, F., Kerrin, M., Munro, N., & Irish, B. (2013). The predictive validity of selection for entry into postgraduate training. British Journal of General Practice, 63, 734-741.

Patterson, F., Lievens, F., Kerrin, M., Zibarras, L., & Carette, B. (2012). Designing selection systems for medicine: The importance of balancing predictive and political validity in high stakes selection contexts. International Journal of Selection and Assessment, 20, 486-496.

Lievens, F., & Sackett, P. (2012). The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. Journal of Applied Psychology, 97, 460-468.

Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced level high-stakes selection. Journal of Applied Psychology, 96, 927-940.

Lievens, F. (2013). Adjusting medical admission: Assessing interpersonal skills via situational judgment tests. Medical Education, 47, 182-189.

Buyse, T., & Lievens, F. (2011). Situational Judgment Tests as a new tool for dental student selection. Journal of Dental Education, 75, 743-749.

Lievens, F., & Sackett, P.R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181-1188.

Lievens, F., Buyse, T., & Sackett, P.R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452.

Lievens, F., & Coetsier, P. (2002). Situational tests in student selection: An examination of predictive validity, adverse impact, and construct validity. International Journal of Selection and Assessment, 10, 245-257.

Alternatives to Traditional SJT Formats

The articles below examine all kinds of alternatives to the traditional SJT format. Examples are webcam SJTs, SJTs with open-ended response formats, gamified SJTs, or experiments with response instructions. In other words, these articles open a window of opportunity for a more creative design and use of SJTs.

Herde, C.N., & Lievens, F. (2023). Multiple, speeded assessments under scrutiny: Underlying theory, design considerations, reliability, and validity. Journal of Applied Psychology, 108, 351-373.

Georgiou, K., & Lievens, F. (2022). Gamifying an assessment method: What signals are organizations sending to applicants? Journal of Managerial Psychology, 37, 559-574.

Rockstuhl, T. & Lievens, F. (2021). Prompt-specificity in scenario-based assessments: Associations with personality vs. knowledge and effects on predictive validity. Journal of Applied Psychology, 106, 122-139.

Lievens, F., Sackett, P. R., Dahlke, J., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority-majority differences and validity. Journal of Applied Psychology, 104, 715–726.

Lievens, F., De Corte, W., & Westerveld, L. (2015). Understanding the building blocks of selection procedures: Effects of response fidelity on performance and validity. Journal of Management, 41, 1604-1627.

Rockstuhl, R., Ang, S., Lievens, F., & Van Dyne, L. (2015). Putting judging situations into Situational Judgment Tests: Verbal protocols and incremental validity. Journal of Applied Psychology, 100, 464-480.

De Soete, B., Lievens, F., Oostrom, J., & Westerveld, L. (2013). Alternative predictors for dealing with the diversity-validity dilemma in personnel selection: The constructed response multimedia test. International Journal of Selection and Assessment, 21, 239-250.

Lievens, F., Buyse, T., & Sackett, P.R. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095-1101.

Lievens, F. & Peeters, H. (2008). Impact of elaboration on responding to situational judgment test items. International Journal of Selection and Assessment, 16, 345-355.

Lievens, F. (2000). Development of an empirical scoring scheme for situational inventories. European Review of Applied Psychology, 50, 117-124.

SJTs & Underlying Constructs

The following articles dig into the Achilles heel of SJTs, namely the construct-related validity of SJTs ("What do they measure?"). Some articles use new methodologies to tackle this issue, whereas other articles are more conceptually-oriented.

Schäpers, P., Freudenstein, J.-P., Mussel, P., Lievens, F., & Krumm, S. (2020). Effects of situation descriptions on the construct-related validity of construct-driven Situational Judgment Tests. Journal of Research in Personality.

Schäpers, P., Mussel, P., Lievens, F., König, C.J., Freudenstein, J.-P., & Krumm, S. (2020). The role of situations in Situational Judgment Tests: Effects on construct saturation, predictive validity, and applicant perceptions. Journal of Applied Psychology.

Schäpers, P., Lievens, F., Freudenstein, J.P, Hüffmeier, J., König, C.J., & Krumm, S. (2019). Removing situation descriptions from Situational Judgment Test items: Does the impact differ for video-based versus text-based formats. Journal of Occupational and Organizational Psychology, 93, 472-494.

Paton, L., Tiffin, P., O’Mara, D., Maccan, C., Lang, J., & Lievens, F. (2020). Situational Judgment Tests for selection: Traditional vs construct-driven approaches. Medical Education, 54, 105-115.

Lievens, F. (2017). Construct-driven SJTs: Towards an agenda for future research. International Journal of Testing, 29, 269-276.

Motowidlo, S.J, Lievens, F., & Ghosh, K. (2018). Prosocial Implicit Trait Policies underlie performance on different Situational Judgment Tests with interpersonal content. Human Performance, 31, 238-254.

Lievens, F. (2017). Integrating situational judgment tests and assessment center exercises into personality research: Challenges and further opportunities. European Journal of Personality, 31, 487-502.

Lievens, F. (2017). Assessing personality-situation interplay in personnel selection: Towards more integration into personality research. European Journal of Personality, 31, 424-440.

Sorrel, M.A., Olea, J., Abad, F.J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and reliability of Situational Judgement Test Scores: A new approach through cognitive diagnosis models. Organizational Research Methods, 19, 506-532.

Lievens, F., & Motowidlo, S.J. (2015). Situational Judgment Tests: From measures of situational judgment to measures of general domain knowledge. Industrial and Organizational Psychology – Perspectives on Science and Practice, 9, 3-22.

Krumm, S., Lievens, F., Hüffmeier, J., Lipnevich, A.A., Bendels, H., & Hertel, G. (2015). How “Situational” is Judgment in Situational Judgment Tests? Journal of Applied Psychology, 100, 399-416.

SJT Faking, Coaching & Retesting

The articles below deal with these three effects that might impact on the viability of SJTs in high-stakes settings. These articles also propose some solutions to avoid these effects.

Weng, Q. Yang, H., & Lievens, F. (2018). Optimizing the validity of situational judgment tests: The importance of scoring method. Journal of Vocational Behavior, 104, 199-209.

Stemig, M., Sackett, P.R., & Lievens, F. (2015). Effects of organizationally-endorsed coaching on performance and validity of Situational Judgment Tests. International Journal of Selection of Assessment, 23, 175-182.

Lievens, F., Buyse, T., Sackett, P.R., & Connelly, B.S. (2012). The effects of coaching on Situational Judgment Tests in high-stakes selection. International Journal of Selection and Assessment, 20, 272-282.

Lievens, F., & Sackett, P.R. (2007). Situational judgment tests in high stakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, 92, 1043-1055.

Cullen, M.J., Sackett, P.R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment, 14, 142-155.

Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college students' success: The influence of faking. Educational and Psychological Measurement, 65, 70-89.

Lievens, F., Buyse, T., & Sackett, P.R. (2005). Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981-1007.

SJTs in an International Context

This final set of articles examines to what extent SJTs need to be adapted to work well across different cultures. My 2006 book chapter lays out the conceptual framework, whereas the 2015 article applies it to Spain and the USA.

Herde, C.N., Lievens, F., Solberg, E.G., Harbaugh, J.L., &. Strong, M.H. (2019). Situational Judgment Tests as measures of 21st century skills: Evidence across European and South American Countries. Journal of Work and Organizational Psychology, 35, 65-74.

Lievens, F., Corstjens, J., Sorrel, M.A., Abad, F.J., Olea, J., & Ponsoda, V. (2015). The cross-cultural transportability of Situational Judgment Tests: How Does a US-based integrity Situational Judgment Test fare in Spain? International Journal of Selection of Assessment, 23, 361-372.

Lievens, F. (2006). International situational judgment tests. In J.A. Weekley & R.E. Ployhart (Eds.) Situational Judgment Tests (pp. 279-300). SIOP Frontier Series. Mahwah, NJ: Lawrence Erlbaum.

By downloading or in any way reproducing these files you accept all responsibility for meeting applicable copyright law. The files are placed here assuming they will only be used in accordance with fair use standards.