A comparative study of dynamic scoring formulas for capture-the-flag competitions

Recently, the use of gamification techniques to teach cybersecurity has increased, predominantly through capture-the-flag security competitions where players solve cybersecurity problems (challenges) to gain points on a scoreboard. To determine how point values are determined for each challenge, a formula is often used to assign point values based on the number of teams that have solved the challenge, tying point values to the difficulty of the challenge. There are many ways to create this formula; however, the effects that different scoring formulas can have on a competition is not clear. This is important because the way that scores are calculated can impact the motivations, morale, and overall experience of participants as the scores are reflected on scoreboards. In this study, we examine the effect that changing the scoring formula used in these competitions would have. We predicted that changing the scoring formulas would have a large effect on the distributions, with more gently sloped formulas moving the center of the scoring distribution to a higher number. This is because of the fundamental differences in the formulas’ mathematical compositions. We found that while the overall distributions and median scores stay largely the same when the competitions were re-scored using different formulas, the variability in the scores differed. This is important because score variation and distribution during a competition can have serious implications in competitions that aim to find talent or introduce people to the field.