Improving Human Decisions by Adjusting the Alerting Thresholds for Computer Alerting Tools According to User and Task Characteristics

Objective: To investigate whether performance (number of correct decisions) of humans supported by a computer alerting tool can be improved by tailoring the tool's alerting threshold (sensitivity/specificity combination) according to user ability and task difficulty. Background: Many researchers implicitly assume that for each tool there exists a single ideal threshold. But research shows the effects of alerting tools on decision errors to vary depending on variables such as user ability and task difficulty. These findings motivated our investigation. Method: Forty-seven participants edited text passages, aided by a simulated spell-checker tool. We experimentally manipulated passage difficulty and tool alerting threshold, measured participants' editing and dictation ability, and counted participants' decision errors (false positives + false negatives). We investigated whether alerting threshold, user ability, task difficulty and their interactions affected error count. Results: Which alerting threshold better helped a user depended on an interaction between user ability and passage difficulty. Some effects were large: for higher ability users, a more sensitive threshold reduced errors by 30%, on the easier passages. Participants were not significantly more likely to prefer the alerting threshold with which they performed better. Conclusion: Adjusting alerting thresholds for individual users' ability and task difficulty could substantially improve effects of automated alerts on user decisions. This potential deserves further investigation. Improvement size and rules for adjustment will be application-specific. Application: Computer alerting tools have critical roles in many domains. Designers should assess potential benefits of adjustable alerting thresholds for their specific CAT application. Guidance for choosing thresholds will be essential for realizing these benefits in practice.