Uncertainty-Aware Metrics for Revealing LLM Bias Toward Behavioral Data Attributes in Cybersecurity
Large Language Models (LLMs) have demonstrated strong performance in content analysis tasks such as scam detection. However, their use in high stakes domains like cybersecurity raises critical concerns about fairness and reliability. Existing bias evaluation methods predominantly rely on performance-based metrics (e.g., prediction probabilities) that often overlook bias emerging from model uncertainty. To address this gap, we propose a novel suite of uncertainty-aware bias metrics that jointly incorporate performance and uncertainty to capture nuanced forms of bias in LLMs.
Our approach introduces two complementary uncertainty quantification strategies: (1) entropy-based metrics, including total, epistemic, and aleatoric uncertainty; and (2) evidential reasoning metrics based on vacuity and dissonance. These metrics are computed via prompt-level diversity and decoding variability, enabling robust, black-box-compatible evaluation. We introduce two multi-objective bias metrics: Sub-optimality Distance, which measures the deviation from an ideal prediction (confidence = 1, uncertainty = 0), and Dominance Score Bias (DSB), which captures directional disparities in performance between subgroups. We validate our framework on both synthetic bias scenarios and real-world scam datasets, analyzing LLaMA3’s performance on behavioral attributes associated with deception. Our metrics reveal model biases missed by existing approaches, including overconfidence toward specific subgroup characteristics. Notably, our experiments show that the DSB metric consistently detects bias across uncertainty types and is more sensitive than traditional AEG metrics to shifts in vacuity distributions, highlighting its value for fairness analysis in LLM-based cybersecurity systems.
Publications
Under Review
Authors: