Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models

Overview

The paper presents the Trustworthiness Calibration Framework (TCF) for evaluating phishing email detection systems using large language models (LLMs). It emphasizes the need for trust-aware evaluation beyond mere accuracy.

Key Insights

• Trustworthiness Calibration Framework (TCF): A methodology to assess phishing detectors across calibration, consistency, and robustness.
• Trustworthiness Calibration Index (TCI): A bounded index measuring trustworthiness.
• Cross-Dataset Stability (CDS): A metric quantifying stability across datasets.
• Performance: GPT-4 outperforms LLaMA-3-8B and DeBERTa-v3-base in trust profile.
• Statistical Analysis: Reliability varies independently of raw accuracy.

BFSI Relevance

• Why Relevant: Phishing detection is critical for cybersecurity in BFSI, impacting data protection and fraud prevention.
• Primary Sector: Financial Services
• Subsectors: Cybersecurity
• Actionable Implications: BFSI professionals should integrate trust-aware evaluation frameworks like TCF to enhance phishing detection systems.