Cost-Aware Selective Text Classification with Calibrated Lightweight Models

Abstract

Reliable text classification requires calibrated confidence, abstention for uncertain cases, and explicit computational-cost reporting. This study evaluates cost-aware selective prediction for calibrated lightweight text classifiers on commodity CPU runtimes. The goal is to find a light-weight classifier that is both deployment friendly and achieves a balance between predictive accuracy, calibration, selective reliability, and computational efficiency. The pipeline trains Multinomial Naive Bayes, Logistic Regression, and LightGBM on TF-IDF features, applies post-hoc probability calibration, and accepts only predictions above a validation-selected confidence threshold. Calibrated Logistic Regression achieves the best practical accuracy-calibration-selective-risk-latency-review-budget trade-off in SMS spam, IMDb sentiment and AGNews topic classification experiments. Calibrated Logistic Regression achieves 0.9857 accuracy and 0.0070 ECE on SMS, 0.8718 accuracy and 0.0162 ECE on IMDb, and 0.8998 accuracy on AGNews. On IMDb, Logistic Regression sacrifices about one accuracy point relative to LightGBM but reduces p95 CPU latency from 2546.06 ms to 3.26 ms. Abstaining on the lowest confidence predictions moves accepted risk down from 0.0143 to 0.00457 (SMS) and 0.1282 to 0.1006 (IMDb). Thresholds are chosen on validation/calibration data, then frozen for a single test run. More generally, the results establish calibrated lightweight classifiers as strong and transparent baselines for cost-effective low-latency text classification in real-world deployments.

GS Cited Analysis