Joshua Holstein

Background

Artificial intelligence systems, particularly Neural Networks (NNs), are frequently deployed in environments where data is imperfect, noisy, or entirely novel. In such contexts, quantifying a model's "uncertainty" is critical for safety and efficiency. However, a fundamental challenge remains in distinguishing between two distinct sources: Aleatoric Uncertainty (AU), which arises from inherent randomness or noise in the data (e.g., blurry images), and Epistemic Uncertainty (EU), which stems from a lack of model knowledge about a particular domain.

In the field of Active Learning (AL), the goal is to selectively query only the most informative data points for human labeling. Standard approaches typically use a monolithic uncertainty measure, often failing on real-world "dirty" datasets because they cannot distinguish between what the model needs to learn (EU) and fundamentally unlearnable noise (AU). While theoretical frameworks for disentangling these uncertainties exist, empirical evidence suggests that "uncertainty leakage" occurs—where aleatoric noise inflates epistemic estimates—leading to suboptimal data selection and degraded model performance.

As demand for data-efficient and robust AI grows, it is essential to determine whether triaging by uncertainty type can improve learning efficiency in noisy environments. A purely computational benchmark is required to evaluate the robustness of modern disentanglement methods and their practical utility in automated data reintegration pipelines.

Research Goal

This thesis presents a computational benchmarking study to evaluate the effectiveness of disentangled uncertainty quantification for data selection in noisy environments.

Research objectives:

• Implement and compare multiple state-of-the-art uncertainty quantification (UQ) methods (e.g., Deep Ensembles, MC-Dropout, and Evidential Deep Learning) to calculate AU and EU components.
• Quantify "Uncertainty Leakage" across different noise levels to determine how accurately current methods can separate noise from novelty in practice.
• Benchmark Triage Strategies (e.g., Monolithic, Epistemic-only, and "Clean-Novelty" filters) using Active Learning loops on datasets with real and synthetic noise (e.g., CIFAR-10N, ImageNet-C).
• Analyze Learning Efficiency by measuring the Area Under the Learning Curve (AULC), identifying the specific noise-to-signal ratios where triage-based selection outperforms traditional monolithic sampling.
• Establish Engineering Principles for the design of automated data re-integration pipelines that are resilient to aleatoric noise.

Your Profile

• Demonstrate an interest in Machine Learning and Trustworthy AI.
• Have solid experience in Python and PyTorch for developing and training deep learning models.
• Seek to contribute to high-performance benchmarking and evidence-based data engineering.
• Experience with (or interest in) Active Learning and Bayesian Neural Networks is a plus.