Nidhi Mishra

Motivation

Multi-modal models integrate heterogeneous inputs — text, images, audio, and video — within a single architecture, yet the quality of these inputs is rarely uniform at inference time. When one modality is degraded while others remain intact, models often fail to reduce their dependence on it, and the degraded input can dominate the fused prediction; the presence of multiple modalities does not by itself confer robustness. As retraining deployed models is frequently infeasible, interventions that operate at inference time, without parameter updates, are of particular interest. Existing approaches, however, either alter a model's internal weighting without changing its predictions, or are tailored to one model and do not transfer to architectures that fuse modalities differently (e.g., Qwen-Omni, Uni-MoE-2, VITA, Phi-4).

Goal

This thesis develops and evaluates a lightweight, inference-time mechanism that conditions multi-modal fusion on modality-specific reliability estimates — strongly enough to affect downstream predictions, and transferably across different fusion architectures. The specific scope can be adapted to the student's interests, for example toward:

• Artifact development (design science): a transferable intervention that injects reliability estimates at the loci governing fusion (attention weights, value representations, gating/routing logits), with adjustable strength; or
• Empirical study: a systematic evaluation, across model families and fusion designs, of when and why reliability-weighted fusion improves robustness.

Expected Contribution

The thesis is expected to contribute:

• a lightweight, transferable intervention artifact with architecture-specific adapters,
• empirical evidence on the locus and magnitude at which reliability conditioning affects downstream predictions,
• a methodology for establishing that observed effects derive from the reliability estimate rather than from incidental numerical perturbation, and/or
• a cross-architecture assessment of the generalizability of reliability-weighted fusion.

Methodological Approaches

Possible methods include:

• design science research,
• prototype development and evaluation,
• benchmark-based experimentation,
• controlled input-degradation and robustness testing,
• ablation and sensitivity analysis,
• comparative evaluation across model architectures.

Requirements

The topic is suitable for students with an interest in one or more of the following areas:

• multi-modal AI, transformer architectures, and model robustness,
• open model internals and inference-time interventions,
• design science research and prototyping,
• experimental and quantitative evaluation.

Solid programming skills in Python are expected. Prior experience with PyTorch and Hugging Face, and familiarity with modifying model inference code, is beneficial. Depending on the selected option, either a stronger technical or a stronger empirical profile is a good fit.

Contact

If you are interested, please send a current transcript of records, a short CV, and a brief motivation (2–3 sentences) to nidhi.mishra@kit.edu. Students interested in this topic are welcome to reach out to discuss possible focus areas and methodological fit.

Literature

Suggested starting points include:

• Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems 30 (2017).
• Baltrusaitis, Tadas, Chaitanya Ahuja, and Louis-Philippe Morency. "Multimodal machine learning: A survey and taxonomy." IEEE Transactions on Pattern Analysis and Machine Intelligence 41.2 (2019): 423-443.
• Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019).
• Zhang, Yu, et al. "Evaluating and Steering Modality Preferences in Multi-modal LLMs." Forty-third International Conference on Machine Learning. 2026.
• Bompilwar, Ritik, and Saurabh Koshatwar. "QualiVision: Multi-Modal Video Quality Assessment with Quality-Aware Fusion and Discriminative Learning Strategies." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.