About the project
Listening in noisy environments is mentally exhausting, but human physiological responses move at fundamentally different speeds. This project uses multimodal deep learning transformers to combine asynchronous wearable signals (EEG, pupillometry, heart rate, skin conductance and others), creating an AI engine that isolates genuine cognitive listening fatigue from everyday acoustic arousal to power next-generation smart hearables.
The Challenge
Understanding speech in noisy environments demands heavy cognitive effort. However, identifying when a listener crosses from effortful listening into mental exhaustion remains an unsolved challenge. Traditional attempts search for single biomarkers—like pupil dilation, heart rate, or skin sweat. But the human body does not respond in lockstep: eyes react in milliseconds, while autonomic sweat and cardiovascular responses lag by seconds. Standard algorithms struggle to make sense of these misaligned speeds.
The AI Solution
This project reframes the problem as an asynchronous multimodal machine learning challenge. You will design and train multimodal transformer architectures and self-supervised models to fuse disparate physiological time series. The primary goal is to disentangle non-specific sensory arousal (e.g., sudden loud noises) from genuine cognitive listening fatigue (working memory depletion).
Methodology and facilities
You will use the world-class acoustic laboratories and calibrated listening booths at the Institute of Sound and Vibration Research (ISVR). You will work with rich, multi-channel physiological datasets (EEG, pupillometry, ECG, galvanic skin response, fNIRS) collected during speech-in-noise paradigms, building neural networks in PyTorch. The resulting algorithms will provide the foundation for closed-loop hearables and assistive listening devices that automatically adapt their signal processing when the user is struggling.
Training
You will receive comprehensive interdisciplinary training bridging modern machine learning and experimental hearing science:
- advanced technical & AI training: hands-on training in state-of-the-art deep learning architectures (multimodal transformers, self-supervised learning, sequence modelling) using PyTorch, alongside access and training on the University's High-Performance Computing cluster (Iridis)
- experimental and biosignal methods: specialist training in the world-class acoustic facilities of the Institute of Sound and Vibration Research (ISVR), covering calibrated audio reproduction, psychophysical speech testing, ethical human research governance, and synchronous multi-channel physiological data acquisition (EEG, pupillometry/eye-tracking, ECG, galvanic skin response)
- professional and academic development: full access to the Southampton Doctoral College training programme, including research data management, scientific writing, intellectual property, and public dissemination
- industry and network exposure: regular opportunities to present at international conferences (e.g., ICASSP, Interspeech, ISH) and interact with our collaborative network across consumer audio and hearing technology developers.
The School of Engineering is committed to promoting equality, diversity inclusivity as demonstrated by our Athena SWAN award. We welcome all applicants regardless of their gender, ethnicity, disability, sexual orientation or age, and will give full consideration to applicants seeking flexible working patterns and those who have taken a career break.
The University has a generous maternity policy, onsite childcare facilities, and offers a range of benefits to help ensure employees’ well-being and work-life balance. The University of Southampton is committed to sustainability and has been awarded the Platinum EcoAward.
