Explicit Consent & Session Start
User initiates session with full transparency on privacy rules, data processing, and recording bounds.
Canonical Runtime Architecture
Synchronized Audio-Visual Pipeline with Dynamic Quality Gating.
NDMS processes facial video and voice audio in real time. The core production pipeline requires zero wearables or EEG hardware, ensuring privacy, accessibility, and high performance on edge devices.
Modality Boundary
Core Runtime Modalities vs. Research Components
Clear separation prevents user confusion between consumer software requirements and laboratory validation setups.
Core Production Modalities (Required at Runtime)
Facial Video + Voice Audio
- Synchronized Video Stream: On-device facial landmarking, micro-expression tracking, zygomatic pose, periorbital dynamics.
- Synchronized Audio Stream: Vocal prosody, pitch contour, spectral energy, pause duration, acoustic tremor.
- Edge Execution: Designed so raw audiovisual data can remain on-device by default; transmission is not required.
- No Special Hardware: Operates on standard webcams, smartphones, and laptops.
Research & Optional Support Layer
EEG Teacher & Self-Report Annotations
- EEG Laboratory Validator: Used in research environments as a physiological reference validator.
- Teacher-Student Distillation: EEG acts as a "teacher" model during training to teach the audiovisual "student" model.
- User Self-Report: Optional user reflection and mood tagging; not required for core model inference.
- Wearables/HRV: Optional future research metadata, strictly non-mandatory.
Technical Pipeline
The 14-Step Multimodal Architecture
How NDMS transforms raw audio-visual streams into dynamic, calibrated masking risk signals.
Synchronized Video + Audio Capture
Parallel acquisition of high-frame-rate facial video and high-sample-rate audio feeds with timestamp synchronization.
Local Facial Feature Extraction
Real-time landmark tracking, facial action unit (AU) intensity estimation, and micro-expression frame isolation on-device.
Local Voice & Prosody Extraction
Acoustic feature extraction including pitch contour (F0), formant frequencies, vocal energy, and speech pause cadences.
Temporal Alignment Engine
Mapping disparate audio (e.g. 16kHz) and video (e.g. 30/60fps) sample rates onto unified sliding temporal windows.
Modality Signal Quality Estimation
Continuous evaluation of ambient lighting, facial occlusion, acoustic noise floor, and signal-to-noise ratio (SNR).
Quality-Conditioned Dynamic Gating
Adaptive weighting mechanism adjusting facial vs vocal weights dynamically. If lighting degrades, the model reduces video weight and relies more on audio.
Temporal Multimodal Fusion
Cross-attention fusion network modeling short-term and long-term temporal dependencies across visual and vocal channels.
Cross-Modal Discordance Modeling
Detecting mismatch between outward facial posture (e.g. forced smile) and vocal prosodic indicators (e.g. flat pitch or acoustic strain).
Calibrated Masking Risk Index (0–1)
Computing a probability score reflecting model evidence of social masking effort in the active temporal window.
Latent / Suppressed Emotion Indicators
Deriving candidate suppressed affective state signals incongruent with displayed facial expression.
Interpretable Facial Heatmap Saliency
Generating visual overlays highlighting facial muscle regions (e.g. periorbital vs lower jaw) driving the masking detection.
Uncertainty & Context Information
Quantifying prediction confidence bounds (e.g. ±0.05) and displaying 'Human Context Required' safety cues.
Personal Baseline & Human Feedback Loop
Updating within-person baseline parameters with optional user reflection to continuously refine individual accuracy over time.
Modality Robustness
Quality-Conditioned Dynamic Gating & Fallback
Real-world environments are noisy. NDMS avoids brittle fusion by dynamically evaluating signal quality.
1Lighting Occlusion FallbackWhen lighting drops or face is turned, video weight automatically decreases while vocal prosody weight increases.
2Acoustic Noise FallbackIn noisy ambient settings, audio weight is suppressed while visual micro-expression analysis takes precedence.
3High Uncertainty GuardrailsIf both modalities suffer severe degradation, the system marks output as 'High Uncertainty' rather than generating false predictions.