Validation · September 2026 evaluation package
Evidence beyond a single accuracy number.
The current package evaluates the audiovisual engineering foundation across locked testing, grouped development CV, discordance estimation, calibration, predictive uncertainty, ablations, robustness, out-of-distribution screening and runtime behavior. It is not yet a masking-specific or clinical validation study.
50 benchmark iterations · 15,944,693 parameters · server-class measurement, not mobile-device latency.
01
Locked test, actor-level uncertainty, grouped development CV.
The current evaluation package contains 7,438 eligible audiovisual samples from 91 CREMA-D actors. The locked test reports clip-level metrics alongside actor-bootstrap 95% confidence intervals. [R1]
A separate five-fold grouped development CV keeps outer-test actors isolated and yields macro-F1 0.444 ± 0.031. The gap between development CV and the final locked test is shown explicitly rather than omitted.
Discrimination
Performance with the error structure exposed
Accuracy, macro-F1 and balanced accuracy are useful only when class-level errors and uncertainty across actors remain visible.
Scores are reported from the locked audiovisual evaluation package; no metric is presented as clinical performance.
The error structure is intentionally visible: for example, the current sad class shows substantial confusion with fear and neutral.
Bootstrap intervals are computed at actor level rather than treating every clip as independent.
Grouped-CV performance is materially lower than the final locked-test score. Both are shown because generalization should not be reduced to the most favorable number.
Cross-modal target
Discordance is continuous—not a hidden-state label
The current target is perceptual audiovisual discordance. It should not be interpreted as ground truth for masking, sincerity, stress or internal emotion.
95% actor-bootstrap CI: 0.3911–0.5024.
95% actor-bootstrap CI: 0.1632–0.1780.
Continuous prediction error in the current locked evaluation.
Calibration · uncertainty · ablation
Confidence and architecture are tested, not assumed
The evaluation asks whether probability estimates behave sensibly and whether key system components contribute measurable value.
At 90% nominal coverage the observed coverage was 89.52%. Calibration is treated as a release criterion, consistent with modern calibration literature. [R3] [R4]
Removing quality gating, cross-modal interaction or multimodality reduces macro-F1 in the current retrained ablation package.
Robustness
Failure modes are part of the result
Controlled corruption reveals where the current audiovisual foundation is resilient and where it remains dependent on specific modalities.
Missing-video and severe visual-darkening conditions remain major failure modes. The quality-aware architecture degrades gracefully in some perturbations, but it does not eliminate modality dependence. [R2]
Engineering ML release gate
All required checks passed the current internal gate.
This is an engineering release gate—not regulatory certification, clinical validation or external approval. The thresholds are internal criteria used to decide whether the model is mature enough for continued prototype integration.
02
Fast server inference, with edge profiling still ahead.
The benchmark measured 6.751 ms mean inference, 6.784 ms p95, and 148.12 samples/s on an NVIDIA RTX PRO 6000 Blackwell Server Edition. Peak GPU memory was approximately 326 MB for the reported benchmark process.
Open assurance risk
Identity information is still recoverable from the representation.
The current identity-leakage audit recovered actor identity at 36% top-1 accuracy across 57 actor classes, compared with approximately 1.75% chance. That is about 20.5× chance. This is not hidden behind the headline metrics: representation privacy and generalization are active engineering problems.
Scientific boundaries
What this package does—and does not—establish
Not masking-specific validation. The package validates audiovisual behavioral-dynamics engineering components. It does not establish that NDMS measures social masking in a target population.
Grouped CV is weaker than the locked-test headline. Five-fold grouped development CV produced macro-F1 0.444±0.031, which is disclosed because generalization should not be inferred from one split.
SMIC is frame-level only in this export. The supplied flat-preprocessed SMIC data report subject_independent=false and loso_eligible=false; LOSO is unavailable because subject IDs are missing. The very high frame-level score must not be presented as subject-independent performance.
Identity leakage remains open. Actor identity is recoverable at 36% top-1 vs 1.75% chance in the current audit, motivating stronger invariance, privacy and generalization work.
No diagnostic or hidden-state interpretation. The package is explicitly not validated for social masking, autism/ADHD diagnosis, deception/sincerity, stress, fatigue, workload or clinical decision-making.
External evidence remains necessary. Fit-for-purpose data, target-population prospective studies, cross-dataset validation, subgroup analyses and independent replication are future evidence layers.
Next evidence layer
Prototype → dedicated dataset → prospective masking validation.
The next phase is not to inflate benchmark numbers. It is to collect consented, fit-for-purpose longitudinal data with masking-relevant ground truth, then test the complete system prospectively and independently.
Read the research program →Selected references
Research and IP context
References contextualize the dataset, multimodal fusion, uncertainty, calibration and OOD methods. They are not independent validation of NDMS.
Validation partnerships
Help stress-test the system before it scales.
NDMS is open to collaborations around multimodal validation, study design, robustness, calibration, dataset collection, privacy and translational research.