176
Scored queries
32 petrol-or-diesel queries, 64 normal-or-abnormal industrial-machine queries, and 80 normal-or-anomalous toy-car queries.
Audio embedding benchmark
Benchmark results for audio embeddings.
Each model converts the same 10-second waveform into an embedding. The evaluation measures whether nearest-neighbor retrieval returns reference clips with the same dataset label as the query.
01Held-out query
A 10-second clip excluded from the reference gallery for evaluation.
02Frozen encoder
BEATs, PANNs, or CLAP processes the same waveform.
03Vector representation
The encoder produces 512–2,048 numeric dimensions.
04Nearest-neighbor retrieval
Cosine similarity ranks the reference gallery.
05Metric calculation
Retrieval metrics compare returned labels with the query label.
Benchmark scope
Retrieval performance is not diagnostic performance.
BEATs had the highest mean average precision in this fixed benchmark. No task contained named real-vehicle fault classes, so these results do not estimate component-level fault discrimination.
120
Eligible references
Each query searched only its task’s held-out reference set: 24 fuel references, 48 industrial-machine references, or 48 toy-car references.
0
Named vehicle-fault classes
No scored label represented a specific component fault. The embeddings retrieved only the broad dataset classes present in the reference gallery.
Uncertainty
Observed lead is benchmark-specific.
BEATs’ mean average precision was 0.015 above CLAP and 0.022 above PANNs. A paired query-level bootstrap places those differences at approximately 0.004–0.027 and 0.007–0.037. The estimate is conditional on 176 queries, limited machine or vehicle identities, one deterministic split, coarse labels, and no named vehicle-fault classes.
Measured on the same 360 clips
Observed ranking performance across models.
mAP evaluates the full ranking. Recall@5 measures top-five retrieval hit rate.
The AUROC values use the same normalized-score evaluation for all models. The tested three-model fusion assigned BEATs 25%, CLAP 50%, and PANNs 25% of the normalized result score. It combines rankings; it does not average incompatible embedding vectors.
Raw float32 vectors only
Storing all three models adds 9.54 GiB per million clips.
This estimate excludes audio, labels, database rows, backups, and the search index.
4.3×
All three vectors use 13,312 bytes per clip. BEATs alone uses 3,072 bytes. CLAP and PANNs add 10,240 bytes per clip.
| Stored clips | BEATs only | All three | Additional storage |
|---|---|---|---|
| 360 — this test | 1.05 MiB | 4.57 MiB | 3.52 MiB |
| 10,000 | 29.30 MiB | 126.95 MiB | 97.66 MiB |
| 100,000 | 292.97 MiB | 1.24 GiB | 976.56 MiB |
| 1,000,000 | 2.86 GiB | 12.40 GiB | 9.54 GiB |
- BEATs
- A transformer that learns sound representations from time-and-frequency features.
- PANNs / CNN14
- A family of pretrained audio models; the 14-layer convolutional variant is evaluated here.
- CLAP
- An audio-and-text model that maps recordings and text descriptions into a shared representation space.
- Frequency bands
- Ranges from low to high pitch. Models inspect how energy in these ranges changes over time.
- CNN
- A model that detects local patterns and combines them into larger sound representations.
- Transformer
- A model that relates parts of a recording even when they are separated in time.
- PCA map
- A two-dimensional projection of a higher-dimensional vector space. It supports overview, not exact retrieval measurement.
- Three-model score fusion
- Each model searches separately; normalized result scores are combined with predefined weights.
Manual retrieval review
Review each query and its nearest reference matches.
The green edge indicates agreement between the reference label and the query label. This is label agreement, not component-level ground truth.
Similarity scores are comparable only within the same model and query. A CLAP score of 0.90 is not inherently greater evidence than a BEATs score of 0.80.
Vector-space projections
Each encoder produces a different neighborhood structure.
These maps project hundreds or thousands of embedding dimensions into two using PCA. Proximity is model-specific; the axes have no physical interpretation.
Application requirements
Vector retrieval is one component of a diagnostic workflow.
Deployment would also require signal-quality validation, vehicle context, calibrated decision thresholds, conservative output, and safety escalation rules.
Validate the signal
Reject wind, speech, clipping, excessive silence, and insufficient level before vectorization.
Retrieve reference evidence
Return multiple labeled neighbors with source context and similarity, rather than an unexplained single class.
Apply a decision policy
Combine acoustic evidence with vehicle context, operating condition, symptoms, and safety rules.
Recommendation
Do not deploy a fault retriever from this benchmark.
CLAP showed modest fault-family ranking signal at 0.190 clip-weighted mAP, just above the random-permutation 95% interval of 0.138 to 0.180. Top-1 accuracy remained 10%, and neither fusion policy improved on CLAP alone. The next investment should be independently recorded, mechanic-reviewed vehicle evidence.
Vehicle-fault follow-up
Proxy-benchmark performance did not transfer to vehicle-fault retrieval.
The follow-up contains 191 clips from only 48 independent content sources. Results are weak-label retrieval diagnostics, not diagnostic accuracy estimates.
Regression only
360 proxy clips
Fuel type, industrial anomaly, and toy-car anomaly proved the pipeline runs. They did not predict fault-family retrieval quality.
Tested · research quarantine
VAFD · 175 segments
Thirty-four named recordings reduce to 33 unique content sources: two differently labeled exhaust sources contain identical audio. The repository has no declared license and labels remain unverified.
Tested · licensed weak labels
15 external sources
Four reference source groups and eleven independent queries tested cross-source transfer. No-start transferred consistently; rough-running and wheel-bearing showed narrower signal.
Observed result
CLAP shows ranking signal; no model is reliable.
- Clip-weighted mAP: BEATs 0.175, CLAP 0.190, PANNs 0.160. Five thousand random permutations averaged 0.158.
- Top-1 accuracy: BEATs 7.5%, CLAP 10.0%, PANNs 3.8%. Random permutations averaged 9.0%.
- Fusion: equal three-model score fusion reached 0.178 mAP; the proposed 60/20/20 BEATs-led blend reached 0.180. Neither improved on CLAP alone.
Remove split leakage
All cuts from one source stay together, and exact duplicate content is treated as one source.
Test real fault labels
Ten audible families were evaluated with all three frozen models and two fusion policies.
Collect confirmed pairs
Record the same vehicle before and after a mechanic-confirmed repair, with known operating condition and microphone placement.
Validate sound events first
Score knock, squeal, rattle, whine, and no-start recognition separately from claims about their mechanical cause.
Require abstention
Only retrieve evidence after the reviewed set demonstrates useful cross-vehicle transfer; otherwise return recording guidance and a cautious handoff.
Evaluation requirements
Report three separate performance questions.
- Recording validation: false rejection of clean clips and false acceptance of speech, noise, clipping, quiet, or incomplete clips.
- Fault retrieval: top-1, top-3, and full-ranking accuracy by reviewed fault label, with confidence intervals based on source groups rather than individual segments.
- End-to-end result: whether the returned description matches the retrieved evidence, whether unsafe certainty is avoided, and whether a human reviewer considers the next check useful.