OVMI

Research paper · live benchmark

Measuring Progress in
Speech Brain–Computer Interfaces

OVMI: A common scale for comparing how much speech BCIs can communicate.

Dulhan Jayalath Benjamin Ballyk Oiwi Parker Jones

PNPL🍍, University of Oxford

Paper Code + package Blog Benchmark

Speech-BCI studies use different vocabularies, datasets, and communication domains, making reported scores hard to compare. Open-Vocabulary Mutual Information (OVMI) evaluates each system against an explicit, common reference communication distribution, so otherwise heterogeneous systems can be placed on the same communication scale. Choose a communication target below to compare published systems.

01 / Benchmark

OVMI Benchmark

Reference communication distribution
Selected entropy 9.77 bits

Broad spoken-English frequency norm derived from British film and television subtitles.

A vocabulary covers different amounts of language under different distributions. Select a reference to update the scores.

Example. The 50-word Willett +LM system carries 6.4% of the lexical information in broad spoken English and 40.4% under the AAC reference.

At a glance

Published systems on the selected communication scale

The vertical position is OVMI / H(p); the horizontal position is publication year. Each panel uses its own y-axis scale.

Attempted speech / invasive Perceived speech / non-invasive
Study group
Speech-BCI operating points evaluated using the selected communication reference.
System Year Setting / modality Vocabulary Reported metric Comparison OVMI bits OVMI bits OVMI / H(p) Uncertainty Source
Loading benchmark data…

02 / Interpretation

How to interpret the benchmark

OVMI puts systems on a common communication scale, but it does not make their underlying experiments equivalent. Studies differ in task, speech type, participants, recording modality, data quantity, and language-model use. A higher OVMI therefore means that a reported system conveys more lexical information relative to the selected reference distribution; it does not mean that its task or recording paradigm is more useful, practical, or clinically relevant. For example, strong performance on perceived speech may reflect good lexical decoding without corresponding to a practical communication interface.

03 / Definition

Why OVMI?

Speech-BCI studies use different vocabularies and communication domains, making their reported scores difficult to compare. OVMI evaluates each system against a common reference communication distribution.

OVMI multiplies the speech BCI's vocabulary coverage of the reference distribution by its in-vocabulary information transfer.

IOVMI(S) = C(S) × I(X;Y | X∈S)
S
supported decoder vocabulary
p
reference communication distribution
C(S)
probability that a word drawn from p lies in S
I(X;Y | X∈S)
information transferred about the intended word, given that it lies in S

04 / Reproduce

Use OVMI

Install from GitHub:

pip install git+https://github.com/neural-processing-lab/OVMI.git

Pass a reference distribution, supported vocabulary, and macro accuracy:

from ovmi import ovmi

reference = {"yes": 120, "no": 80, "water": 20}
vocabulary = ["yes", "no"]

score = ovmi(reference, vocabulary, accuracy=0.70)
print(score)

Package documentation and source →

Citation

Cite this paper

The repository does not yet contain the final arXiv identifier. Replace PLACEHOLDER when it is assigned.

@article{jayalath2026ovmi,
  title={On the Problem of Measuring Progress in Speech Brain--Computer Interfaces},
  author={Jayalath, Dulhan and Ballyk, Benjamin and Parker Jones, Oiwi},
  journal={arXiv preprint arXiv:PLACEHOLDER},
  year={2026}
}