MCFL — Mel Convolutions For Labeling

English · 中文 · Русский

Fast multilingual phoneme labelling of singing on a CPU. MCFL is a small convolutional network that works directly on the mel spectrogram (no Whisper). It learns to repeat a WFL-ASR model (distillation) and, once trained, weighs 1 to 15 MB. Running it needs only numpy and onnxruntime, so it fits a small server (VPS) or a bot.

The project is inspired by WFL-ASR (Whisper For phoneme Labeling) and follows its format: the same 20 ms frame tags (B/I/O), the same boundary offsets inside a frame, the same Viterbi decoder, the same phonemes shared between languages (merged_phoneme_groups).

What is inside

• Training a WFL teacher: a regular WFL-ASR model (Whisper-base frozen + Conformer, language embedding) on your data. The best epoch is saved as a regular WFL-ASR model folder, usable on its own.
• Training MCFL from that teacher or from any existing WFL-ASR model.
• Export to ONNX (fp32 and int8) and a check of the finished package.
• Inference: one file, runtime/mcfl_infer.py, without PyTorch.

Labelling is without lyrics only: the model finds the phonemes and boundaries itself, in the chosen language. Alignment to a given text is not there yet.

How the model works

audio 16 kHz -> STFT (as a convolution) -> 80 log-mel -> stride-2 convolution (20 ms frames, as WFL)
-> + language embedding -> N blocks: depthwise dilated convolution + 2 pointwise convolutions, residual
-> B/I/O tags per frame + start and end offsets inside the frame

It is all one ONNX graph: audio and language id in, probabilities out. Only the chosen language's phonemes and the shared pauses (AP SP cl vf exh) can come out, with the language's own names.

Training: KL to the teacher's probabilities + 0.5 × cross-entropy to the reference labels + 2 × L1 of the boundary offsets. The teacher runs on the GPU during training, on the same augmented batch, so nothing is precomputed or stored on disk.

Sizes

size	channels	parameters	int8	network speed (1 CPU thread)
xxxs	96	~0.4M	~1 MB	~5× faster than s
xxs	128	~0.8M	~1.5 MB	~2.7×
xs	192	~1.7M	~2.5 MB	~1.7×
s	256	~3.4M	~4 MB	1× (default)
m	384	~7M	~8 MB	~2× slower
l	512	~13M	~14 MB	~3× slower

For reference: s, decoding included, labels about 150–450 seconds of audio per second on one thread of a modern CPU. The WFL teacher (Whisper-base) on the same thread does about 10.

Installation

git clone <this repository> MCFL
cd MCFL
pip install -r requirements.txt          # training (install PyTorch with CUDA separately, for your GPU)
pip install -r runtime/requirements.txt  # inference only

Training needs an NVIDIA GPU. It also works on a CPU, but very slowly. On the first start Whisper-base (~290 MB) is downloaded from Hugging Face.

Data

One folder per language, any sub-folders inside. Next to every .wav there is a .lab (HTK, times in 100 ns):

data/
  ru/  singerA/song1.wav  singerA/song1.lab  ...
  en/  singerB/...
  val/ ru/...  en/...      <- optional: your own validation files

• The first sub-folder inside a language is taken as the voice. Training draws languages and voices in a balanced way.
• Long files are cut at pauses into chunks of up to 26 s. This happens in memory only; nothing is written to disk.
• Without a val/ folder, 5% of each language's files (at least 2) are held out for validation.
• Phonemes shared between languages are set with a file of merged_phoneme_groups in the WFL-ASR format (data.phoneme_groups in the settings). Without it, equal phoneme names in different languages count as one phoneme.

Training

Everything at once: WFL teacher → MCFL → inference package:

python -m mcfl.train --data data --run runs/my_model --size s

On Windows: train.bat --data D:\data --run runs\my_model.

Finished stages are skipped, so after Ctrl+C the same command continues from the last finished epoch.

Result:

runs/my_model/
  wfl/          the teacher's best epoch: a regular WFL-ASR model folder (best.pt, config.yaml, phonemes.txt,
                langs.txt, lang_phonemes.json, phoneme_merge_map.json, encoder/)
  mcfl/         the best MCFL checkpoint (best.pt), metrics.csv, teacher_val.json
  package/      the MCFL model for inference (+ package.zip)

The stages can also be run one by one:

python -m mcfl.train_wfl  --data data --run runs/my_model          # the teacher only
python -m mcfl.train_mcfl --run runs/my_model --size xs            # a student of runs/my_model/wfl
python -m mcfl.export     --run runs/my_model                      # the package

With an existing WFL-ASR model (no teacher training):

python -m mcfl.train --data data --run runs/from_wfl --teacher path/to/wfl_model --size s

The phoneme list and language ids then come from the teacher. Chunks with phonemes or languages the teacher does not know are left out, and the log says so.

What training looks like

Everything in the log is in English. After every epoch there is a per-language table: PER and boundary error in ms. For MCFL the same numbers of the teacher, decoded exactly the same way, are shown next to it. The overall score is the mean over languages of PER + 0.5 × boundary error (lower is better); it picks the best epoch.

The procedure carries the experience of earlier runs, so no manual tuning is needed:

• A short schedule. Warmup, then a cosine down to 2% over 20 epochs, and training ends there. With a long cosine, early stopping fired while the learning rate was still high, before the phase where most of the gain usually comes.
• Early stopping: 5 epochs without an improvement larger than 0.1.
• EMA of the weights: validation and saving use the smoothed weights.
• NaN/Inf guards: a batch with non-finite audio, loss or gradients is skipped and counted, never applied.
• Augmentation: speed 0.9–1.1 (the labels are stretched with the audio), gain, noise at a random SNR.

All settings are in configs/default.yaml. Your own values go into a file passed with --config my.yaml; it only needs what changes.

Inference

pip install -r runtime/requirements.txt
python runtime/mcfl_infer.py song.wav --lang ru -m runs/my_model/package

song.lab appears next to song.wav. The package can simply be copied to a server: it contains mcfl_infer.py itself.

from mcfl_infer import MCFL

m = MCFL("runs/my_model/package", threads=1)   # load once
segments = m.label_file("song.wav", lang="ja")  # [(start_sec, end_sec, phoneme), ...]

Package contents: model.onnx, model_int8.onnx, labels.txt, langs.txt, lang_phonemes.json, phoneme_merge_map.json, config.json (decoding settings picked during export), mcfl_infer.py, requirements.txt. The int8 version is the default only if it is not worse than fp32 on the check chunks.

For a bot on 1–2 cores: one process per core, threads=1, the model loaded in advance. Each process needs a few tens of MB of memory. Long files are processed in 120 s windows with an overlap.

Credits and license

• Inspired by WFL-ASR by Ghin_MLo7. third_party/wfl_asr/model.py is its network, unchanged (MIT, see third_party/wfl_asr/LICENSE). It is used to train the teacher, so the teacher is a regular WFL-ASR model.
• MCFL is released under the MIT license (LICENSE).