[ASR] Support for transcription of multi-channel audio for AED models

[anteju]

What does this PR do ?

Currently, AED models do not use channel selector.
If the input manifest is pointing to multi-channel audio files, transcription will fail with

AssertionError: Expected MonoCut.

This PR adds support for

1. setting channel_selector to an integer value, e.g., 0/1 to always select the first/second channel in the input file
2. setting channel_selector to a string value to use a field from the input manifest to select the channel

Collection: ASR

Changelog

• Propagate channel_selector in _setup_transcribe_dataloader
• Added map with _select_channel in nemo/collections/common/data/lhotse/dataloader.py

Usage

Example of use, assuming Canary model is loaded

channel_selector = 0 # use the first channel
channel_selector = 1 # use the second channel
channel_selector = 'reference_channel' # use 'reference_channel' field from the manifest file

predicted_text = canary_model.transcribe(
      mc_manifest, # manifest with multi-channel audio
      batch_size=1,  # batch size to run the inference with
      channel_selector=channel_selector,  # channel
)

Similarly, it can be used with transcribe_speech.py

CHANNEL_SELECTOR=0
CHANNEL_SELECTOR=1
CHANNEL_SELECTOR=reference_channel # use 'reference_channel' field from the manifest file

python ${NEMO_ROOT}/examples/asr/transcribe_speech.py \
    pretrained_name=nvidia/canary-1b \
    dataset_manifest=./mc_manifest.json \
    channel_selector=${CHANNEL_SELECTOR}

Toy Example:

Scripts & toy dataset:

• pr_example_01.tar.gz

Run as

# test transcribe_speech.py
bash test_transcribe.sh 0
bash test_transcribe.sh 1
bash test_transcribe.sh reference_channel

# test python API
python test_transcribe.py --channel-selector 0
python test_transcribe.py --channel-selector 1
python test_transcribe.py --channel-selector reference_channel

Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment jenkins on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to Support for multi-channel audio data #8728

[anteju]

jenkins

[pzelasko]

Looking good! Left two comments.

[pzelasko]

We should leave the default behavior of .map here, otherwise it might try to apply this to text-only examples.

Suggested change

	cuts = cuts.map(partial(_select_channel, channel_selector=config.channel_selector), apply_fn=None)
	cuts = cuts.map(partial(_select_channel, channel_selector=config.channel_selector))

[anteju]

Done.

[pzelasko]

hmmm .with_channels is only defined on MultiCut, perhaps we should add a check like:

if cut.num_channels == 1: 
    return cut
else:
    return cut.with_channels(channel_idx)

WDYT?

[anteju]

Makes sense, added this (will push in a bit).

[pzelasko]

LGTM

[anteju]

jenkins