Add change_vocabulary and save_tokenizers() support to Multitask ASR models

[titu1994]

What does this PR do ?

Adds support to change vocabulary of multi task models.
Adds support to save the tokenizer directory of all ASR models that implement ASRBPEMixin.

Collection: [ASR]

Changelog

• Adds utility method save_tokenizers() that extracts all the internal tokenizers of a model into the specified directory.
• Adds support method to change tokenizers of a multi task model.

Usage

from nemo.collections.asr.models import EncDecMultiTaskModel
asr_model = EncDecMultiTaskModel.from_pretrained("xyz")

# Save tokenizers - this creates a new directory at pwd called `all_tokenizers`
# Then it populates it with all the tokenizer objects stored in the model.
asr_model.save_tokenizers('./all_tokenizers/') 

# Change model vocabulary
tokenizer_cfg = OmegaConf.create(asr_model.cfg.tokenizer)
with open_dict(tokenizer_cfg):
    tokenizer_cfg.pop('langs', None) # remove old `langs` part of config from aggregate tokenizer

    # create new langs config
    tokenizer_cfg.langs = OmegaConf.create(dict(
        spl_tokens=dict(
            dir='all_tokenizers/spl_tokens/',
            type='bpe',
        ),
        en=dict(
            dir='all_tokenizers/en/',
            type='bpe',
        ),
        de=dict(
            dir='all_tokenizers/de/',
            type='bpe',
        ),
    ))

# Change the Multi task models tokenizers and vocabulary
asr_model.change_vocabulary(tokenizer_cfg, new_tokenizer_type='agg')

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

[titu1994]

jenkins

[pzelasko]

LGTM

[nithinraok]

why : after nemo

[titu1994]

Nemo files modify config of registered items like this to denote registered artifacts.

[nithinraok]

why number 8 here and not another int?

[titu1994]

This is from the original code. @krishnacpuvvada