grep max-count option

[tolot27]

It would be nice if csvtk grep would support a -m NUM, --max-count=NUM parameter like the usual grep utility. The rational behind this is performance improvements.

I'm repeatedly looking for unique values (stored in a file) in a column of a very large database containing only unique values (primary keys). It takes some time to go over the hole database and I believe if I could provide something like --max-count=1 it would ran faster. But that requires a a map for the querying values to there occurences already found.

Here is the command I use: pigz -dc prot.accession2taxid.gz | csvtk grep -t -f accession.version -P some_accessions.txt | csvtk cut -t -f accession.version,taxid

prot.accession2taxid.gz is several gigabases large.

[shenwei356]

I thought it has a flag like seqkit grep, but it doesn't.

      --delete-matched        delete matched pattern to speedup

It would gain some speed improvement, without using extra data structure.

[tolot27]

Yeah that would cover the most frequent (IMHO) use case of --max-count=1. But it should work for "normal" and regex patterns. In seqkit grep it looks like it is only used for regex patterns.

[shenwei356]

I'll add this weekend

[shenwei356]

It works for "normal" and regex patterns now:

• csvtk_darwin_amd64.tar.gz
• csvtk_linux_amd64.tar.gz
• csvtk_windows_amd64.exe.tar.gz

[tolot27]

Many thanks. I got a slight runtime improvement. Not much because all hits were near the end of the file.