Drop blosc

[mrocklin]

Blosc is somewhat faster than lz4, but not hugely so. It also creates more than its fair share of version mismatch issues. I suspect that more people would be happy than sad if we dropped the use of blosc by default.

Thoughts?

[jrbourbeau]

Based on you comment I think I know, but just to double-check, lz4 isn't prone to the same types of version mismatch issues?

cc @jakirkham who may have thoughts on this topic

[mrocklin]

We handle lz4 in the comm handshake. Blosc is hard-coded into a couple of serialization functions I think, it tends to cause more problems. We can also improve how we handle blosc, but I'm not sure anyone has ever really complained about compression over the wire that much, so it may be premature optimization.

…

On Wed, Aug 25, 2021 at 8:37 PM James Bourbeau ***@***.***> wrote: Based on you comment I think I know, but just to double-check, lz4 isn't prone to the same types of version mismatch issues? cc @jakirkham <https://github.com/jakirkham> who may have thoughts on this topic — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5269 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBLSHW5LDHFVP7I5KLT6WLHLANCNFSM5C2HCJMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[jakirkham]

What's the motivation for dropping? What are the issues? We've had this for a long time and don't recall any notable issues. Kind of seems out-of-the-blue

[mrocklin]

It is a frequent cause of version mismatch errors. Blosc is not a commonly installed library. It needs to be consistently installed or not installed. But, more broadly, if it isn't providing significant value then we should be defaulting to removal I think. My sense is that there are few real-world workloads today that would benefit strongly from blosc over lz4. I would be happy to see evidence to the contrary though.

…

On Wed, Aug 25, 2021 at 9:32 PM jakirkham ***@***.***> wrote: What's the motivation for dropping? What are the issues? We've had this for a long time and don't recall any notable issues. Kind of seems out-of-the-blue — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5269 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTEHLYKLLUDH6JI6E23T6WRSBANCNFSM5C2HCJMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[jakirkham]

Blosc is already something we use with Zarr (currently vendored) and may start using as an independent package ( zarr-developers/numcodecs#262 ). At which point users will have Blosc installed if they use Zarr.

The novel features of Blosc is doing partial chunk reads and type aware compression. This may or may not be relevant to us depending on how we handle the different memory issues we are currently trying to address in communication/serialization/spilling.

That said, can appreciate the desire to lighten dependencies. Also on GPUs we will need something different anyways. Maybe an alternate path forward would be to allow compression to be configureable in a compression agnostic way by outside users. We are close to that now, but not quite.