ENH: Lazy tifffile

[scottclowe]

Add a new datahandler, DataHandlerTifffileLazy, which uses tifffile to handle TIFFs but does so in a lazy manner, loading the data frame-by-frame when determining the traces (like DataHandlerPillow).

The advantage over DataHandlerPillow is it can handle more dtypes and shapes in the TIFF files.

This replaces DataHandlerPillow as the lowmemory_mode datahandler.

[codecov]

Codecov Report

Merging #179 (1d69acc) into master (fd9d282) will increase coverage by 0.17%.
The diff coverage is 95.91%.

@@            Coverage Diff             @@
##           master     #179      +/-   ##
==========================================
+ Coverage   92.65%   92.83%   +0.17%     
==========================================
  Files           8        8              
  Lines         831      879      +48     
  Branches      165      175      +10     
==========================================
+ Hits          770      816      +46     
- Misses         31       32       +1     
- Partials       30       31       +1     

Flag	Coverage Δ
unittests	92.71% <95.91%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
fissa/extraction.py	95.86% <95.83%> (-0.02%)	⬇️
fissa/core.py	96.22% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd9d282...1d69acc. Read the comment docs.

[swkeemink]

One important potential bug.

We should also clarify in the explanation of the low_memory option in the notebooks that it only applies for loading tiff files which might be too high dimensional, and won't work for the option we provide of just inputting arrays instead of tifffiles (if that's not already done properly).

(Incidentally perhaps we should have a separate dataloader for arrays, and detect in the beginning if the provided images are tiffs or strings, and then load the appropriate data loader.)

[swkeemink]

The current function can only take in a TiffFile object, no arrays.

Example:

import tifffile
import numpy as np
image = np.random.rand(10, 10,10)
tifffile.TiffFile(image)

will give an error.
(Specifically, TiffFile can take as a first argument 'a file name, seekable binary stream, or FileHandle')

[scottclowe]

One important potential bug.

It's not a bug, it's just the docstring is wrong. Like the Pillow loader, lazy loader doesn't support a list of file names.

We should also clarify in the explanation of the low_memory option in the notebooks that it only applies for loading tiff files which might be too high dimensional, and won't work for the option we provide of just inputting arrays instead of tifffiles (if that's not already done properly).

It isn't only for TIFF files which are high dimensional. It's for any (large) TIFF files.

(Incidentally perhaps we should have a separate dataloader for arrays, and detect in the beginning if the provided images are tiffs or strings, and then load the appropriate data loader.)

We could. Although you would want to check which dataloader to use for each argument in the list, since there is no guarantee the list is all file name strings or all arrays (it could be a mixture). I don't think this is worth prioritising.

[swkeemink]

We could. Although you would want to check which dataloader to use for each argument in the list, since there is no guarantee the list is all file name strings or all arrays (it could be a mixture). I don't think this is worth prioritising.

Agreed fully, was more of a side-note.

With the docstring update happy with this PR otherwise.

[scottclowe]

With the docstring update happy with this PR otherwise.

Great! Thanks for spotting the inconsistency between docstring and behaviour.