NaN filing between partitions with overlap
Created by: zain-sohail
Created a new branch because the one in PR #162 has too many other comments. I think after adding a test, this is good to go.
Changes from the other PR:
- removed progress bar as it has a larger overhead than just calling len
- removed the compute command because len directly gives the metadata access to the num_rows.
To be done:
adding tests with multiple files
-
partial data -
one fully nan file (cols) -
consective empty files
last one might fail so we would need null metadata for a more comprehensive treatment https://github.com/OpenCOMPES/sed/pull/162#discussion_r1350242308
also might make sense to keep this as a seperate function or even a generic method that other loaders can use as I think it's pretty useful