Add datasets from Embrace Uncertainty and recent SMLPs - #30
Conversation
it seems that the original arrow files sometimes have Float32 instead of Float64 which can mess up some later digits
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #30 +/- ##
============================================
- Coverage 100.00% 86.11% -13.89%
============================================
Files 1 1
Lines 10 36 +26
============================================
+ Hits 10 31 +21
- Misses 0 5 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dmbates
left a comment
There was a problem hiding this comment.
Looks good. Thanks for adding the datasets from Embrace Uncertainty and from the SMLPs. Do we want to use smaller data types for the columns in tables with many rows, like ml32_ratings? I didn't check if the Arrow file is compressed or not. I tend to keep the Arrow file uncompressed, so it can be memory-mapped, and use smaller sized data types, like Float32 instead of Float64
|
I compressed the data from ml32_ratings because it made a really substantial difference in file size, but didn't mess with the eltypes. It might be worthwhile to do something fancy and include special casing for huge datasets where we store them compressed on OSF to reduce download size, but then decompress them locally so that they can be memory mapped. Or maybe provide an option to convert them manually to uncompressed storage locally so that users can opt into larger files on disk. (There have been a few complaints on Discourse about the size of The good news about this new layout / loading mechanism is that it's relatively easy to update things piecemeal. |
I also took this opportunity to copy the pattern that @dmbates has down with using a scratch space and dynamic download of the associated arrow files instead of a monolithic artifact blob. For most users, I suspect this won't make much of a difference though it might slightly slow down CI. That said, I think this is far easier to maintain and add to incrementally than the big blob, so I think it's a net win.
Counting this as a major version bump because apparently a few of the source Arrow tables (especially sleepstudy) have Float32 storage of some variables, which can break type-specific checks and which can lead to small values changes in least significant digits of both the data and statistics computed from them.