Remove cdo/python-cdo from fre-cli, add xarray time averager#830
Remove cdo/python-cdo from fre-cli, add xarray time averager#830
Conversation
…ion warnings, remove from required deps Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/83cd5af3-5071-4657-ba27-a32d87e1ded5 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/83cd5af3-5071-4657-ba27-a32d87e1ded5 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/83cd5af3-5071-4657-ba27-a32d87e1ded5 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
cdo/python-cdo use in fre-cli
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #830 +/- ##
==========================================
+ Coverage 84.12% 84.29% +0.16%
==========================================
Files 71 72 +1
Lines 4944 5016 +72
==========================================
+ Hits 4159 4228 +69
- Misses 785 788 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
|
@copilot let's change this from "deprecate" to remove, i slightly misunderstood the meaning in the context of software. also add another mode of time averaging and climatology generation using xarray. tests please. and keep the pkg= |
…er, update tests and pylintrc Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/1c5f6605-f239-4dd0-94b7-53b5458889fd Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Pushed in facdfba:
Known issue: the weighted time-average path ( |
|
@copilot fix unit test failure RE |
…er weighted average Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/1fb856bc-30a4-4be7-bb77-b2f42b051266 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Fixed in a77657e. There were two root causes:
|
|
@copilot we need coverage on the xarray time averager. more unit tests! |
…ghted seas/month parametrized cases Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/ac9c29b8-e26b-4463-804e-8f22154798d3 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
…ix docstring Agent-Logs-Url: https://github.com/NOAA-GFDL/fre-cli/sessions/ac9c29b8-e26b-4463-804e-8f22154798d3 Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Added in 8250249. New
Also added the missing weighted |
|
ready for a |
|
a |
|
ilaflott
added
new functioning
This CI failure is not caused by this PR. The The root cause: Fix needed on the |
|
a full, successful test with |
|
long winded- i mean, this PR requires NOAA-GFDL/fre-workflows#222 |
Describe your changes
CDO/python-cdo has always been an awkward fit — it doesn't handle
calendar='julian'well and adds a heavy external dependency. This PR removes CDO entirely and adds an xarray-based time averager as a replacement:cdoTimeAverager.pyis now a thin stub that emits a loudFutureWarning("CDO/python-cdo has been REMOVED") and delegates all work to the newxarrayTimeAveragerxarrayTimeAveragerclass (xarrayTimeAverager.py): xarray-based time averaging and climatology generation supportingavg_type='all','seas', and'month'(both weighted and unweighted)frepytoolsTimeAverager→NumpyTimeAverager: the class infrepytoolsTimeAverager.pyis nowNumpyTimeAveragerwith the old name kept as a backward-compatible aliaspkg='cdo'entry point preserved but warns loudly viaFutureWarningand routes to xarray under the hood — users are directed to switch topkg='xarray'orpkg='fre-python-tools'pkg='xarray'as a new CLI choice in bothgen-time-averagesandgen-time-averages-wrappergenerate_time_averages.py:Cdo().mergetime()→xr.open_mfdataset()(this was used by all packages, not just CDO)frenctoolsTimeAverager.py:Cdo().select()→xr.open_dataset().sel(time=ds['time'].dt.month == idx)cdo/python-cdofrom required dependencies inenvironment.yml,meta.yaml,pyproject.tomlclass-rgxregex to allow mixedCase class names (e.g.xarrayTimeAverager,cdoTimeAverager) alongside PascalCasexarrayTimeAverager:_compute_time_weightsnow robustly handles all three cases xarray produces fortime_bndsdifferences:timedelta64(converted viaastype('timedelta64[s]') / 86400), cftimeobjecttimedeltas (using.days + .seconds/86400), and numeric float/int (direct cast) — using numpy.valuesinstead of the unreliable xarray.dtaccessor_is_numeric()guard in all weighted helper functions to skip non-numeric time-dependent variables (e.g.average_T1/average_T2decoded asdatetime64by xarray for JULIAN-calendar files) that cannot be multiplied by float weightstest_xarrayTimeAverager.py(53 tests across 6 test classes):TestIsNumeric— dtype classification helper for float, int, uint, datetime64, timedelta64, objectTestComputeTimeWeights— all threetime_bndsencoding paths (float, timedelta64, no-bnds fallback) plus known-value correctness checksTestWeightedTimeMean— numerical correctness, dim elimination, attribute preservation, non-numeric variable handlingTestWeightedSeasonalMean— season grouping, MAM/JJA/SON weighted values verified against known formulasTestWeightedMonthlyMean— month dim, 12-month output, 2-year weighted mean correctnessTestXarrayTimeAveragerGenerateTimavg— full NetCDF round-trip tests for all/seas/month × weighted/unweighted, per-month file creation, ValueError path, input file not clobberedseasandmonthcases (single-file and multi-file) totest_generate_time_averages.pytest_cdoTimeAverager.pyupdated to verifyFutureWarningis emittedIssue ticket number and link (if applicable)
Checklist before requesting a review