NEW: Dependency config design and content#2812
Open
tech3371 wants to merge 68 commits intoIMAP-Science-Operations-Center:dependency_refactorfrom
Open
NEW: Dependency config design and content#2812tech3371 wants to merge 68 commits intoIMAP-Science-Operations-Center:dependency_refactorfrom
tech3371 wants to merge 68 commits intoIMAP-Science-Operations-Center:dependency_refactorfrom
Conversation
tech3371
commented
Mar 4, 2026
| - ``p`` - pointing | ||
| - ``h`` - hourly | ||
| - ``d`` - days | ||
| - ``l`` - last_processed |
Contributor
Author
There was a problem hiding this comment.
similar to last_processed, we need nearest in the past.
But Hi wants nearest 7 irrespective of past or future.
Contributor
Author
There was a problem hiding this comment.
options:
past_nearest, future_nearest, any_nearest.
or
(7p) - this means any future or past data.
(1n, 0n) - this means get me last nearest data from past.
What to do if hi science file event comes and then need to looks up for SWE dependency which is daily?
pointing number 9 file came, we look nearest 7 files, then derive date range using earlier and latest pointing id of that 7 files and look data range from the pointing table. Then use date range to query for SPICE and other dependency. If dependency is found,
Contributor
Author
There was a problem hiding this comment.
one class for daily and pointing. Then parent class can be inherit those as needed for ENA or not.
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
tmplummer
reviewed
Mar 4, 2026
| ~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Primary descriptor can be one of the following: | ||
| Upstream Product Name |
Contributor
There was a problem hiding this comment.
I personally like Descriptor better for this. I think that product name has several possible meanings.
Contributor
Author
There was a problem hiding this comment.
cool. Then let's keep descriptor and I will make remaining changes.
tmplummer
reviewed
Mar 4, 2026
| - For science or ancillary data, the product names are defined by the instrument and SDC. | ||
|
|
||
| - For ``spin`` and ``repoint`` data types, ``historical`` is the only valid descriptor. | ||
| - For ``spice`` data types, ``historical`` and ``best`` are the valid product names. |
Contributor
There was a problem hiding this comment.
What about predict? I specify Hi spice dependencies individually for example:
ephemeris_reconstructed, spice, historical, hi, l1c, 45sensor-pset, HARD_NO_TRIGGER, DOWNSTREAM
ephemeris_predicted, spice, best, hi, l1c, 45sensor-pset, HARD_NO_TRIGGER, DOWNSTREAM
Contributor
Author
There was a problem hiding this comment.
we had intension of using 'best' at one time but may be we didn't enforce it. We can remove that option.
tmplummer
reviewed
Mar 4, 2026
| ~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Same as primary_data_type, but for the dependent file. | ||
| Kickoff_job (Optional) |
Contributor
There was a problem hiding this comment.
My nitpick on terminology: It seems like we use trigger more widely.
Suggested change
| Kickoff_job (Optional) | |
| Trigger_job (Optional) |
tmplummer
reviewed
Mar 4, 2026
| - (imap_frames, spice, historical) | ||
|
|
||
| (l1b, 45sensor-goodtimes): | ||
| - (hi, l1b, 45sensor-de, true, true, (-3p, 3p)) |
Contributor
There was a problem hiding this comment.
One possible way to do this:
(l1b, 45sensor-goodtimes):
# Entry for getting the 7 nearest pointings
- (hi, l1b, 45sensor-de, {required: true, trigger: true, nearest: 7p})
# Entry for getting the past 3 and future 3 pointints, if they exist
- (hi, l1b, 45sensor-de, {required: false, trigger: true, past: 3p, future: 3p})
# Entry for getting the 3 nearest available pointings in the past
- (hi, l1b, 45sensor-de, {required: false, trigger: true, nearest_past: 3p})
* Fix bug causing goodtimes to not filter out non-qualified events * Add unrelated bugfix to cli * precommit and mypy fixes * Fix remaining l1b de with event coord * fix docs build * PR feedback * PR feedback 2 * Add missing hi/utils.py tests
…er#2820) * fix exposure tiem * small bug * fix test * clear up comment
maxinelasp
reviewed
Mar 10, 2026
Contributor
maxinelasp
left a comment
maxinelasp
left a comment
There was a problem hiding this comment.
I agree with Tim's comments. I think the CLI docs in cli.py are also out of date, it might be nice to update them too, but I don't think that needs to happen in this PR.
| downstream or upstream dependencies of a file are, and what the nature of those dependencies | ||
| are, we need some way to request the upstream or downstream dependencies of a given file. | ||
| The current dependencies between instruments are recorded in `sds-data-manager Repo | ||
| <https://github.com/IMAP-Science-Operations-Center/sds-data-manager/blob/dev/sds_data_manager/lambda_code/SDSCode/pipeline_lambdas/dependency_config.csv>`_. |
Contributor
There was a problem hiding this comment.
Fix link - still points to CSV, not to YAMLs
Contributor
Author
There was a problem hiding this comment.
Good catch! I will update this link later when are close to done.
IMAP-Science-Operations-Center#2827) * Change CDF output variables to UTC string, matching algorithm document * Pushing minor change to re-trigger checks
…-Center#2828) * sort of changes * things working sort of * more investigation * triggertime * matchign * idex l1b trigger mode/origin/level fix * remove code * add new validation file to the external data download config * trigger origin update * update code * pr feedback fix comments
…nter#2836) * add aid * fix test
* updated values from GLOWS feedback * fixed valid min/max values
…er#2817) * add 90 efficency code * de get efficiencies call * 90 eff test
…ce-Operations-Center#2826) * updates to variables * ISTP compliant arrays * istp compliant arrays * pr comments * pr comments and logger statements * fixes from running * remove print statements * remove print
…tions-Center#2839) * Fix bug that allowed NaNs to propigate incorrectly when combining ram/anti maps * Ensure that NaN intensities do not contribute to weighted average when combining maps
* Update calibration dataset to use start_time_utc as the coordinate for calibration factors and sort data by time. Update corresponding unit test to reflect changes * Get calibration factor needed for the observational day and pass it to HistogramL2 * Add mock calibration dataset and test function that parses the dataset for the calibration factor needed for flux calculations. Update existing tests where needed to fix broken tests * Divide flux by calibration factor and update existing unit tests where needed * Move filtering of calibration dataset to the HistogramL2 class * Address PR comments - simplify flux calculation logic and fix how mid-epoch is extracted from numpy array * Convert calibration timestamps from numpy strings to datetime64 for safer comparison with l1b dataset epoch * Fix typo in Rayleigh spelling * Add additional comments for clarification
…ations-Center#2928) * add backwards compatibility for old psets * fix check * temporary test for coverage
* bug in downsampling counts * restore check
* ultra l1c fov restriction * fix test path * fix test * remove todo * restored changes that belong in another PR
…ter#2931) * code and validation tests for upstream ion cull * add new arrays to mock goodtimes datasets
* code and validation tests for upstream ion cull * initial code and tests * fix test * pr comments * fix test * PR comments
* Add background config to hi utils Add background counts binning helper to hi utils * Add fixture to conftest * Remove unused accessor functions * Add background config test file * Copilot feedback changes * Fix documentation build * Fix typo in csv comment section * Add proper unit tests for utils iter functions * Clean up Config definitions
* priority l1b des * priority handling at l1b * add l1b priority de handling * test for priority products * fix regex pattern * external test data mark * raise error instead of using first l1a de
* Add background calculation to hi l1c * Add background ancillary dependency to Hi L1C in cli.py * Remove esa_energy_step and spin_angle_bin from calculation of background_counts * Copilot feedback changes * Add external data decorator to new test * Remove external kernel requirement from test * Copilot feedback changes * One more failing test due to external data fix * remove checks for mixed good/bad DEs b/c I see unexpected bad DEs in l1a
* Return structure of calibration xarray dataset to what is needed for the combined_dataset method in the GlowsAncillaryCombiner class * Fix method that gets the calibration factor to correctly parse the structure of the calibration data and fix related tests * Address PR comments - fix filtering to find time values before or equal to the mid-epoch time and clean up comments * Remove datetime64 data type for start_time_utc data var in calibration dataset since they are strings in the real inputs * Remove sorting of time_block dimension
* added spice documentation * replaced dev api endoints in docs with prod api endoints * Update docs/source/data-access/openapi.yml --------- Co-authored-by: Tim Plummer <timothy.plummer@lasp.colorado.edu>
* Initial stab at converting Lo Instrument Team auto-good-times into SDC compatible CDF production code. * goodtimes and unit tests * reverted DE rates * added test for better coverage * increased test coverage * comment fixes * Updated bgrates creation to create a single rate and variance per species for the entire pointing, consistent with the current Ancillary file. Also added logic to handle creation of output files with 0 start and end times when no GoodTimes are detected. Added test case to cover no goodtimes detected and updated other test cases as needed per the other change. Validated output against current ancillary files using flight data for 2026-02-21 through 02-28. * test coverage * added print statements for pipeline debug * potential fix added * another fix for modifying array * epoch metadata print statements * temp fix for epoch meta * skipping ruff branching temporarily --------- Co-authored-by: David Gathright <david.gathright@lasp.colorado.edu>
* small bug * comment update
…-Science-Operations-Center#3011) Co-authored-by: Menlo Innovations - CAVA Project <harrison@menloinnovations.com>
…ter#2998) * priority l1b des * priority handling at l1b * add l1b priority de handling * fix regex pattern * raise error instead of using first l1a de * add in depth tests * comment * add logging statement * add logging statement * fix extendedspin * error handling * comment update * PR comments
* quicklook rst file * fix: Added quicklooks to toctree
…ations-Center#2944) * Update glows_l1b_data.py By default geometry.cartesian_to_latitudinal() uses degrees=True so one needs to set explicitly degrees=False here to use circmean/circstd later with low=-np.pi, high=np.pi bounds * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Finish fixing GLOWS bug where degrees where getting passed to circmean and circstd functions Add specific test coverage checking that spin_axis_orientation is now as expected --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Tim Plummer <Timothy.Plummer@lasp.colorado.edu>
* fix ultra cadence desc * remove comment
tech3371
force-pushed
the
dependency_config_doc
branch
from
3993997 to
2f7939f
Compare
tech3371
force-pushed
the
dependency_config_doc
branch
from
2f7939f to
a0fb2f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Summary
closes IMAP-Science-Operations-Center/sds-data-manager#1151
Overview
PR Doc - https://imap-processing--2812.org.readthedocs.build/en/2812/
File changes
This contains final design of new config file. It contains information such as filename convention, new file content and required/optional fields and defaults used. The part that I need feedback the most is the time range options and the example content.
Testing