Skip to content

add option to fits reader to open L1 products also in UTC >L1 UTC per default (via keyword in header)#447

Merged
nicHoch merged 6 commits intoi4Ds:masterfrom
nicHoch:TIMESYS_fits_read
Mar 23, 2026
Merged

add option to fits reader to open L1 products also in UTC >L1 UTC per default (via keyword in header)#447
nicHoch merged 6 commits intoi4Ds:masterfrom
nicHoch:TIMESYS_fits_read

Conversation

@nicHoch
Copy link
Collaborator

@nicHoch nicHoch commented Dec 5, 2025

add keyword: get_timeformat_from_TIMESYS to Product(path) method.
If set time and timedel column in data table are converted to timesystem provided in the TIMESYS header keyword.

some products properties like utc_timerange, min_exposure ... might do autoconversions based on the internal timesystem but it should be avoided to open L1 files in UTC for processing them further.

Use get_timeformat_from_TIMESYS=True only for reading excess to L1 data, read/write/combing might lead to timing accuracy issues

@nicHoch nicHoch requested a review from samaloney December 5, 2025 12:34
@nicHoch nicHoch self-assigned this Dec 5, 2025
@nicHoch nicHoch added this to the v1.7.0 milestone Dec 5, 2025
@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2025

Codecov Report

❌ Patch coverage is 81.48148% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.04%. Comparing base (3ae9671) to head (9b9c9dc).
⚠️ Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
stixcore/products/product.py 86.95% 6 Missing ⚠️
stixcore/products/level1/quicklookL1.py 20.00% 4 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (3ae9671) and HEAD (9b9c9dc). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (3ae9671) HEAD (9b9c9dc)
2 1
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #447      +/-   ##
==========================================
- Coverage   72.23%   67.04%   -5.19%     
==========================================
  Files          78       76       -2     
  Lines        8085     8136      +51     
==========================================
- Hits         5840     5455     -385     
- Misses       2245     2681     +436     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@samaloney samaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure allowing operations in different time systems is a good idea. A product should either have time in OBT or UTC and the conversion should only be done once in the from_level0 or similar methods?

All products (>L0) will always have obt_timerange from products will have an utc_timerange but only if the time sys in header is UTC and not sure supporting on the fly conversion is a good idea because the spice kernel dependence.

This aren't blocker just thoughts

@nicHoch
Copy link
Collaborator Author

nicHoch commented Jan 29, 2026

i agree that it is not ideal to provide some of the on the fly conversions. The auto converting most of the time (and if proper used never) will not happen as our process steps will do this explicit.

In general this L1 product could have state depended time systems and i think it would be bad experience if the properties will raise an exception if used

@nicHoch nicHoch force-pushed the TIMESYS_fits_read branch from 9b9c9dc to 748ae6a Compare March 9, 2026 13:56
@nicHoch nicHoch force-pushed the TIMESYS_fits_read branch from 748ae6a to 5a7c670 Compare March 12, 2026 13:05
@nicHoch nicHoch changed the title change internal time system of L1 products to be SCETime or UTC add option to fits reader to open L1 products also in UTC >L1 UTC per default (via keyword in header) Mar 17, 2026
@nicHoch nicHoch requested a review from samaloney March 17, 2026 15:57
@samaloney samaloney requested a review from Copilot March 23, 2026 09:33
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an opt-in reader behavior to interpret FITS time columns using the TIMESYS/DATE-OBS header context (via get_timeformat_from_TIMESYS=True) so L1 products can be opened in UTC when needed, while keeping SCET/OBT as the default for processing workflows.

Changes:

  • Add get_timeformat_from_TIMESYS option to Product(path) reading logic to choose UTC vs OBT-based offsets when reconstructing time/timedel.
  • Update time-range properties and product combination logic to handle non-SCETime internal time formats.
  • Rename exposure to min_exposure across FITS header generation and tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
stixcore/util/scripts/end2end_testing.py Adds local debug code in __main__ (currently breaks script execution).
stixcore/products/tests/test_factory.py Adds test coverage for reading time formats (SCET vs UTC).
stixcore/products/product.py Implements time-format selection from headers; updates timerange logic; renames exposure API; introduces UTC tz-aware constant.
stixcore/products/level3/flarelistproduct.py Adds warning log when SCET is approximated from UTC using SPICE.
stixcore/products/level3/flarelist.py Adds warning logs; renames exposure to min_exposure.
stixcore/processing/tests/test_publish.py Updates tests to use min_exposure.
stixcore/processing/tests/test_end2end.py Alters pytest markers for the end-to-end identical test.
stixcore/io/product_processors/tests/test_processors.py Updates tests to use min_exposure and expect XPOSURE from it.
stixcore/io/product_processors/fits/processors.py Writes XPOSURE from min_exposure; minor refactor of scet_timerange usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Collaborator

@samaloney samaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems ok, one test marker commented out?

Comment on lines -76 to -77
# @pytest.mark.end2end
@pytest.mark.remote_data
@pytest.mark.end2end
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

@nicHoch nicHoch merged commit 53de1ae into i4Ds:master Mar 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants