Aliyun, Dell: Handle EOF in OSS and ECS InputStream read() variants - #17965
damansingh1313 wants to merge 2 commits into
Conversation
OSSInputStream and EcsSeekableInputStream advanced pos/next and incremented read metrics before checking whether the underlying stream had reached EOF (-1). This corrupted the tracked position by one byte and passed a negative count to the read-bytes counter. GCSInputStream, S3InputStream, and ADLSInputStream already guard against this; apply the same fix here.
|
Hi @anoopj @amogh-jahagirdar, |
| } | ||
|
|
||
| @Test | ||
| public void testReadSingleEOF() throws Exception { |
There was a problem hiding this comment.
This test here doesn't exercise this fix. I think it might be passing even without the fix. This is because the single-byte path already returned -1 correctly pre-fix (return stream.read()). The only bug there is pos/next desync.
Consider adding a getPos() assertion:
assertThat(in.read()).isEqualTo(EOF);
assertThat(in.getPos()).isEqualTo(data.length);
This applies to the other testReadSingleEOF as well.
There was a problem hiding this comment.
Addressed, thanks for pointing it out — added the getPos() check to both tests.
anoopj
left a comment
There was a problem hiding this comment.
Code change LGTM. Left a comment on testing
anoopj
left a comment
There was a problem hiding this comment.
cc @amogh-jahagirdar for a review.
Why is this change needed?
OSSInputStream(aliyun) andEcsSeekableInputStream(dell) advancepos(andnext, in OSS's case) and increment thereadBytes/readOperationsmetric counters before checking whether the underlying stream has reached EOF (-1). When a read call actually returns-1:read()) or decremented by one byte (read(byte[], int, int), since-1is added topos/next), desyncing the stream's internal position bookkeeping from the real position in the object.readBytescounter is incremented by-1in the buffered-read path, producing a nonsensical negative byte count in read metrics.S3InputStream,GCSInputStream, andADLSInputStreamalready guard against this (fixed in #16055);OSSInputStreamandEcsSeekableInputStreamwere the two implementations left over from that fix. This PR applies the same guard to both.A previous attempt at this exact fix (#16266) was opened but auto-closed for inactivity rather than rejected on merit — this PR picks that up with tests added for both the single-byte and buffered
read()overloads on each class.Closes #16062
What user-facing changes does this include?
None. This only affects internal position tracking and read metrics inside
OSSInputStreamandEcsSeekableInputStreamwhen a read reaches EOF; there is no change to any public API, table format, or configuration.Are there any user-facing changes?
How was this change tested?
Added
testReadSingleEOFandtestReadBufferedEOFto bothTestOSSInputStreamandTestEcsSeekableInputStream, mirroring the tests added forS3InputStream/GCSInputStream/ADLSInputStreamin #16055.Verified locally that all four new tests fail against the pre-fix code and pass after the fix; ran
spotlessCheckfor both modules.