[Blob] Add some metric when write blob from http.#8464
Open
wwj6591812 wants to merge 1 commit into
Open
Conversation
Contributor
Author
|
@JingsongLi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
For tables with BLOB columns, writers may fetch external HTTP resources through blob descriptors. After
blob-write-null-on-missing-fileandblob-write-null-on-fetch-failureare available, users can choose to write NULL instead of failing the write for missing files or non-404 fetch failures. However, it is still difficult to observe how often blob fetches succeed, fail, or are converted to NULL during write.Changes
This PR adds table-level blob fetch metrics for the write path. The metrics include total fetch attempts, successful fetches, successfully fetched bytes, NULLs written by missing-file handling, NULLs written by fetch-failure handling, unhandled fetch failures, and failure categories such as HTTP 404, non-404 HTTP 4xx, HTTP 5xx, invalid URI, and other errors.
The implementation uses fixed metric names under the
blobFetchmetric group and does not extend the publicMetricGroupAPI with dynamic subgroups. It also records success only after the blob stream is fully copied, and records failures from both opening and reading the blob stream. Missing-file NULLs produced by the Flink descriptor existence pre-check are also counted when the writer receives a NULL blob field carrying a blob descriptor.Kmon Result
Tests
Added unit coverage for metric registration and failure classification in
BlobFetchMetricsTest, and writer-side reporter coverage inBlobFormatWriterTestfor successful fetches, fetch failures written as NULL, unhandled fetch failures, and pre-checked missing-file NULLs.Verified with:
mvn -pl paimon-format,paimon-core -am -DfailIfNoTests=false -Dtest=BlobFormatWriterTest,BlobFetchMetricsTest test