Skip to content

[Blob] Add some metric when write blob from http.#8464

Open
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:add_http_blob_metric_0705
Open

[Blob] Add some metric when write blob from http.#8464
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:add_http_blob_metric_0705

Conversation

@wwj6591812

@wwj6591812 wwj6591812 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Background

For tables with BLOB columns, writers may fetch external HTTP resources through blob descriptors. After blob-write-null-on-missing-file and blob-write-null-on-fetch-failure are available, users can choose to write NULL instead of failing the write for missing files or non-404 fetch failures. However, it is still difficult to observe how often blob fetches succeed, fail, or are converted to NULL during write.

Changes

This PR adds table-level blob fetch metrics for the write path. The metrics include total fetch attempts, successful fetches, successfully fetched bytes, NULLs written by missing-file handling, NULLs written by fetch-failure handling, unhandled fetch failures, and failure categories such as HTTP 404, non-404 HTTP 4xx, HTTP 5xx, invalid URI, and other errors.

The implementation uses fixed metric names under the blobFetch metric group and does not extend the public MetricGroup API with dynamic subgroups. It also records success only after the blob stream is fully copied, and records failures from both opening and reading the blob stream. Missing-file NULLs produced by the Flink descriptor existence pre-check are also counted when the writer receives a NULL blob field carrying a blob descriptor.

Kmon Result

image

Tests

Added unit coverage for metric registration and failure classification in BlobFetchMetricsTest, and writer-side reporter coverage in BlobFormatWriterTest for successful fetches, fetch failures written as NULL, unhandled fetch failures, and pre-checked missing-file NULLs.

Verified with:

mvn -pl paimon-format,paimon-core -am -DfailIfNoTests=false -Dtest=BlobFormatWriterTest,BlobFetchMetricsTest test

@wwj6591812

Copy link
Copy Markdown
Contributor Author

@JingsongLi
Hi,Please CC, Thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant