Docs, Flink: Update Flink configuration and write options in Iceberg documentation - #17938
Docs, Flink: Update Flink configuration and write options in Iceberg documentation#17938talatuyarer wants to merge 2 commits into
Conversation
talatuyarer
changed the title
Guosmilesmile
left a comment
Guosmilesmile
left a comment
There was a problem hiding this comment.
Thanks for the PR! Regarding exposing these configurations, I don't think we necessarily need to expose all of them.
For example, table.exec.iceberg.use-flip27-source is already enabled by default, and we also intend to use the FLIP-27 source. So I wonder if this configuration can be hidden.
For table.exec.iceberg.expose-split-locality-info, I think it may no longer take effect with the new source. We should confirm whether this configuration still needs to be exposed.
|
Good catch @Guosmilesmile On I think we should keep every user-settable option visible in the reference. But I am fine any decision to follow. |
| | compression-strategy | Table write.orc.compression-strategy | Overrides this table's compression strategy for ORC tables for this write | | ||
| | write-parallelism | Upstream operator parallelism | Overrides the writer parallelism | | ||
| | branch | main | Branch to write to | | ||
| | table-refresh-interval | null (disabled) | Interval at which the writers refresh the table metadata, so that long-running jobs pick up changes such as updated table properties. Experimental. | |
There was a problem hiding this comment.
This one was actually added so the credentials are refreshed by the new requests to the catalog.
I'm still not convinced that this is the way that this should be solved - that is why this is experimental.
I would not like to expose this in the doc.
| |--------------------------------------------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | table.exec.iceberg.infer-source-parallelism | true | If true, the source parallelism for batch reads is inferred from the number of scan splits, capped by `table.exec.iceberg.infer-source-parallelism.max` and by the query limit if one is set. If false, the source parallelism is taken from the Flink configuration. Streaming reads never infer parallelism. | | ||
| | table.exec.iceberg.infer-source-parallelism.max | 100 | Maximum inferred parallelism for the source operator. | | ||
| | table.exec.iceberg.expose-split-locality-info | none | Whether to expose split host information to use Flink's locality-aware split assigner. Only applies to the deprecated `FlinkSource`, i.e. when `table.exec.iceberg.use-flip27-source` is `false`; the FLIP-27 `IcebergSource` ignores this option. If unset, locality information is exposed automatically when the table's storage can provide block locations (for example HDFS). | |
There was a problem hiding this comment.
I'm not sure who uses this feature. I think we should deprecate it and remove instead of exposing it in the doc.
| | table.exec.iceberg.expose-split-locality-info | none | Whether to expose split host information to use Flink's locality-aware split assigner. Only applies to the deprecated `FlinkSource`, i.e. when `table.exec.iceberg.use-flip27-source` is `false`; the FLIP-27 `IcebergSource` ignores this option. If unset, locality information is exposed automatically when the table's storage can provide block locations (for example HDFS). | | ||
| | table.exec.iceberg.fetch-batch-record-count | 2048 | Target number of records per fetch batch in the Iceberg source reader. | | ||
| | table.exec.iceberg.worker-pool-size | max(2, available cpu) | Size of the worker pool used to plan or scan manifests. Defaults to the shared Iceberg worker pool size, which is controlled by the `iceberg.worker.num-threads` system property. | | ||
| | table.exec.iceberg.split-assigner-type | simple | Split assigner type that determines how splits are assigned to readers. Currently the only type is `simple`, which doesn't provide any guarantee on order or locality. | |
There was a problem hiding this comment.
Since the only value is simple I would suggest not to expose this in the doc
| | table.exec.iceberg.fetch-batch-record-count | 2048 | Target number of records per fetch batch in the Iceberg source reader. | | ||
| | table.exec.iceberg.worker-pool-size | max(2, available cpu) | Size of the worker pool used to plan or scan manifests. Defaults to the shared Iceberg worker pool size, which is controlled by the `iceberg.worker.num-threads` system property. | | ||
| | table.exec.iceberg.split-assigner-type | simple | Split assigner type that determines how splits are assigned to readers. Currently the only type is `simple`, which doesn't provide any guarantee on order or locality. | | ||
| | table.exec.iceberg.use-flip27-source | true | Use the [FLIP-27](https://cwiki.apache.org/confluence/spaces/FLINK/pages/95653746/FLIP-27+Refactor+Source+Interface) based `IcebergSource` implementation. Set to false to fall back to the deprecated `FlinkSource`. | |
There was a problem hiding this comment.
We should start thinking about removing the FlinkSource, so not expose it now.
| | table.exec.iceberg.worker-pool-size | max(2, available cpu) | Size of the worker pool used to plan or scan manifests. Defaults to the shared Iceberg worker pool size, which is controlled by the `iceberg.worker.num-threads` system property. | | ||
| | table.exec.iceberg.split-assigner-type | simple | Split assigner type that determines how splits are assigned to readers. Currently the only type is `simple`, which doesn't provide any guarantee on order or locality. | | ||
| | table.exec.iceberg.use-flip27-source | true | Use the [FLIP-27](https://cwiki.apache.org/confluence/spaces/FLINK/pages/95653746/FLIP-27+Refactor+Source+Interface) based `IcebergSource` implementation. Set to false to fall back to the deprecated `FlinkSource`. | | ||
| | table.exec.iceberg.use-v2-sink | false | Use the SinkV2 based `IcebergSink` implementation, see [Sink V2 based implementation](flink-writes.md#sink-v2-based-implementation). | |
There was a problem hiding this comment.
We should keep this, and start thinking about to set the default to true
There was a problem hiding this comment.
+1 for changing the default soon. We've been considering changing the default here, but the issue is that savepoints will become incompatible. That said, we've done this before with the source.
| | table.exec.iceberg.worker-pool-size | max(2, available cpu) | Size of the worker pool used to plan or scan manifests. Defaults to the shared Iceberg worker pool size, which is controlled by the `iceberg.worker.num-threads` system property. | | ||
| | table.exec.iceberg.split-assigner-type | simple | Split assigner type that determines how splits are assigned to readers. Currently the only type is `simple`, which doesn't provide any guarantee on order or locality. | | ||
| | table.exec.iceberg.use-flip27-source | true | Use the [FLIP-27](https://cwiki.apache.org/confluence/spaces/FLINK/pages/95653746/FLIP-27+Refactor+Source+Interface) based `IcebergSource` implementation. Set to false to fall back to the deprecated `FlinkSource`. | | ||
| | table.exec.iceberg.use-v2-sink | false | Use the SinkV2 based `IcebergSink` implementation, see [Sink V2 based implementation](flink-writes.md#sink-v2-based-implementation). | |
There was a problem hiding this comment.
+1 for changing the default soon. We've been considering changing the default here, but the issue is that savepoints will become incompatible. That said, we've done this before with the source.
| | flink-maintenance.rewrite.enabled | false | Run data file compaction after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). | | ||
| | flink-maintenance.expire-snapshots.enabled | false | Expire old snapshots after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). | | ||
| | flink-maintenance.delete-orphan-files.enabled | false | Delete orphan files after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). | | ||
| | flink-maintenance.convert-equality-deletes.enabled | false | Convert equality deletes to deletion vectors after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). | |
There was a problem hiding this comment.
Previously there was some discussion whether to include maintenance options here, but I think it makes sense to reference at least the enable flags here.
4 participants
This PR fixes three incorrect statements in the Flink docs and fills two documentation gaps in
flink-configuration.md.