Implement ZstdBlobStoreDAO

[quantranhong1999]

Why

cf #5515

After API changes are finished cf apache#2960.

We need to implement ZstdBlobStoreDAO that takes care of the blob compression.

How

• Configuration in blob.properties:

compression.enabled=true
compression.threshold=16KB

• Implement ZstdBlobStoreDAO:
  The idea is to be fully retro-compatible.

We would, upon S3 save, compress if:

• enabled
• threshold is met
  Then we would set metadata onto the compressed object: content-encoding=zstd + content-original-size=...

Upon read:

• if content-encoding=zstd uncompress, otherwise "server flat"
• Be sure to uncompress on parallel processors.

Use com.github.luben:zstd-jni

CF

byte[] compressed = Zstd.compress(data);
byte[] original = Zstd.decompress(compressed, originalSize);

• In BlobStoreModulesChooser, if compressed is enabled, we load the ZstdBlobStoreDAO

DoD

• Unit tests:
  • implement BlobStoreDAOContract
  • make sure ZstdBlobStoreDAO can read existing uncompressed blobs (without content-encoding=zstd metadata)
• Simple IT to verify guice binding for ZstdBlobStoreDAO ?

[chibenwa]

Add a compress.min-ratio property taking a float between 0 and 1, default 1

We need also to plug this with only uncompress, no compress

[quantranhong1999]

Would be good to record metrics IMO:

• count of blob compressed
• total size before compression
• total size after compression
• time to compress? uncompress?

[quantranhong1999]

WIP: apache#3016

(not much yet)

[chibenwa]

Configuration in blob.properties:

compression.enabled=true
compression.threshold=16KB
compression.uncompress.only=true

?

[quantranhong1999]

compression.enabled=true
compression.threshold=16KB

Yes.

compression.uncompress.only=true

I tend to rely only on compression.min-ratio.
compression.min-ratio=0 could mean "compression.uncompress.only=true". No compression, always uncompress (if needed).
Ok?

[chibenwa]

I tend to rely only on compression.min-ratio.
compression.min-ratio=0 could mean "compression.uncompress.only=true". No compression, always uncompress (if needed).

Very smart.

I buy it.

Please remember to document explicitly this intent !