DuckDB Extensions

[Rizxcviii]

DuckDb supports different extensions to enable specific integrations with different file systems. For our use, we're working with Apache iceberg. Is there an option to additionally install extensions when using this lambda layer?

[bengeois]

Great question @Rizxcviii. For now, the published layers include the base DuckDB build with the default set of extensions. However, I’m exploring a way to publish additional layers for specific DuckDB extensions.

That said, it’s actually possible to install and load additional extensions dynamically in a lambda using:

con.install_extension("mysql")
con.load_extension("mysql")

However, this approach has some limitations:

• It won’t work in private VPCs without Internet access.
• It adds cold start overhead, since the extension has to be downloaded the first time.

[Gudsfile]

DuckDB Extensions for AWS Lambda are proposed with the DuckDB NodeJS layer https://github.com/tobilg/duckdb-nodejs-layer. Maybe something similar could be done for the Python layer.