feat: Spec multi-result-set API by zeroshade · Pull Request #3871 · apache/arrow-adbc

zeroshade · 2026-01-08T18:04:11Z

Extracted from #3607 with influence by the comments there and main...CurtHagenlocher:arrow-adbc:MoreResults, this contains a proposal for handling multi-result set query execution via ADBC by adding a new function for drivers, AdbcStatementNextResultSet.

This also includes the necessary changes for an ADBC API Revision 1.2.0 (macro defines and so on). The comment above the function includes all the semantic definitions of the behavior.

zeroshade · 2026-01-08T18:08:09Z

After discussion and agreement on the proposed function, I'll update the relevant implementations here of the other driver managers to leverage and expose this (python, rust, Go, etc.)

lidavidm · 2026-01-09T01:27:11Z

Should we make a branch for spec changes?

zeroshade · 2026-01-09T01:32:31Z

Should we make a branch for spec changes?

That makes sense, gather the spec updates in the branch and then eventually decide to push it as one thing.

lidavidm · 2026-01-09T02:53:33Z

I made https://github.com/apache/arrow-adbc/tree/spec-1.2.0

zeroshade · 2026-01-09T15:39:58Z

I've changed the base for this PR to point to the spec-1.2.0 branch.

Now we just need people to weigh in on the proposal 😄

paleolimbot

Nice!

The previous PR also had:

ADBC_EXPORT
AdbcStatusCode AdbcStatementNextResultSetSchema(struct AdbcStatement* statement,
                                                struct ArrowSchema* schema,
                                                struct AdbcError* error);

Is that no longer planned or no longer needed?

c/include/arrow-adbc/adbc.h

zeroshade · 2026-01-19T22:05:55Z

I've updated the PR based on the comments and discussion. Everyone please take a look and let me know what you think!

c/include/arrow-adbc/adbc.h

lidavidm · 2026-01-20T06:47:13Z

c/include/arrow-adbc/adbc.h

+  /// \brief opaque implementation-defined state
+  void* private_data;
+
+  /// \brief Release the result set and any associated resources.


We can say this also cancels the query if not completed?

docs updated to say this.

zeroshade · 2026-01-20T16:55:09Z

Updated based on the most recent comments, converted the callback members of AdbcResultSet to instead be free functions (along with the corresponding callback members in the AdbcDriver struct.

davidhcoe · 2026-01-21T00:11:11Z

@jadewang-db and @eric-wang-1990 are interested in this.

c/include/arrow-adbc/adbc.h

felipecrv · 2026-01-22T02:16:05Z

c/include/arrow-adbc/adbc.h

+  /// ADBC_VERSION_1_0_0. The driver must not access the new fields,
+  /// which will carry undefined values.


Shouldn't the manager populate them with functions that fail immediately?

I'm not actually sure what this is trying to say, now that you point it out.

I believe this is in line with the current behavior. No attempt is made to pre-populate with functions that fail immediately (likely to avoid having to create separate versions for every callback going forward). We currently leave these members undefined in the struct. If we want to change this behavior, I'd prefer we do it in a different PR, not this one.

Probably we should describe the user perspective and driver perspective separately.

From the user perspective: the manager will return a fully populated struct, but some of the functions may fail with NOT_IMPLEMENTED.

From the driver perspective: they should not access members outside of the version specified by the driver manager. (Though this is already required by the contract on the init func, and also you have no other way of knowing the size of the struct, nor do you know whether it's the driver manager or something else calling you.) I don't think we need to specify what exactly the driver manager does here.

Ping @zeroshade, I think we should at least avoid specifying exactly what the driver manager does here, just that the driver manager expects that if it gives the driver a particular version, it should not try to access any functions defined after that version.

How's the updated version?

paleolimbot

Just optional nits on the names from me!

Now that the new functions are not embedded callbacks on the AdbcMultiResultSet it's easy to add schema-only execution if we need it.

c/include/arrow-adbc/adbc.h

abonander

👍 nice and feasible to implement for the ClickHouse driver!

zeroshade · 2026-02-10T16:56:50Z

@lidavidm @paleolimbot I've updated based on the feedback including adding an ExecuteSchemaMulti function. Any other comments on this?

lidavidm · 2026-02-11T01:41:32Z

c/include/arrow-adbc/adbc.h

+///
+/// The AdbcMultiResultSet must outlive the returned partitions.


I'm not sure about this (isn't the point of the partitions to be independent of the API objects?)

fair enough

lidavidm · 2026-02-11T01:41:59Z

c/include/arrow-adbc/adbc.h

+/// \param[out] error An optional location to return an error message if necessary.
+///
+/// \return ADBC_STATUS_NOT_IMPLEMENTED if the driver only supports fetching results
+///   as partitions, ADBC_STATUS_INVALID_STATE if called at an inappropriate time,


When is "an inappropriate time"?

artifact of the previous design. it no longer applies here give that we've decoupled everything by having these standalone functions on the separate MultiResultSet object

lidavidm · 2026-02-11T01:43:08Z

c/include/arrow-adbc/adbc.h

+/// then call `MultiResultSetNext` repeatedly until it returns ADBC_STATUS_OK and
+/// sets the release callback to NULL, indicating that there are no more result sets.


I think a state we haven't addressed is whether you can call Next while still reading the previous result set or not. I hazard many databases will not support concurrency, especially because the queries may depend on each other...

Should we leave it up to a driver to determine whether it's allowed? (i.e. "A driver MAY invalidate the current result set when MultiResultSetNext is called") or should we explicitly state that it is invalidated when calling MultiResultSetNext and thus require that behavior?

I personally lean towards the latter rather than the former for consistency, but it would prevent a driver from technically allowing it if it supports it. Thoughts?

I think we can state MAY, and most code will then have to assume it will be invalidated; if you're coding to a specific driver release, then maybe you can make a stronger assumption.

lidavidm · 2026-02-11T01:46:05Z

c/include/arrow-adbc/adbc.h

+  /// ADBC_VERSION_1_0_0. The driver must not access the new fields,
+  /// which will carry undefined values.


Ping @zeroshade, I think we should at least avoid specifying exactly what the driver manager does here, just that the driver manager expects that if it gives the driver a particular version, it should not try to access any functions defined after that version.

c/include/arrow-adbc/adbc.h

feat: Spec multi-result-set API

ed8252a

zeroshade requested a review from lidavidm as a code owner January 8, 2026 18:04

zeroshade requested review from CurtHagenlocher, amoeba, felipecrv, kou and paleolimbot January 8, 2026 18:04

github-actions bot added this to the ADBC Libraries 22 milestone Jan 8, 2026

zeroshade added 3 commits January 8, 2026 13:09

linting

8ec1d95

fix tests

6abbf86

pre-commit

133e895

lidavidm modified the milestones: ADBC Libraries 22, ADBC Libraries 23, ADBC API Specification 1.2.0 (Async) Jan 9, 2026

zeroshade changed the base branch from main to spec-1.2.0 January 9, 2026 15:39

paleolimbot reviewed Jan 9, 2026

View reviewed changes

c/include/arrow-adbc/adbc.h Outdated Show resolved Hide resolved

Update based on comments

52e646e

lidavidm reviewed Jan 20, 2026

View reviewed changes

c/include/arrow-adbc/adbc.h Outdated Show resolved Hide resolved

c/include/arrow-adbc/adbc.h Show resolved Hide resolved

c/include/arrow-adbc/adbc.h Outdated Show resolved Hide resolved

lidavidm reviewed Jan 20, 2026

View reviewed changes

convert to free functions

d88fccb

felipecrv reviewed Jan 22, 2026

View reviewed changes

c/include/arrow-adbc/adbc.h Show resolved Hide resolved

felipecrv reviewed Jan 22, 2026

View reviewed changes

zeroshade added 2 commits January 26, 2026 15:00

updates based on commments

81d6a2b

pre-commit

ec8a174

zeroshade requested review from abonander, davidhcoe, felipecrv, lidavidm and paleolimbot February 5, 2026 23:49

paleolimbot approved these changes Feb 6, 2026

View reviewed changes

c/include/arrow-adbc/adbc.h Outdated Show resolved Hide resolved

abonander approved these changes Feb 7, 2026

View reviewed changes

add ExecuteSchemaMulti

e8173f3

zeroshade requested a review from paleolimbot February 10, 2026 16:56

paleolimbot approved these changes Feb 10, 2026

View reviewed changes

lidavidm approved these changes Feb 11, 2026

View reviewed changes

Update header docs based on comments

0141a8f

lidavidm approved these changes Feb 12, 2026

View reviewed changes

c/include/arrow-adbc/adbc.h Outdated Show resolved Hide resolved

zeroshade added 2 commits February 11, 2026 20:22

fix typo

77cf124

remove old wording from 1.1.0

b9a6f52

zeroshade merged commit 4248d7c into apache:spec-1.2.0 Feb 18, 2026
2 checks passed

		/// ADBC_VERSION_1_0_0. The driver must not access the new fields,
		/// which will carry undefined values.

		///
		/// The AdbcMultiResultSet must outlive the returned partitions.

		/// then call `MultiResultSetNext` repeatedly until it returns ADBC_STATUS_OK and
		/// sets the release callback to NULL, indicating that there are no more result sets.

Comments

Conversation

zeroshade commented Jan 8, 2026

Uh oh!

zeroshade commented Jan 8, 2026

Uh oh!

lidavidm commented Jan 9, 2026

Uh oh!

zeroshade commented Jan 9, 2026

Uh oh!

lidavidm commented Jan 9, 2026

Uh oh!

zeroshade commented Jan 9, 2026

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zeroshade commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeroshade commented Jan 20, 2026

Uh oh!

davidhcoe commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abonander left a comment

Choose a reason for hiding this comment

Uh oh!

zeroshade commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects