Remove upper restriction to pandas

[Natim]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes googleapis/google-cloud-python#15618 🦕

[gemini-code-assist]

Summary of Changes

Hello @Natim, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the project's dependency configuration to remove the upper version limit for the pandas library. This modification ensures that the project can integrate with pandas versions 3.0.0 and newer, addressing potential compatibility issues and allowing the use of the latest features and improvements from the pandas ecosystem.

Highlights

• Dependency Management: The upper version restriction for the pandas library has been removed from the noxfile.py configuration, allowing for compatibility with pandas versions 3.0.0 and above during prerelease testing.
• Project Dependencies: The setup.py file has been updated to remove the upper version constraint on pandas, changing it from < 3.0.0 to no upper limit, thus enabling the project to work with future pandas releases.

Changelog

• noxfile.py
  • Removed the upper version constraint pandas<3.0.0rc0 from the prerelease session's installation command, now simply installing pandas.
• setup.py
  • Updated the install_requires list to remove the upper version constraint ,< 3.0.0 for pandas, allowing pandas >= 1.5.3.

Activity

• No human activity (comments, reviews, or progress updates) has been recorded on this pull request since its creation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request removes the upper version constraint on the pandas dependency to allow compatibility with pandas 3.0 and later. The change in noxfile.py correctly updates the pre-release testing session to use the latest pandas pre-release, which is a good step for ensuring forward compatibility.

My review includes one suggestion for setup.py. While removing the < 3.0.0 restriction is correct, I recommend adding a new upper bound of < 4.0.0. This is a common best practice that provides stability for users by ensuring compatibility with the pandas 3.x series while preventing unexpected breakages from a future, untested pandas 4.0 major release.

[gemini-code-assist]

While this change enables support for pandas 3.0, removing the upper version bound entirely can introduce risk for users. Since db-dtypes is tightly coupled with pandas internals, future major versions of pandas (e.g., 4.0) are likely to have breaking changes that would affect this library.

To provide more stability for users, it's a common best practice to cap the dependency at the next major version. This would allow all pandas 3.x versions but prevent an automatic upgrade to an untested pandas 4.0. It ensures users don't get a broken environment unexpectedly.

Suggested change

	"pandas >= 1.5.3",
	"pandas >= 1.5.3, < 4.0.0",

[Natim]

We can wait for it to fail to limit it. Since it already covers pandas v1 and v2 there is no need to be so protective at this library level.

IMO, It is the project's responsibility to decide whether they want to upgrade or not. In this case going from v1.4.4 to v1.5.0, rollback to a previous version of pandas in our project while 3.0.0 is already supported.

[Natim]

In the meantime I was able to tell my project to override the restriction using:

[tool.uv]
override-dependencies = ["pandas>=3.0.0"]

But as you will guess, I would rather not...

[Natim]

Alternatively we can also describe it like that:

[tool.uv]
dependency-metadata = [
  { name = "db-dtypes", requires-dist = [
      'numpy>=1.24.0',
      "packaging>=24.2.0",
      "pandas>=1.5.3",
      "pyarrow>=13.0.0",
  ] }
]

[chalmerlowe]

I am leaning toward deferring this til after the migration to the mono-repo. There is more complexity here than just fixing the compliance tests. We need to ensure that our normal test suite also runs under both pandas 2.x and 3.x. Here is an action plan.

Action Plan: Fixing Unit Tests (Pandas 2.x & 3.x Compatibility)

When we pick this Issue #388 up, we need to ensure our tests continue to pass on legacy Pandas and the new strict Pandas 3.0. The following are a first pass analysis of the tests that fail in our unit tests. I have not checked system tests. As a first pass analysis, there is no guarantee that these solutions are correct, so take this action plan with a grain of salt.

1. Fix test_any and test_all (Robust Handling)

Issue: Pandas 3.0 stricter behavior raises TypeError for boolean reductions on dates, while Pandas 2.x allowed it (possibly returning True/False based on null checks).

Action: Update tests/unit/test_dtypes.py in test_any / test_all:

• Add a version check or try/except block.
• Example Pattern:

try:
    assert a.any()
except TypeError:
    # Pandas 3.0+ raises TypeError for datetime64 reduction
    if pd.__version__ >= "3.0.0":
        pass  # Expected behavior
    else:
        raise  # Unexpected failure on older pandas

Better: Use pytest.raises(TypeError) inside a check:

if pd.__version__ >= "3.0.0":
    with pytest.raises(TypeError):
        a.any()
else:
    assert a.any()

2. Fix test_min_max_median (Warning Check)

Issue: The warning ("empty slice") is missing in recent versions. pytest.warns fails if no warning is emitted.

Action: Update tests/unit/test_dtypes.py in test_min_max_median:

• Use pytest.warns(recwarn) or a conditional context manager (like warnings.catch_warnings(record=True)) to inspect captured warnings after the fact.
• Example Pattern:

with pytest.warns(RuntimeWarning) if pd.__version__ < "3.0.0" else contextlib.nullcontext():
    # ... median() call ...

Alternative: Simply remove the check if we don't care about enforcing the presence of the warning on older versions (as long as the result is correct). The result assertion assert empty.median() is pd.NaT is the critical part.

3. Fix test_fillna (Method Deprecation)

Issue: Pandas 3.0 deprecated/removed fillna(method=...).

Action: Update tests/unit/test_dtypes.py in test_fillna:

The deprecation path is tricky. Older pandas supports fillna(method=...) but might not support .bfill() directly on ExtensionArrays if not implemented.

Robust approach: Check Pandas version and dispatch correctly.

if pd.__version__ >= "3.0.0" and meth:
    # Use dedicated methods for newer pandas
    if meth in ["backfill", "bfill"]:
        res = a.bfill(limit=limit)
    elif meth in ["pad", "ffill"]:
        res = a.ffill(limit=limit)
else:
    # Legacy way
    res = a.fillna(value, method=meth, limit=limit)

np.testing.assert_array_equal(res._ndarray, expect._ndarray)

Note: This assumes db_dtypes implements or inherits bfill/ffill. If not, we might need to implement them or stick to fillna if it still works via some compatibility layer. Given the error is TypeError (str < int), it suggests the new pandas 3.0 fillna implementation might be checking method argument types differently or strictly expecting keyword-only arguments. Verify if passing method=meth as a keyword argument (instead of positional) solves it first? But likely the dispatch logic is needed.