Skip to content

Convert numpy integer and float scalars to Python numbers on insert - #877

Open
TanbirRamim wants to merge 1 commit into
simonw:mainfrom
TanbirRamim:numpy-scalars
Open

TanbirRamim wants to merge 1 commit into
simonw:mainfrom
TanbirRamim:numpy-scalars

Conversation

@TanbirRamim

@TanbirRamim TanbirRamim commented Sep 23, 2026 •

Copy link
Copy Markdown

Fixes #876.

Raw numpy scalars like np.int64(5) were passed straight to sqlite3, which stores them via the buffer protocol as BLOBs (b'\x05\x00...'). The column types were already right, since COLUMN_TYPE_MAPPING knows the numpy types, but the values were not. jsonify_if_needed() now calls .item() on np.integer / np.floating values when numpy is installed.

I left out other numpy types (bool_, datetime64, ...) because they aren't in the type mapping yet and .item() isn't always what you'd want for them.

Tests: added test_insert_raw_numpy_scalars, which uses pytest.importorskip("numpy"), so it runs in the numpy: 1 CI legs without needing pandas. It fails on main and passes with the fix. The full suite passes with and without numpy installed. black, flake8, mypy, pyright and ty check are clean. This doesn't touch the test that #875 rewrites.


📚 Documentation preview 📚: https://sqlite-utils--877.org.readthedocs.build/en/877/

Copilot AI lite review requested due to automatic review settings September 23, 2026 15:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Raw numpy scalar values get silently stored as corrupted BLOBs instead of being converted

2 participants