feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization #3093

chaokunyang · 2025-12-25T16:51:36Z

Why?

The cross-language (xlang) struct serialization had inconsistent nullable and ref flag handling across different language implementations (Java, Rust, C++, Go, Python). This caused:

Schema hash mismatches: Different languages computed different fingerprints for the same struct schema, causing version compatibility issues
Ref flag inconsistency: Some languages wrote ref flags for non-nullable fields while others didn't
Null handling bugs: Null entries in maps were not properly handled in some implementations

What does this PR do?

Core Protocol Alignment

Unified nullable semantics across all languages:

In xlang mode, fields are non-nullable by default (except Optional types and boxed primitives)
Ref flags are only written when nullable=true - non-nullable fields skip the ref flag entirely
This matches the xlang protocol specification and ensures consistent wire format

Key changes per language:

Language	Changes
Java	New `Fingerprint.java` for consistent struct fingerprint computation; updated serializers for xlang nullable defaults
Rust	Simplified `field_need_write_ref_into()` to only check nullable flag
C++	Updated `field_is_nullable` logic; fixed map null entry handling; aligned struct/ext skip logic
Go	Updated codegen encoder/decoder for nullable handling; new pointer utilities
Python	Updated typedef encoder/decoder; aligned struct serialization

Code Refactoring (Java)

Extracted FieldInfo and FieldTypes from ClassDef.java (reduced from ~900 lines to ~3 lines)
Added logging infrastructure (ForyLogger, LogLevel, Logger, Slf4jLogger)
Enhanced TypeRef with additional type utilities
Updated TypeResolver with improved type handling

Bug Fixes

C++ map serializer: Fixed null entry handling - now properly inserts entries with default-constructed keys/values instead of skipping
C++ skip logic: Fixed skip_struct and skip_ext to handle different struct type categories correctly (COMPATIBLE_STRUCT, NAMED_COMPATIBLE_STRUCT, NAMED_STRUCT, STRUCT)
C++ enum handling: Fixed enum field nullability in xlang mode

CI Improvements

Added dedicated python_xlang job for Python cross-language testing
Removed unnecessary bazel setup from Java CI jobs (not needed for Java tests)

Test Updates

Updated cross-language tests in all languages to use non-nullable defaults
Changed std::optional<T> to T in C++ test structs to match xlang semantics
Added comprehensive xlang test coverage for Java, Rust, C++, Go, Python

Related issues

#1017
#2982
#2906

Does this PR introduce any user-facing change?

Does this PR introduce any public API change?
- In xlang mode, struct fields are now non-nullable by default. Users who relied on implicit nullability should add explicit @ForyField(nullable = true) or use Optional<T> types.
Does this PR introduce any binary protocol compatibility change?
- Yes, the ref flag behavior has changed for xlang mode. Non-nullable fields no longer write ref flags. This is a breaking change for existing xlang serialized data.

Benchmark

N/A (Protocol alignment, no performance-related changes)

Use lazy_import for numpy and handle the case when numpy is not available by comparing arrays as lists instead.

…r test The circular reference changes in xread_ref/xread_no_ref broke the xlang protocol because they changed how read_ref_ids is populated. This reverts those changes. Also skip the circular reference test for XLANG mode since XLANG doesn't support true circular references - objects must be registered after they are fully constructed.

Also skip the dict3 circular reference test for XLANG mode since it also uses circular references (dict3 containing itself).

Add back the ref_tracking check before pushing -1 to read_ref_ids. Without this check, when ref_tracking is disabled, the code would fail because NoRefResolver doesn't have read_ref_ids.

This commit fixes cross-language compatibility issues between Java and Python when handling Object/Any types in xlang mode: 1. Java FieldTypes.java: Treat Object.class as nullable in xlang mode to match Python's typing.Any behavior 2. Java TypeResolver.java: Update isFieldNullable() to treat Object.class as nullable in xlang mode 3. Java Fingerprint.java: Include boxed types and Object.class in nullable check for struct fingerprint computation 4. Python typedef.py: Treat typing.Any as nullable in xlang mode, and default ref tracking to false in xlang mode (matching Java's behavior) 5. Python _fory.py and serialization.pyx: Fix circular reference handling by not pushing -1 to read_ref_ids when called from xread_ref, since try_preserve_ref_id already handles the ref_id tracking

Object (Java) and typing.Any (Python) should NOT be nullable by default in xlang mode. Only Optional[T] types should be nullable. This fixes the fingerprint mismatch issue while maintaining the correct xlang semantics that non-Optional types are non-nullable.

Numpy is already a build dependency in pyproject.toml, so it's always installed. Remove the lazy import and conditional check for numpy.

The function just returned the nullable parameter, making it unnecessary. Inline def.nullable directly at the call sites.

Document xlang field nullable/ref defaults: - Non-nullable by default in xlang mode - Only Optional<T> types are nullable - Ref flags only written for nullable fields - Language-specific examples and customization

Allow adding new docs without renumbering: - index: 0 - getting-started: 10 - serialization: 30 - field-nullability: 40 - zero-copy: 50 - row_format: 60 - troubleshooting: 70

- field-nullability.md: focused on null handling only - reference-tracking.md: new doc for shared/circular refs

Allow field() to be called without the id parameter. When omitted, defaults to -1 which uses field name encoding instead of tag ID. This aligns with Java's @ForyField(id default -1).

chaokunyang requested a review from theweipeng as a code owner December 25, 2025 16:51

chaokunyang force-pushed the align_xlang_ref_nullable branch from 14d994d to 1f2f649 Compare December 25, 2025 20:08

chaokunyang requested a review from PragmaTwice as a code owner December 25, 2025 20:08

chaokunyang force-pushed the align_xlang_ref_nullable branch from 1f2f649 to 9471a2e Compare December 26, 2025 03:42

chaokunyang requested review from LiangliangSui, pandalee99 and urlyy December 26, 2025 03:42

pandalee99 approved these changes Dec 26, 2025

View reviewed changes

chaokunyang force-pushed the align_xlang_ref_nullable branch 2 times, most recently from 6b57909 to b772a7f Compare December 26, 2025 08:36

chaokunyang added 2 commits December 26, 2025 17:00

align xlang struct serialization

20b6c77

Merge remote-tracking branch 'asf/main' into align_xlang_ref_nullable

36d3243

chaokunyang force-pushed the align_xlang_ref_nullable branch from b772a7f to 36d3243 Compare December 26, 2025 15:56

chaokunyang added 4 commits December 27, 2025 00:52

fix python xlang tests

ba354c7

fix py xlang tests

73d96ee

fix xlang tests

6136ed6

make java ci do not install pyfory

130962e

chaokunyang force-pushed the align_xlang_ref_nullable branch 12 times, most recently from 877f7af to e0205bd Compare December 29, 2025 15:31

refactor java fields handle nullable/ref

9df452c

chaokunyang added 9 commits December 30, 2025 07:54

fix(python): make numpy optional in cross-language tests

cd6a8a0

Use lazy_import for numpy and handle the case when numpy is not available by comparing arrays as lists instead.

fix(python): skip all circular reference tests for xlang mode

a3615f1

Also skip the dict3 circular reference test for XLANG mode since it also uses circular references (dict3 containing itself).

fix(python): restore ref_tracking check in xread_no_ref

02bb767

Add back the ref_tracking check before pushing -1 to read_ref_ids. Without this check, when ref_tracking is disabled, the code would fail because NoRefResolver doesn't have read_ref_ids.

refactor(python): use direct numpy import in cross-language tests

3a1d0b3

Numpy is already a build dependency in pyproject.toml, so it's always installed. Remove the lazy import and conditional check for numpy.

refactor(go): inline fieldNeedWriteRef function

2836458

The function just returned the nullable parameter, making it unnecessary. Inline def.nullable directly at the call sites.

remove dead code

a4184e9

chaokunyang changed the title ~~feat: align xlang struct serialization~~ feat: unifiy xlang struct serialization for nullable meta Dec 30, 2025

chaokunyang mentioned this pull request Dec 30, 2025

RoadMap for 1.0 #1017

Open

16 tasks

chaokunyang changed the title ~~feat: unifiy xlang struct serialization for nullable meta~~ feat: consistent nullable meta for xlang struct fields serialization Dec 30, 2025

fix python xlang tests ci

4a7a14f

chaokunyang force-pushed the align_xlang_ref_nullable branch from deefc6c to 4a7a14f Compare December 30, 2025 04:08

chaokunyang changed the title ~~feat: consistent nullable meta for xlang struct fields serialization~~ feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization Dec 30, 2025

chaokunyang added 11 commits December 30, 2025 12:13

ci: install numpy for python xlang tests

3b64513

ci: use install_pyfory for python xlang tests

e92f423

docs(xlang): add field nullability and ref tracking guide

64f49e3

Document xlang field nullable/ref defaults: - Non-nullable by default in xlang mode - Only Optional<T> types are nullable - Ref flags only written for nullable fields - Language-specific examples and customization

docs(xlang): use 10-increment sidebar positions

dad352b

Allow adding new docs without renumbering: - index: 0 - getting-started: 10 - serialization: 30 - field-nullability: 40 - zero-copy: 50 - row_format: 60 - troubleshooting: 70

docs(xlang): separate nullability and reference tracking docs

fb50445

- field-nullability.md: focused on null handling only - reference-tracking.md: new doc for shared/circular refs

docs(xlang): rename to field-reference-tracking.md

d439496

docs(xlang): add nullable vs ref tracking comparison

0233343

docs(xlang): add boxed types and pointer types to nullable list

225a249

docs(xlang): add per-field ref tracking defaults and customization

8dac269

feat(java): make @ForyField.id() optional with default -1

3b7b9cc

feat(python): make field() id optional with default -1

92105de

Allow field() to be called without the id parameter. When omitted, defaults to -1 which uses field name encoding instead of tag ID. This aligns with Java's @ForyField(id default -1).

chaokunyang merged commit db57e1b into apache:main Dec 30, 2025
57 checks passed

This was referenced Dec 30, 2025

fix(rust): enable Union type cross-language serialization between Rust and Java #3094

Merged

RoadMap for C++ serialization 1.0 #2906

Open

Roadmap For Go Serialization 1.0 #2982

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization #3093

feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization #3093

Uh oh!

chaokunyang commented Dec 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization #3093

feat(java/python/rust/go/c++): align nullable meta for xlang struct fields serialization #3093

Uh oh!

Conversation

chaokunyang commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

What does this PR do?

Core Protocol Alignment

Code Refactoring (Java)

Bug Fixes

CI Improvements

Test Updates

Related issues

Does this PR introduce any user-facing change?

Benchmark

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chaokunyang commented Dec 25, 2025 •

edited

Loading