Improve complexbasefield perf for primtive types

[bagerard]

Alternative fix for #2888

Speed up to_python on read-heavy workloads (DictField / ListField)

Problem

Reading a Document that contains large dict/list-shaped fields makes to_python dominate deserialization. On a synthetic-but-realistic 1M-primitive-leaf tree, mongoengine takes ~1.2 s where pymongo takes ~200 ms — a 6× overhead. Profiling shows the vast majority is ComplexBaseField.to_python re-entering itself for every leaf via the untyped-branch recursion:

else:
value_dict[k] = self.to_python(v) # ← recursed even for int/str/None

Every primitive leaf pays the full isinstance(str) → hasattr(to_python) → _import_class("BaseDocument") → hasattr(items) chain before returning itself.

What this PR changes

Two complementary optimizations in ComplexBaseField.to_python, plus one signal from the read path.

1. Primitive short-circuit in the untyped per-item loop (safe, no API change)

if type(v) in _PRIMITIVE_TYPES: # frozenset(str, int, float, bool, NoneType, bytes)
value_dict[k] = v # skip re-entering to_python entirely
elif isinstance(v, Document): ...

Preserves every existing branch (Document → DBRef, .to_python() delegation, nested recursion). Just avoids the tail call for leaves that don't need conversion. ListField inherits ComplexBaseField.to_python and gets this for free.

2. BSON-native trust signal from _from_son

to_python gains a keyword-only bson_native=False parameter:

def to_python(self, value, *, bson_native=False): ...

When bson_native=True is passed and self.field is None (untyped DictField), the whole subtree is guaranteed to contain only BSON-native types — none of the untyped-branch conversions apply — so to_python aliases the input instead of rebuilding it. This is where the biggest win comes from.

The flag is set in exactly one place: _from_son in mongoengine/base/document.py. Every other caller (Document.init, dereference.py, ReferenceField.to_python, user code, tests) defaults to False and behaves identically to before.

The flag is forwarded through the two recursion points inside ComplexBaseField.to_python:

• typed branch: self.field.to_python(v, bson_native=bson_native) (via the safe-call helper below)
• untyped branch: self.to_python(v, bson_native=bson_native) for nested containers

Every field whose to_python calls _from_son (EmbeddedDocumentField, GenericEmbeddedDocumentField) is a trust re-entry point: the BSON-native context restarts fresh for the embedded document's own fields. That's why nested EmbeddedDocumentField(EmbeddedDoc with a DictField) gets the same speedup as a top-level DictField.

3. Backwards-compatible dispatch (_to_python_safe_call + class marker)

User-defined Field subclasses that override to_python(self, value) without knowing abo. Two pieces of plumbing keep them safe:

• BaseField._to_python_accepts_bson_native = False (class-level flag; True on ComplexB
• BaseField._to_python_safe_call(self, value, bson_native=False) — mirrors the existing _to_mongo_safe_call idiom. Reads the class flag and forwards bson_native only when the field declared it.
  _from_son calls field._to_python_safe_call(value, bson_native=True) rather than field.to_python(value, bson_native=True) directly. Scalar fields ignore the flag; complex fields consume it. The class marker replaces per-call co_varnames inspection so scalar-heavy documents don't pay an

Running the benchmark code provided in the other MR provides similar perf for pymongo/mongoengine

pymongo took 0.35s
mongoengine took 0.36s

[terencehonles]

I think this looks pretty good. My initial concern would have been for subclasses of the primitive types (i.e. IntEnum), but I see you've used a strict type in primitive_types so that shouldn't be a problem. To prevent a regression you may want to add a test w/ a subclass of a primitive type.

I'll have to add that we've moved off of MongoEngine (we're completing a PostgreSQL migration), so we won't be contributing more here but we've appreciated all the hard work with the library!