Register module paths as site directories; dedupe sys.path (1.8.0)

[ndonkoHenri]

Fixes flet-dev/flet#5071

Problem

CPython processes .pth files only for site directories (site.py over PYTHONHOME's Lib/site-packages) — never for PYTHONPATH entries. Every serious_python_* plugin exposes the app's bundled site-packages through PYTHONPATH / module_paths only, so any package that relies on a .pth file to extend sys.path or run bootstrap code silently broke in packaged apps.

pywin32 is the canonical case. Its layout only works through pywin32.pth, which adds win32, win32\lib and pythonwin to sys.path and imports pywin32_bootstrap — which in turn registers pywin32_system32 as a DLL directory so the .pyd extensions can resolve pywintypes3xx.dll. With the .pth unprocessed, import win32com in a flet build windows app fails with ModuleNotFoundError: No module named 'pywintypes'.

Fix

sp_apply_module_paths now emits the module-path list into a temporary and runs a fixed epilogue (SP_MODULE_PATHS_EPILOGUE) that:

1. Dedupes sys.path: The plugins pass the same list both as PYTHONPATH (consumed by Py_Initialize) and as module_paths (inserted post-init), so every entry appeared twice. Existing occurrences are removed — compared case-/separator-insensitively via os.path.normcase(os.path.abspath(...)) — and the list is re-inserted at the front, preserving the caller's precedence order.
2. Registers every module path that is a directory with site.addsitedir() — the same function site.py uses for real site dirs: .pth path lines are appended after the existing entries exactly as in a regular installation; import lines execute. Non-directories (Android's stdlib/site-packages zips, an absent __pypackages__) are skipped.
3. Cleans its temporaries out of __main__'s globals, which this snippet shares with the user's program.

A malformed .pth is not fatal: site prints its usual "Error processing line …" warning and continues, matching stock CPython behavior.

Also: buffer sizing for the apostrophe escape (24f4d76)

While touching sp_apply_module_paths, a pre-existing (since 1.5.0) flaw in its buffer math was corrected. The estimate for the generated _sp_paths literal budgeted ~3 output bytes per input char, but the apostrophe escape (close the raw string, concatenate "'", reopen) emits 8. An apostrophe-dense path could exhaust the buffer, and the copy guard both reserved too little for its own write (4 bytes vs the escape's 8) and truncated the path silently — after which total - (p - code) underflowed as size_t, handing the epilogue snprintf an effectively unbounded size (heap overflow). Only reachable with pathological paths (a single apostrophe, e.g. C:\Users\O'Brien\…, stays well within budget), but it was wrong.

The buffer is now sized for the true worst case (8 bytes per input char), which makes the guard unreachable; it is kept as a defensive bound with a margin covering its largest write. Validated with a byte-accurate replica of the generator (including snprintf's would-be-length advancement) driven with adversarial inputs — 500-apostrophe paths, dense mixes — plus round-trip compile()/exec of every generated program: no overflow, guard never reached, paths reproduce exactly.

Verification

Reproduced and verified on a Windows 11 VM against Flet v0.86.5:

**Test Code**

import traceback

import flet as ft


def main(page: ft.Page):
    page.scroll = ft.ScrollMode.AUTO
    try:
        import win32com.client

        shell = win32com.client.Dispatch("WScript.Shell")
        result = ft.Text(
            f"win32com OK — WScript.Shell dispatched: {shell}",
            color=ft.Colors.GREEN,
        )
    except Exception:
        result = ft.Text(traceback.format_exc(), color=ft.Colors.RED)

    import sys

    page.add(
        result,
        ft.Text("sys.path:"),
        *[ft.Text(p, size=12) for p in sys.path],
    )


ft.run(main)

[project]
name = "issue-5071"
version = "0.1.0"
description = "Repro for flet-dev/flet#5071 - pywin32 (win32com) in built Windows app"
requires-python = ">=3.10"
dependencies = [
    "flet",
    "pywin32; sys_platform == 'win32'",
]

[tool.flet]
org = "com.mycompany"
product = "issue-5071"

• Before: import win32com.client → No module named 'pywintypes'; every sys.path entry listed twice.

• After (CI-built DLL dropped over dart_bridge.dll in the built app): win32com.client.Dispatch("WScript.Shell") succeeds; sys.path entries appear once, with win32, win32\lib, pythonwin appended at the end.