Experimental macOS (Intel) build of riptide_cpp / riptable — findings & patch notes


With the help of chat GPT PRo, I’ve been experimenting with building and using `riptable` / `riptide_cpp` on **macOS (Intel)**.  
I’m sharing these results because the C++ core actually compiles on macOS with some small portability tweaks — but the Python extension ultimately **segfaults during module initialization**.

This issue summarizes:

- ✔️ full environment details  
- ✔️ the exact build steps  
- ✔️ every error encountered along the way  
- ✔️ patches applied (file-by-file)  
- ❌ the final unresolved crash (`PyModule_Create2` inside `PyInit_riptide_cpp`)  
- ❓ questions/suggestions for maintainers  

---

# 1. Environment

### ✔️ Hardware
- **Machine**: 2017 iMac (Intel)
- **CPU**: Intel(R) Xeon(R) W-2191B @ 2.30GHz  
- **Cores**: Logical 36 / Physical 18  
- **Arch**: x86_64

### ✔️ OS
```
macOS 15.7.2 (24G325)
Darwin Kernel Version 24.6.0
```

### ✔️ Conda / Python
```
conda 24.7.1
Environment: py311 → Python 3.11.9
```

### ✔️ Build Toolchain
- CMake **3.29.4** (conda-forge)
- Ninja **1.13.1**
- AppleClang **17.0.0.17000013**
- SDK: macOS15.5.sdk

### ✔️ Dependencies installed in env
- numpy 2.3.1  
- tbb + tbb-devel  
- benchmark  
- zstd  
- scikit-build-core  

### ✔️ Version under test
- riptable 1.17.1 (PyPI)
- riptide_cpp 1.19.0 (PyPI)  
- repo cloned from GitHub (`git rev-parse HEAD` for my build: **INSERT HASH**)

---

# 2. Summary of What Works / Fails

### ✔️ Works
- Manual **CMake build** of the C++ core (`riptide_cpp`)  
- Building all test binaries:
  - `riptide_test` ✔️ builds
  - `riptide_python_test` ✔️ builds
  - `sdsfile_test` ✔️ builds

### ⚠️ Partial
- Some C++ tests run; at least one (`riptide_test`) **crashes with `Illegal instruction`** during test discovery.

### ❌ Fails
- `pip install riptable` or `pip install .` for `riptide_cpp` using `scikit-build-core`  
  - cmake 4.1.3 invoked by backend fails due to:
    - Expectation of `libpython3.11.a`
    - Incorrect CLI argument: `--=SEE_OVERRIDES_BELOW=--`
- Python `import riptide_cpp`  
  - Initially: **`ud2`** trap inside `GetProcCount()`  
  - After patching: **segfault in `PyImport_Import` via `SetupLogging()`**  
  - After further patching: **segfault inside `PyModule_Create2` from `PyInit_riptide_cpp`**

---

# 3. Step-by-Step Reproduction

## 3.1 `pip install riptable` failure (CMake invoked by scikit-build-core)

**Errors:**
```
WARNING: libpython3.11.a is not a real file
CMake Error: Unknown argument --=SEE_OVERRIDES_BELOW=--
*** CMake configuration failed
```

### ✔️ Reproduction:
```
conda create -n py311 python=3.11
conda activate py311
pip install riptable
```

---

# 4. Manual C++ Build Using CMake 3.29.4

## 4.1 Clone + configure
```
git clone https://github.com/rtosholdings/riptide_cpp.git
cd riptide_cpp
mkdir build && cd build
```

### Python / NumPy hints
```
export PY_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_paths()['include'])")
export PY_LIBRARY=$(python -c "import sysconfig, os; print(os.path.join(sysconfig.get_config_var('LIBDIR'), 'libpython'+sysconfig.get_config_var('LDVERSION')+'.dylib'))")
export NUMPY_INCLUDE_DIR=$(python -c "import numpy; print(numpy.get_include())")
```

### Configure:
```
cmake .. -DCMAKE_BUILD_TYPE=Release \
  -DPython3_INCLUDE_DIR="$PY_INCLUDE_DIR" \
  -DPython3_LIBRARY="$PY_LIBRARY" \
  -DPython3_NumPy_INCLUDE_DIRS="$NUMPY_INCLUDE_DIR"
```

---

# 5. Fixes Applied During Build (with error snippets)

## ✔️ FIX 1 — missing `benchmark` config
```
Could not find benchmarkConfig.cmake
```

## ✔️ FIX 2 — missing TBB config
```
Could not find TBBConfig.cmake
```

## ✔️ FIX 3 — unsupported JCC mitigation flags
```
-mbranches-within-32B-boundaries not supported by Apple's assembler
```

## ✔️ FIX 4 — AVX intrinsics without flags
```
requires target feature 'avx'
```

## ✔️ FIX 5 — `std::format` / `std::make_format_args` error
```
no matching function for call to 'make_format_args'
```

## ✔️ FIX 6 — template misuse in `basic_ops.h`
```
std::is_same_v<..., y> → y is not a type
```

## ✔️ FIX 7 — `ud2` in `GetProcCount`
```
GetProcCount: ud2
```

## ✔️ FIX 8 — crash inside `SetupLogging`
```
PyImport_Import → SetupLogging() → PyInit_riptide_cpp
```

---

# 6. FINAL CRASH (Unresolved)

### ❌ Current Behavior:
```
python -c "import riptide_cpp"
Segmentation fault: 11
```

### Backtrace:
```
frame #0: libpython3.11.dylib`PyModule_Create2 + 11
frame #1: riptide_cpp.cpython-311-darwin.so`PyInit_riptide_cpp + 255
```

### Interpretation:
- The module initialization sequence appears incompatible with CPython 3.11’s expectations on macOS Intel.
- Likely an issue with the `PyModuleDef` structure or a corruption/assumption in `PyInit_riptide_cpp`.

---

# 7. Questions

- **Is macOS (Intel) considered a supported or intended platform?**  
  While Linux/Windows are primary targets -- JCC flags, affinity code, and logging -- a macOS (Intel or Apple Silicon) would be most welcome.

- **Would you consider a “best-effort macOS mode”?**  
  Examples:  
  - Disable formatted logging  
  - Fallback CPU detection  
  - SIMDe-friendly dispatch  
  - Avoid affinity logic

- **Packaging:**  
  The `scikit-build-core` + CMake path currently cannot build on macOS conda environments due to reliance on `libpython3.11.a` and invalid flags.

- **Module init:**  
  Any guidance on known-good patterns for `PyModuleDef` + Python 3.11?  
  The crash is reliably inside `PyModule_Create2`.

---

# 8. Appendix — High-Level Patch List

- `cmake/AddCommonSettings.cmake`  
- `src/logging/logging_service.h`  
- `src/basic_ops.h`  
- `src/MathWorker.cpp`  
- `src/Logger.cpp`  

Full diffs are available on request.

---


Experimental macOS (Intel) build of riptide_cpp / riptable — findings & patch notes #175

Description

1. Environment

✔️ Hardware

✔️ OS

✔️ Conda / Python

✔️ Build Toolchain

✔️ Dependencies installed in env

✔️ Version under test

2. Summary of What Works / Fails

✔️ Works

⚠️ Partial

❌ Fails

3. Step-by-Step Reproduction

3.1 pip install riptable failure (CMake invoked by scikit-build-core)

✔️ Reproduction:

4. Manual C++ Build Using CMake 3.29.4

4.1 Clone + configure

Python / NumPy hints

Configure:

5. Fixes Applied During Build (with error snippets)

✔️ FIX 1 — missing benchmark config

✔️ FIX 2 — missing TBB config

✔️ FIX 3 — unsupported JCC mitigation flags

✔️ FIX 4 — AVX intrinsics without flags

✔️ FIX 5 — std::format / std::make_format_args error

✔️ FIX 6 — template misuse in basic_ops.h

✔️ FIX 7 — ud2 in GetProcCount

✔️ FIX 8 — crash inside SetupLogging

6. FINAL CRASH (Unresolved)

❌ Current Behavior:

Backtrace:

Interpretation:

7. Questions

8. Appendix — High-Level Patch List

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3.1 `pip install riptable` failure (CMake invoked by scikit-build-core)

✔️ FIX 1 — missing `benchmark` config

✔️ FIX 5 — `std::format` / `std::make_format_args` error

✔️ FIX 6 — template misuse in `basic_ops.h`

✔️ FIX 7 — `ud2` in `GetProcCount`

✔️ FIX 8 — crash inside `SetupLogging`