macOS Apple Clang: no-OpenMP build fails with duplicate mangled template definitions, while OpenMP is not detected automatically

[kimmanbo]

Expected Behavior

On macOS ARM64(Apple Silicon) with Apple Clang:

1. Configuring with REQUIRE_OPENMP=ON should either detect an installed Homebrew libomp, or provide actionable instructions for supplying its flags and library path.
2. Configuring with REQUIRE_OPENMP=OFF should produce a working build without OpenMP.

Why claims Apple Clang?

Adding below options for using gcc instead Apple Clang resolved both issues.

• -DCMAKE_C_COMPILER=/opt/homebrew/bin/gcc-16
• -DCMAKE_CXX_COMPILER=/opt/homebrew/bin/g++-16

Current Behavior

There appears to be a failure chain on macOS:

1. With REQUIRE_OPENMP=ON, configuration fails because FindOpenMP does not automatically detect Homebrew libomp.

2. With REQUIRE_OPENMP=OFF, configuration succeeds, but Apple Clang fails while compiling src/linclust/kmermatcher.cpp with multiple:

   error: definition with same mangled name '...' as another definition
   

This makes disabling the OpenMP requirement an ineffective workaround.

Steps to Reproduce

0. Pre-requisites

• brew install libomp
• LIBOMP_PREFIX="$(brew --prefix libomp)"

1. Duplicate mangled-name definitions without OpenMP

Use a fresh build directory:

cmake -S . -B build-asan-no-openmp \
    -DCMAKE_BUILD_TYPE=ASan \
    -DHAVE_SANITIZER=1 \
    -DHAVE_TESTS=1 \
    -DREQUIRE_OPENMP=OFF

cmake --build build-asan-no-openmp \
    --target mmseqs-framework \
    --parallel "$(sysctl -n hw.ncpu)"

Configuration succeeds, but compilation of kmermatcher.cpp fails. Representative errors:

src/linclust/kmermatcher.cpp:43:51: error: definition with same mangled name
'_Z22initKmerPositionMemoryIsLb0ELb0EEP12KmerPositionIT_XT0_EXT1_EEm'
as another definition

src/linclust/kmermatcher.cpp:1912:6: error: definition with same mangled name
'...' as another definition

src/linclust/kmermatcher.cpp:83:27: error: definition with same mangled name
'...' as another definition

The affected templates include:

• initKmerPositionMemory
• fillKmerPositionArray
• writeKmersToDisk

The diagnostic points to the same source definition for both the definition and the previous definition.

I also reproduced the error after removing -fsanitize=address from the compile command, so ASan itself does not appear to be required. The relevant condition appears to be the Apple Clang build without OpenMP.

A possible contributing factor is that kmermatcher() instantiates several kmermatcherInner<..., false> specializations, while kmermatcher.cpp later contains explicit instantiation definitions for functions reached through those specializations. The failing symbols appear to be the IncludeSeqLen=false variants.

I am not certain whether this is:

• an Apple Clang template-instantiation/code-generation bug,
• invalid or fragile explicit-instantiation usage in kmermatcher.cpp, or
• an interaction between the two.

2. OpenMP cannot be found automatically

Configure using another fresh build directory:

cmake -S . -B build-asan-openmp-required \
    -DCMAKE_BUILD_TYPE=ASan \
    -DHAVE_SANITIZER=1 \
    -DHAVE_TESTS=1 \
    -DREQUIRE_OPENMP=ON

Configuration fails with:

CMake Error at src/CMakeLists.txt:266 (message):
  -- Could not find OpenMP. Skip check with -DREQUIRE_OPENMP=0.

The generated cache shows that CMake did not find usable OpenMP flags or libraries:

OpenMP_C_FLAGS=NOTFOUND
OpenMP_CXX_FLAGS=NOTFOUND
OpenMP_C_LIB_NAMES=NOTFOUND
OpenMP_CXX_LIB_NAMES=NOTFOUND
OpenMP_libomp_LIBRARY=NOTFOUND

Existing Workaround

The repository's macOS ARM64 CI already supplies the Homebrew libomp settings manually. A similar configuration works locally:

LIBOMP_PREFIX="$(brew --prefix libomp)"

cmake -S . -B build-asan-openmp \
    -DCMAKE_BUILD_TYPE=ASan \
    -DHAVE_SANITIZER=1 \
    -DHAVE_TESTS=1 \
    -DREQUIRE_OPENMP=ON \
    -DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp -I${LIBOMP_PREFIX}/include" \
    -DOpenMP_C_LIB_NAMES=omp \
    -DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp -I${LIBOMP_PREFIX}/include" \
    -DOpenMP_CXX_LIB_NAMES=omp \
    -DOpenMP_omp_LIBRARY="${LIBOMP_PREFIX}/lib/libomp.a"

With these OpenMP settings, kmermatcher.cpp compiles with Apple Clang, including when ASan is enabled.

Or, using gcc as mentioned at "Why claims Apple Clang?" section.

Suggested Improvements

Would it be appropriate to do one or more of the following?

1. Document the required Homebrew libomp CMake options for Apple Clang.
2. Include those options in the OpenMP failure message instead of only suggesting REQUIRE_OPENMP=0.
3. Detect Homebrew libomp automatically on macOS when using Apple Clang.
4. Investigate or restructure the explicit template instantiations in kmermatcher.cpp so a no-OpenMP build also compiles reliably.

Environment

• MMseqs2 commit: d401e78c2d18a822cdb1527d7464a043f6035a15

• Build type: self-compiled, ASan

• OS: macOS 26.4.1, ARM64

• Apple Clang:

  Apple clang version 21.0.0 (clang-2100.1.1.101)
  Target: arm64-apple-darwin25.4.0
  

• CMake: 4.4.3

• Homebrew libomp: 23.1.1

• Homebrew prefix: /opt/homebrew/opt/libomp