feat(arm): complete 1x4_32W NEON kernel, auto-detect Android Termux, and add Galaxy benchmarks - #624
Open
UnoKim (uno-km) wants to merge 10 commits into
Open
Conversation
Changes: Unified QK Standard: Strictly enforced QK_I2_S = 128 across NEON and Scalar paths to match the standard GGUF packing layout. Refactored Loop Logic: Removed legacy group32_num and la_num chunks. Replaced with a clean, block-level loop to prevent pointer corruption. NEON Optimization: Implemented a dual 16-byte chunk load strategy within the 32-byte weight block to maximize SIMD register utilization. Mathematical Alignment: Synchronized bit-unpacking order (MSB to LSB) with the AVX2 reference. Implemented 32-stride interleaved memory fetching for activations (y). Removed redundant (-1) offset mapping to leverage zero-mean distribution properties, matching the high-performance AVX2 kernel behavior. Result: Completely resolved the word salad issue on Exynos/Snapdragon chips. Validated logical consistency across AVX2, NEON, and Pure C++ Scalar fallback paths.
…g QK layout to 128 - Standardized `QK_I2_S` to 128 for `__ARM_NEON` to match the x86 GGUF packing standard. - Fixed memory misalignment in `quantize_i2_s` by updating the packing stride to 32. - Refactored `ggml_vec_dot_i2_i8_s` NEON kernels (1x1, 1xN, Nx1) to use a dynamic block-level loop (`nb = n / QK`) instead of hardcoded 64-stride loop unrolling. - Aligned interleaved memory fetching (`vld1q_s8`) with the AVX2 logic. - Upgraded accumulator horizontal sum to `vaddlvq_s32` (64-bit) to prevent potential 32-bit integer overflow in extended context scenarios. Tested on Exynos 1380 (Android PRoot) with `-t 8`. Output generation is now 100% stable without word salad.
…in setup_env, and add on-device Galaxy benchmarks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR provides four key enhancements for the ARM / Mobile inference ecosystem in
BitNet:__ARM_NEONpath forggml_vec_dot_i2_i8_s_1x4_32Winsrc/ggml-bitnet-mad.cpp. Processes 4 parallel rows against shared activation vectors using QK=128 layout and ARMv8.2-Asdothardware dot-product acceleration.setup_env.py): Added auto-detection for Android / Termux environments to automatically inject-DGGML_NEON=ON -DGGML_ARM_DOTPROD=ONCMake flags, enabling seamless out-of-the-box build on mobile ARM devices.run_inference.py): Prepended local build directories toLD_LIBRARY_PATHduring inference invocation to prevent dynamic linker collision with system libraries in mobile Linux / Termux environments.On-Device Benchmark Results (BitNet b1.58 2B-4T i2_s)
Tested natively inside Android Termux with 4 worker threads:
Real-Device Verification Workflow (Samsung Galaxy A35 5G)
📸 Click to toggle step-by-step verification screenshots
1. Termux environment login & hardware check
2. Branch checkout (
feat/arm-neon-tiling-and-termux)3. Automatic environment setup & model quantization prep
Command:
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s4. CMake compilation & GGUF model preparation
Auto-detected Termux environment and applied
-DGGML_NEON=ON -DGGML_ARM_DOTPROD=ON:Native build execution (
cmake --build build --target llama-cli -j6):5. Verified Real-time On-Device Inference
Running 2B-4T i2_s model directly on Exynos 1380 with valid token completion:
Changes Summary
src/ggml-bitnet-mad.cpp: Added__ARM_NEONloop with vector load andvdotq_s32/vmlal_s8accumulation.setup_env.py: Added Termux check (com.termuxprefix detection) and ARM NEON CMake flags injection.run_inference.py: Added explicit dynamic library search path isolation for Android/Linux.README.md: Documented Termux prerequisite packages and on-device performance table.