Skip to content

[Feature] Add query-chunked TileLang DSA selector - #2068

Open
Denny991 wants to merge 3 commits into
InternLM:mainfrom
Denny991:glm52-indexer-query-chunk
Open

[Feature] Add query-chunked TileLang DSA selector#2068
Denny991 wants to merge 3 commits into
InternLM:mainfrom
Denny991:glm52-indexer-query-chunk

Conversation

@Denny991

@Denny991 Denny991 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The GLM-5.2 DSA Indexer selector originally processed all query tokens in a
single TileLang launch. For long sequences, this creates a large temporary
dense score buffer and increases peak GPU memory usage.

This PR introduces query chunking for the selector. Query tokens are processed
in independent chunks, while the fixed-ID Top-K output semantics remain
unchanged.

Changes

  • Add optional query_chunk_size support to the DSA selector interface.
  • Process TileLang selector queries chunk by chunk.
  • Preserve packed-sequence causal ranges for every query chunk.
  • Safely pad non-aligned query tails for TileLang kernel constraints.
  • Keep only the final contiguous torch.int32 Top-K ID tensor.
  • Use an 8K query chunk by default in the GLM-5.2 recipe.
  • Keep the PyTorch selector unchanged.
  • Keep the existing SparseMLA and cuDNN forward/backward kernels unchanged.
  • Add unit tests for chunking, tail padding, short sequences, and one-shot
    versus chunked Top-K parity.

Usage

The GLM-5.2 recipe uses an 8K query chunk by default.

To select a custom chunk size:

export INDEXER_TOPK_QUERY_CHUNK_SIZE=2048

To disable query chunking:

export INDEXER_TOPK_QUERY_CHUNK_SIZE=0

Low-level selector APIs still use None as their default, so existing callers
are not changed implicitly.

Behavior

  • Query length smaller than 8K is processed in a single chunk.
  • Query length larger than 8K is split into multiple query chunks.
  • The final output keeps the same shape and dtype:
    (seq_len, 1, topk) and torch.int32.
  • Invalid causal positions remain represented by -1.
  • The chunk size only limits temporary selector workspace; it does not change
    the selected Top-K IDs.

Memory Impact

The temporary dense selector score buffer changes from [S, S_k] to
[C, S_k], where C is the query chunk size.

For S = S_k = 16K:

Query mode Chunk size Dense score workspace
One-shot None approximately 1.00 GiB
Chunked 8192 approximately 512 MiB
Chunked 2048 approximately 128 MiB

These numbers describe the selector workspace only. Total model peak memory
also includes activations, communication buffers, and other temporary tensors.

Validation

  • Targeted selector and attention tests passed on 8× NVIDIA H200.
  • One-shot and chunked selectors produced identical Top-K IDs on packed inputs.
  • Non-aligned query tails and short-query cases passed.
  • Reduced-model GLM-5.2 E2E training completed with packed 16K input and a
    2K query chunk.
  • Losses remained finite, with no OOM, CUDA, NCCL, or traceback errors.
  • Python compilation and git diff --check passed.

Limitations

  • Query chunking is implemented for the TileLang-backed DSA selector.
  • The DSA loss implementation is not modified.
  • Existing GLM-5.2 batch-size and parallelism constraints remain unchanged.

@Denny991
Denny991 force-pushed the glm52-indexer-query-chunk branch from 97e4765 to 9366a49 Compare September 4, 2026 08:40
@Denny991
Denny991 force-pushed the glm52-indexer-query-chunk branch from 9366a49 to c4d7169 Compare September 4, 2026 09:00
Satisfy formatting checks and make the distributed MoE accuracy test deterministic.
@Denny991
Denny991 force-pushed the glm52-indexer-query-chunk branch from b13ba28 to b2c3c59 Compare September 4, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant