[Usage]:Can the GPT attention plugin be used for self-attention in the encoder?

[0Unicorn]

System Info

System Information:

• CUDA version:13.0
• GPU model(s):NVIDIA L20
• TensorRT-LLM version:1.2.0

How would you like to use TensorRT-LLM

My model's encoder needs to use rope and sliding window, but it seems the BERT attention plugin doesn't support these features. Can the GPT attention plugin be used for the encoder model? If not, how can this problem be solved?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

You're correct that the BERT attention plugin does not support RoPE or sliding window attention. It only supports learned absolute position embeddings and standard bidirectional (padding) attention masks. There was no RoPE when BERT model was introduced.

The GPT attention plugin does support both RoPE and sliding window, but it's designed for autoregressive decoders and requires a KV cache — which encoder models don't use. Adapting it for an encoder would require setting up the KV cache infrastructure unnecessarily, adding significant complexity and memory overhead. So I'm not sure if that's a practical approach.

Are there any publicly available models similar to yours?

[0Unicorn]

@karljang Google's new Google/T5Gemma-B-B-PrefixLM model, based on the enc-dec architecture, incorporates new features like ropes and sliding windows in its encoder, causing incompatibility with the BERT attention plugin.

Currently, I see the GPT attention plugin supports use_cache=False, but I'm unsure if this is a viable solution.

Without using the plugin, the sliding window functionality is unavailable.

[karljang]

STDiT model (tensorrt_llm/models/stdit/model.py) uses the GPT attention plugin (Attention layer) with kv_cache_type=KVCacheType.DISABLED for its cross-attention (see STDiTCrossAttention at line 389 and prepare_attention_inputs at line 1359). While STDiT uses this for cross-attention rather than self-attention, it demonstrates that the GPT attention plugin works correctly with use_cache=False and no KV cache.

Hope this helps! Let us know how it goes.

However, please be informed that our team has been concentrating our investment on the PyTorch workflow and decoder-only models. Consequently, the TensorRT backend and enc-dec models will have limited support. 😄

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.

[0Unicorn]

Does the sliding window capability provided by the GPT attention plugin support bidirectional sliding? The official documentation mentions maintaining a KV cache of the past n tokens, rather than bidirectional sliding in the encoder.

[karljang]

Oh, I missed your question :)

The GPT attention plugin's sliding window is causal only. It implements a cyclic KV cache that retains the most recent N tokens, not a bidirectional window. So it won't give you the symmetric sliding window behavior an encoder needs (N tokens before AND after each position).

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.