Skip to content

add MMLU benchmark val / test - #1

Merged
LZHgrla merged 11 commits into
InternLM:mainfrom
LZHgrla:lzh/add_mmlu
Jul 14, 2023
Merged

add MMLU benchmark val / test#1
LZHgrla merged 11 commits into
InternLM:mainfrom
LZHgrla:lzh/add_mmlu

Conversation

@LZHgrla

@LZHgrla LZHgrla commented Jul 13, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

MMLU benchmark val / test. This implementation follows the approach of the original implementation and QLoRA.

Model Split Method Original Impl. Ours
LLaMA 7B val zero-shot 32.3 33.3
val five-shot 33.6 33.2
test zero-shot 32.5 32.9
test five-shot 35.1 35.6

Note: We use the preprocessed MMLU dataset from QLoRA, instead of the original dataset.

@LZHgrla
LZHgrla requested a review from pppppM July 13, 2023 03:01
@LZHgrla
LZHgrla merged commit e840b6c into InternLM:main Jul 14, 2023
@LZHgrla
LZHgrla deleted the lzh/add_mmlu branch July 21, 2023 07:10
HIT-cwh added a commit that referenced this pull request Aug 28, 2023
@HIT-cwh @LZHgrla
[Doc] Add dataset pipeline doc & Improve Doc (#38)
569d916 
* add dataset pipeline doc

* add dataset pipeline doc

* fix bugs

* fix bugs

* refine doc

* fix bugs

* Update README.md

* Update README.md

* update docs (#1)

* Update README.md

* fix pre-commit

* rename xTuner to XTuner

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* fix pre-commit

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update finetune.md

* Update finetune.md

* Update chat.md

* fix pre-commit

* add zh_cn chat and finetune doc

* Update chat.md

* Update README.md

* del tool_usage

* Update README.md

* Update chat.md

* Update chat.md

* Update README.md

* Update README.md

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* fix pre-commit

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* refactor data pipeline doc

* add colorist llama2

* fix incremental pretraining doc

---------

Co-authored-by: LZHgrla <36994684+LZHgrla@users.noreply.github.com>
Co-authored-by: LZHgrla <linzhihao@pjlab.org.cn>
@apachemycat apachemycat mentioned this pull request Jun 26, 2024
Open
llkn-2 pushed a commit to llkn-2/xtuner that referenced this pull request Jul 31, 2024
@LZHgrla
add MMLU benchmark val / test (InternLM#1)
12b149b 
* add mmlu dataset configs

* add mmlu metric

* fix bugs

* implement predict for sft model

* remove dummy file

* clean code

* modify prefix for mmlu test

* add test.py

* add mmlu val/test for gunaco config

* use float16 for gunaco

* add METAINFO and add logger
llkn-2 pushed a commit to llkn-2/xtuner that referenced this pull request Jul 31, 2024
@HIT-cwh @LZHgrla
[Doc] Add dataset pipeline doc & Improve Doc (InternLM#38)
e56b62e 
* add dataset pipeline doc

* add dataset pipeline doc

* fix bugs

* fix bugs

* refine doc

* fix bugs

* Update README.md

* Update README.md

* update docs (InternLM#1)

* Update README.md

* fix pre-commit

* rename xTuner to XTuner

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* fix pre-commit

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update finetune.md

* Update finetune.md

* Update chat.md

* fix pre-commit

* add zh_cn chat and finetune doc

* Update chat.md

* Update README.md

* del tool_usage

* Update README.md

* Update chat.md

* Update chat.md

* Update README.md

* Update README.md

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* fix pre-commit

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* refactor data pipeline doc

* add colorist llama2

* fix incremental pretraining doc

---------

Co-authored-by: LZHgrla <36994684+LZHgrla@users.noreply.github.com>
Co-authored-by: LZHgrla <linzhihao@pjlab.org.cn>
@claude claude Bot mentioned this pull request Feb 28, 2026
Open
@claude claude Bot mentioned this pull request Apr 22, 2026
Merged
HAOCHENYE added a commit that referenced this pull request Jun 29, 2026
@HAOCHENYE
[Refactor] Consolidate V4 decoder into a single V4DecoderLayer class
395db1e 
The previous V4 layer was split across three classes that existed only to
work around an over-narrow abstraction:

  MoEDecoderLayer (parent)
       └── _V4InnerBlock      ← inherited but never used the parent's forward;
                                added attn_block / ffn_block / set_context
                                to fit an HCInnerBlock protocol
       └── HCDecoderLayer     ← "generic" wrapper with V4 as its only user;
                                ``attn_block(x) -> x`` couldn't carry DSA's
                                position_embeddings / seq_ctx / input_ids,
                                so set_context shoved them on the inner block
                                as mutable state
       └── _V4DecoderLayer    ← bridge that called set_context, called the
                                HC wrapper, then read the stashed router
                                results back off the inner block

Three classes, one ``set_context`` side-channel, one ``_last_router_results``
stash-and-grab, one ``assert hc_layer.inner is inner`` to police a
dual-registration invariant — to express what is conceptually one decoder
layer (HC residual mix + DSA + MoE FFN). Per CLAUDE.md rule #1: "do not
justify patch-on-patch, layered, spaghetti-like implementations in the name
of backward-compatibility protection". The "generic HC wrapper" abstraction
had exactly one user.

Replace all three with :class:`V4DecoderLayer` that owns every submodule and
parameter directly:

  V4DecoderLayer.forward(hidden_states,
                         *, position_embeddings,
                            position_embeddings_compressed,
                            seq_ctx, input_ids)
      -> (hidden_states_out, router_logits, router_weights)

All inputs flow through arguments; router results flow out through the
tuple; no hidden state on ``self`` between calls. The compile-target
sub-methods (``_attn_compute`` / ``_ffn_pre_compute`` / ``_ffn_post_compute``
/ ``_shared_experts_forward``) stay as separate methods so
``V4_EP_COMPILE_CFG`` can target them individually with the same boundary
the previous ``_V4InnerBlock`` exposed.

Knock-on changes:

* ``hc_block.py`` keeps ``hc_pre`` / ``hc_post`` / ``HCWrapperConfig`` /
  ``_unshard_hc_params`` (the math + DTensor helper) but deletes the
  ``HCDecoderLayer`` class and ``HCInnerBlock`` protocol.
* ``decoder_layer/__init__.py`` re-exports drop ``HCDecoderLayer`` and
  ``HCInnerBlock``.
* ``DeepSeekV4._build_one_layer`` constructs ``V4DecoderLayer`` directly
  (was: ``_V4InnerBlock`` + ``HCDecoderLayer`` + ``_V4DecoderLayer``).
* ``_translate_layer_tail`` no longer strips ``hc_layer.`` or
  ``hc_layer.inner.`` prefixes — params now arrive at the flat
  ``layers.L.hc_attn_*`` / ``layers.L.input_layernorm.weight`` /
  ``layers.L.experts.*`` layout.
* ``V4_EP_COMPILE_CFG`` / ``V4_NON_EP_COMPILE_CFG`` point at
  ``V4DecoderLayer._attn_compute`` / ``_ffn_pre_compute`` /
  ``_ffn_post_compute`` (was: ``_V4InnerBlock.attn_block`` / equivalents).
* ``tests/module/test_hc_block.py`` no longer exercises a (now-deleted)
  wrapper class; it tests ``hc_pre`` / ``hc_post`` directly with the same
  closed-form assertions on the degenerate-init path.

Verified: 51 module + V4 model tests pass (the two ``to_hf_key_list_coverage``
checks that need a local BF16 checkpoint are skipped). Sample of the new
flat param layout under a 2-layer toy model::

    hc_head_fn                                -> hc_head_fn
    layers.0.hc_attn_fn                       -> layers.0.hc_attn_fn
    layers.0.input_layernorm.weight           -> layers.0.attn_norm.weight
    layers.0.post_attention_layernorm.weight  -> layers.0.ffn_norm.weight
    layers.0.self_attn.wq_a.weight            -> layers.0.attn.wq_a.weight
    layers.0.experts.fused_w1w3.weight        -> [N × layers.0.ffn.experts.i.{w1,w3}.weight]
    layers.0.shared_experts.gate_proj.weight  -> layers.0.ffn.shared_experts.w1.weight

No HF-side semantics change — the HF key bridge produces the same target
names as before; only the XTuner-side path stripping is simpler.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pppppM pppppM Awaiting requested review from pppppM

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LZHgrla