レイヤーは1つで十分か?単一のTransformerレイヤーがフルパラメータRLトレーニングに匹敵
Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train (arxiv.org)
要約
本研究では、大規模言語モデル(LLM)の強化学習(RL)後学習における各Transformerレイヤーの寄与を体系的に調査しました。 驚くべきことに、単一のTransformerレイヤーのトレーニングで、フルパラメータRLトレーニングと同等以上の成果が得られることが判明しました。この発見は、RL適応がモデルの特定の部分に集中していることを示唆しており、効率的なモデル調整の可能性を示しています。
全文翻訳
Computer Science > Machine Learning
arXiv:2607.01232 (cs)
[Submitted on 1 Jul 2026 (v1), last revised 2 Jul 2026 (this version, v2)]
Title:Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Authors:Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong
View a PDF of the paper titled Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training, by Zijian Zhang and 6 other authors
Abstract:Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.
強化学習(RL)は、大規模言語モデル(LLM)の後学習における中心的な要素となっていますが、RLの適応がTransformerのどの層にどのように分散しているかについては、ほとんど理解されていません。既存のアプローチでは、通常、すべてのモデルパラメータを均一に更新しており、RLの後学習中に得られる利益にすべての層が同様に貢献していると暗黙的に仮定しています。本研究では、RLトレーニングの体系的な層ごとの研究を通じて、この仮定に異議を唱えます。驚くべきことに、単一のTransformer層をトレーニングするだけで、フルパラメータRLトレーニングで達成された利益の大部分を回復でき、場合によってはそれを上回ることさえあることがわかりました。この現象を定量化するために、層貢献度という量を導入しました。これは、単独で層をトレーニングすることによって回復された、完全なRL改善の割合を測定します。Qwen3、Qwen2.5の2つのモデルファミリー、GRPO、GiGPO、Dr. GRPOの3つのRLアルゴリズム、および数学的推論、コード生成、エージェント意思決定を含む複数のタスクドメインにまたがる7つのモデル全体で、非常に安定したパターンが観察されます。RLの利益は、少数の、多くの場合単一のTransformer層に高度に集中しています。さらに驚くべきことに、同じ構造パターンが一貫して現れます。高貢献層はTransformerスタックの中央に集中し、入力端と出力端に近い層の貢献度は大幅に少なくなります。結果として得られる層のランキングは、データセット、タスク、モデルファミリー、RLアルゴリズム全体で強く相関したままです。
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.01232 [cs.LG] |
| (or arXiv:2607.01232v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01232 arXiv-issued DOI via DataCite |
Submission history
From: Zijian Zhang [view email]
[v1] Wed, 1 Jul 2026 17:59:54 UTC (268 KB)
[v2] Thu, 2 Jul 2026 17:28:23 UTC (268 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training, by Zijian Zhang and 6 other authors
[画像: license icon] view license
Additional Features
Current browse context:
cs.LG
Change to browse by:
References & Citations
Loading...
BibTeX formatted citation
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)
- Author
- Venue
- Institution
- Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)