Block Recursive Transformers for Structured Parameter Sharing

Abstract

Recursive Transformers depend not only on how many parameters are shared, but also on how shared parameters are organized across depth. Existing static schemes adopt two extreme reuse topologies: layer-wise repetition, where each layer is reused independently, and global cyclic reuse, where the entire stack is repeatedly applied as a single shared function. We argue that sharing granularity is an important factor in recursive parameter sharing. This study proposes Block Recursive Transformers (BRT), which partition layers into contiguous blocks and recursively reuse each block before moving to the next. This intermediate topology enables local refinement within blocks while preserving depth-wise transitions, without adding routing modules, auxiliary states, or extra training objectives. Across decoder-only models up to 1.7B parameters, with additional encoder-decoder and encoder-only studies, BRT improves validation negative log-likelihood and downstream accuracy over static recursive sharing baselines. Ablations and representation analyses further show that intermediate block granularity better balances local refinement and depth-wise differentiation.

Visualizations

Transformer depth is not a homogeneous sequence of interchangeable layers: lower layers tend to preserve local and lexical information, while upper layers integrate higher-level features and become more closely tied to prediction. This motivates a reuse unit that is larger than one layer but smaller than the full network. We use Centered Kernel Alignment (CKA) to examine how representations evolve under different reuse topologies. The visualization shows gradual changes in the vanilla Transformer, limited depth-wise differentiation under Sequence sharing, and saturation across recursion steps under Cycle sharing. BRT instead maintains higher similarity within each block and lower similarity across blocks, supporting local refinement while preserving transitions between depth regions.

CKA similarity heatmaps for Vanilla, Sequence, Cycle, and BRT models

Main Results

Under a fixed 20B-token training budget, BRT provides the strongest average few-shot accuracy among static recursive sharing strategies for SmolLM-360M with both two and four recursions. It remains competitive with the vanilla model while using substantially fewer unique parameters.

Model	Recursions	Sharing	Parameters	Tokens	ARC	WG	MMLU	HS	PIQA	Average
Vanilla	—	—	315M	20B	34.87 ± 0.31	50.92 ± 0.41	25.12 ± 0.17	28.32 ± 0.16	59.45 ± 0.32	39.84 ± 0.14
Recursive	2	Sequence	157M	20B	32.90 ± 0.40	50.03 ± 0.42	24.88 ± 0.34	27.51 ± 0.22	59.61 ± 0.31	38.98 ± 0.30
		Cycle	157M	20B	34.12 ± 0.50	50.37 ± 1.13	24.39 ± 0.26	28.37 ± 0.10	59.90 ± 0.43	39.43 ± 0.20
		Middle-Sequence	167M	20B	32.96 ± 0.11	50.76 ± 0.34	25.21 ± 0.41	28.02 ± 0.34	59.17 ± 0.50	39.22 ± 0.23
		Middle-Cycle	167M	20B	34.73 ± 0.21	51.12 ± 0.63	25.12 ± 0.19	28.59 ± 0.12	59.23 ± 0.10	39.76 ± 0.12
		BRT (Ours)	157M	20B	35.48 ± 0.35	51.32 ± 0.64	25.46 ± 0.22	28.98 ± 0.07	59.74 ± 0.50	40.20 ± 0.23
Recursive	4	Sequence	78M	20B	32.03 ± 0.33	50.83 ± 1.08	24.38 ± 0.21	27.34 ± 0.22	57.88 ± 0.28	38.49 ± 0.26
		Cycle	78M	20B	33.48 ± 0.23	50.22 ± 1.04	25.51 ± 0.18	28.46 ± 0.10	58.60 ± 0.35	39.25 ± 0.13
		Middle-Sequence	98M	20B	32.56 ± 0.54	51.79 ± 0.52	24.90 ± 0.32	27.62 ± 0.46	58.61 ± 0.53	39.09 ± 0.38
		Middle-Cycle	98M	20B	33.67 ± 0.23	51.89 ± 0.59	25.18 ± 0.23	28.47 ± 0.23	59.55 ± 0.24	39.75 ± 0.13
		BRT (Ours)	78M	20B	34.43 ± 0.09	51.91 ± 0.41	25.32 ± 0.16	28.73 ± 0.35	59.43 ± 0.21	39.96 ± 0.10

Across our small-model experiments, BRT scales consistently with additional training data for both two and four recursions. On SmolLM-135M, BRT improves over the vanilla model at 20B tokens under both recursion settings. On SmolLM-360M, it outperforms the static recursive baselines across training-token budgets and remains competitive with the vanilla model, especially with two recursions. These results show that block-level reuse scales reliably across small model sizes, data budgets, and different recursion numbers.

Average few-shot accuracy across model sizes, recursion settings, and training-token budgets

Block Ablation

We study how the number of recursive blocks affects performance, both when the first and last layers are kept independent and when all layers participate in recursion. Across these configurations, two blocks achieve the strongest average performance in our experiments. This suggests that a moderate block size may offer a useful balance between coordinating transformations within a block and preserving distinct stages across depth. The fully recursive setting follows a similar trend, indicating that the observed benefit is associated with block-level organization as well as the treatment of boundary layers.

Blocks (S)	Fully recursive	Sharing	Parameters	Tokens	ARC	WG	MMLU	HS	PIQA	Average
1	Yes	BRT	157M	10B	31.35 ± 0.39	50.75 ± 0.59	24.93 ± 0.26	27.33 ± 0.13	58.01 ± 0.16	38.47 ± 0.24
2		BRT	157M	10B	32.10 ± 0.35	50.31 ± 0.26	24.74 ± 0.22	27.70 ± 0.11	59.05 ± 0.75	38.77 ± 0.24
3		BRT	147M	10B	32.08 ± 0.36	50.31 ± 0.36	24.53 ± 0.31	27.54 ± 0.13	57.92 ± 0.31	38.47 ± 0.04
4		BRT	157M	10B	32.07 ± 0.17	49.40 ± 0.53	24.35 ± 0.29	27.49 ± 0.11	58.48 ± 0.32	38.34 ± 0.07
16		BRT	157M	10B	30.42 ± 0.50	51.22 ± 1.17	24.71 ± 0.19	26.79 ± 0.20	57.20 ± 0.20	38.09 ± 0.32
1	No	BRT	167M	10B	31.16 ± 0.20	51.85 ± 1.09	24.70 ± 0.19	27.39 ± 0.16	57.65 ± 0.44	38.55 ± 0.28
2		BRT	157M	10B	31.95 ± 0.45	51.57 ± 0.16	24.84 ± 0.21	27.47 ± 0.08	58.94 ± 0.53	38.95 ± 0.07
3		BRT	167M	10B	31.92 ± 0.38	50.76 ± 0.37	24.96 ± 0.36	27.78 ± 0.08	58.44 ± 0.13	38.76 ± 0.08
4		BRT	138M	10B	31.47 ± 0.32	50.60 ± 0.71	24.68 ± 0.27	27.23 ± 0.13	58.13 ± 0.44	38.42 ± 0.17
15		BRT	167M	10B	30.31 ± 0.58	51.28 ± 1.16	24.23 ± 0.17	27.37 ± 0.04	57.18 ± 0.68	37.90 ± 0.29

Architecture Generalization

To test whether block-recursive sharing generalizes beyond decoder-only models, we evaluate BRT on the T5-220M encoder-decoder architecture and the ModernBERT-149M encoder-only architecture. For T5, BRT achieves the lowest validation negative log-likelihood when recursion is applied to the encoder, the decoder, or both stacks, with decoder-only recursion reaching 5.52. For ModernBERT, BRT reaches 2.16, improving over Middle-Cycle at 2.20 and Middle-Sequence at 2.26. These results show that the benefit of intermediate block-level reuse extends across different Transformer architecture families.

Validation negative log-likelihood for BRT, Middle-Sequence, and Middle-Cycle on T5 and ModernBERT

BibTeX

BibTeX citation coming soon.

Research Collaborations

Beyond this work, Josep Marques is interested in research collaborations in multimodal learning, particularly on efficiency and Embodied AI. For potential collaborations and discussions, please contact jmarques01@skku.edu.