IntBMoE: 전체 참여형 Mixture-of-Experts를 위한 전문가 구성 내 블록 수준 조건화 통합
기존 Mixture-of-Experts (MoE) 설계는 participation, execution, materialization이라는 세 가지 요소를 독립적으로 제어하지 못하는 한계가 있습니다. 본 논문은 dense expert composition과 sparse block execution을 결합하여 이 세 요소를 분리하는 IntBMoE 구조를 제안합니다. IntBMoE는 작은 learned codebook에서 추출된 block을 사용하여 효율적인 expert 구성을 가능하게 합니다.
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into