This article is for informational purposes only and is not an endorsement to adopt any specific technology or product.
관련해서 SKT’s A.X K2, another efficient MoE도 함께 참고하시면 좋습니다.
관련해서 Meta’s open-weight Muse Glimmer도 함께 참고하시면 좋습니다.
The Story
Here’s a problem that’s been quietly bugging the people who build large language models. The trick everyone reached for to make models bigger without making them slower — “Mixture-of-Experts,” or MoE — has a nasty side effect. It bloats.
Let me back up and explain MoE in plain terms, because the whole story hinges on it. A normal neural network runs every input through every parameter. That gets expensive fast as models grow. MoE breaks the model into a bunch of specialized sub-networks called “experts,” and a little traffic cop called the “router” sends each token to only a few of them. So the model can hold a huge number of parameters, but any single token only wakes up a small slice of them. Big brain, cheap per-token compute. That’s why DeepSeek, Mixtral, and most of the frontier labs lean on it.
The catch: every layer of the network gets its own private set of experts. Stack a hundred layers, each with its own thousand experts, and you’re carrying a hundred thousand experts around in memory — even though a lot of them end up learning nearly the same thing. Redundant parameters, ballooning file size, memory you have to pay for whether or not you ever route a token through it.
A team at UNIST (the Ulsan National Institute of Science and Technology) in Korea took a swing at exactly this. Professor Kim Taehwan’s group, with master’s student Hong Gunwoo as first author, built a structure they call “GMoE” — Global Mixture-of-Experts. The paper, titled “GMoE: Global Mixture of Experts with Logit Propagation,” was accepted at ACL 2026, which is about as prestigious as natural-language-processing venues get.
The idea is almost embarrassingly simple once you hear it. Instead of giving every layer its own private stash of experts, GMoE keeps one big pool of “shared experts” that all layers can reach into, plus a small handful of “dedicated experts” — one per layer — for whatever’s genuinely local to that layer. So in their own example: instead of 100 layers each holding 1,000 experts (100,000 total), you keep 1,000 shared experts plus 100 dedicated ones. Roughly 1,100 instead of 100,000.
The numbers they report are the interesting part. On a mid-sized model, GMoE shrank the parameter count from 549 million down to 204 million — about a 63% cut — while average accuracy barely moved: 39.51% versus 39.55% for the baseline. All figures are the researchers’ own reported results, and the team says it tested the approach across small, medium, and large models.
There’s a second result that I think is actually the more interesting one, and it’s easy to skim past. MoE routers have a chronic disease called “routing collapse.” The router gets lazy and keeps shoveling tokens to the same few favorite experts, so a big chunk of your expensive experts sit idle and never really learn anything. GMoE reportedly spread the load out a lot better — the team says the number of distinct routing paths actually used jumped past 81,000, more than triple the conventional setup, and the concentration on any single path dropped from a range of roughly 25–46% down to about 11%. In other words, more of the experts were pulling their weight.
Part of how they get there is a piece the paper calls “logit propagation.” Rather than letting each layer’s router decide in isolation, a shared GRU-based global router passes its routing logits forward from layer to layer, so the choices stay coordinated across the whole stack instead of drifting into the same few experts independently.
The Takeaway
The reason this one caught my eye isn’t the headline “63% smaller.” It’s where the saving comes from.
Most of the efficiency stories I’ve been writing about lately attack the problem from the outside. Liquid AI’s tiny edge models shrink the whole model down so it fits on a phone. Thinking Machines’ Inkling-Small trims the footprint to a quarter. And on the hardware side, the real bottleneck for AI has quietly become memory, not raw compute. GMoE is playing on the architecture level — it’s saying the standard MoE design was wasteful to begin with, holding redundant copies of experts it didn’t need. Fix the design, and you get most of the shrink for free, before you ever touch quantization or distillation.
That distinction matters because these approaches stack. A leaner architecture, then a smaller footprint, then better memory hardware underneath — they’re not competing, they’re layers of the same fight. And the fight is all about one thing: fitting more capability into less memory. That’s the through-line connecting basically every efficiency story worth reading right now.
I’d flag two honest caveats. First, the accuracy numbers here (~39%) are low in absolute terms, which tells you these are relatively small research-scale models on a benchmark, not GPT-class systems — the claim is “same accuracy, 63% smaller,” not “state of the art.” Whether the trick holds its shape at frontier scale is the open question, and it’s the question that decides whether anyone in industry actually adopts it. Second, “shared experts” isn’t a UNIST invention in the abstract — DeepSeek and others already use a shared-expert idea to catch common knowledge. What’s new here is pushing the sharing across layers globally, plus the reported win on routing diversity.
There’s also a Korean angle worth naming. A lot of the marquee MoE work — DeepSeek, Mixtral — comes out of large industry labs sitting on enormous compute. GMoE is an academic result out of a Korean university graduate program, first-authored by a master’s student, landing at a top-tier venue. Architecture research is one of the few corners of AI where you don’t need a warehouse of GPUs to say something that matters, and this is a clean example of that. My read: keep an eye on whether a bigger lab picks up the cross-layer sharing idea and scales it. That’s usually the tell for whether an academic trick has legs.
This article is for informational purposes only and is not an endorsement to adopt any specific technology or product.
Photo: Growtika / Unsplash
댓글 남기기