Next-Concept Modeling: Implications for GenAI At Scale

The key constraint on generative AI is no longer intelligence. It is the cost of serving it. A change in the unit of generation moves that ceiling, and it can arrive before anyone notices.

The constraint has moved

For most of the last three years, the question people asked about generative AI was: can the model do it? That question is largely answered. The models are capable enough that the bottleneck for most applications has drifted to a much more mundane place: what does it cost to serve one to a customer who expects an answer in seconds?

The model you can run is the model you can afford to serve. That second clause is doing more work than it looks. Deployment is not a one-time capital decision. It is a recurring bill that scales with every user, every request, and every token of output. And that bill is set by two constants that have barely moved:

• generation is sequential
• the unit of generation is a token.

Those two constants are what make generative AI expensive to deploy. So the interesting question is not which model is smarter. It is what happens to the cost curve when the constants change.

The two things that set the bill

Almost every model in production predicts the next token, appends it, and repeats. That has two costs that compound at scale.

The first is that generation is sequential and length-bound. Cost grows linearly with output length, and you cannot parallelise the loop, because each token depends on the one before it. As inference became the dominant computational workload, this loop became the expensive part of the stack. It is why people built memory-efficient serving like PagedAttention (Kwon et al., 2023), and why transformer inference is a research target in its own right (Pope et al., 2022). We have spent years optimising the same loop.

The second is that a token is not a meaningful unit. A token is an artefact of a tokenizer, a compromise between vocabulary size and sequence length. It is not a concept, and it is not reliably a word. So a model is asked to build meaning out of pieces that carry none, and to commit to every intermediate one to reach the end of a thought.

If you want to know why generative AI costs what it costs, these two are the reason.

Two roads to a solution

Recent work splits cleanly along those two lines.

The first strategy makes the same loop cheaper. Diffusion language models generate the whole sequence at once and refine it over steps (Sahoo et al., 2024; Gu et al., 2024), and this has left the lab: Mercury is a commercial ultra-fast diffusion model (Khanna et al., 2025), and DiffusionGemma was developed by Google (DiffusionGemma Team, 2026). Speculative decoding and Medusa quicken the loop itself by drafting several tokens at once (Leviathan et al., 2023; Cai et al., 2024). All of this is real progress, and all of it keeps the token as the unit. It makes the same computation cheaper... one token at a time.

The second approach changes what a step is. Large Concept Models model language in a sentence-embedding space (Yu et al., 2024). A 2026 paper pushes that to predicting the next concept in a discrete latent space. Dynamic Large Concept Models make that space adaptive (Qu et al., 2025). These change the unit of generation, and most of them stay sequential.

One term runs through all of this, and it is worth being precise. "Concept" is used differently by each of these groups. Large Concept Models, the discrete next-concept paper, Dynamic Large Concept Models and Next-Concept Modeling all mean something different by it. When someone praises "concept-level" work, the claim is a design choice, not a shared definition.

The one that does both

Next-Concept Modeling (NCM) is a recent approach that aims to resolve both objectives at once (Kumbong et al., 2026). It is a two-stage, coarse-to-fine generator. A multiscale residual VQ-VAE compresses text into a hierarchy of discrete codes, building on the vector-quantised autoencoder (van den Oord et al., 2017) and residual quantisation (Lee et al., 2022). Coarse levels capture global structure; fine levels capture detail. A transformer then generates one level at a time, emitting every code within a level in parallel.

It changes the unit of generation, to a hierarchy of discrete concepts, and it changes the schedule, generating each scale in parallel. The name is the argument: next-token prediction replaced by next-concept prediction, coarse concepts first, then the fine facts that realise them.

On the deployment question, the figures that matter are the ones that hit the bill. The authors report 10 to 31 times faster wall-clock generation than the strongest diffusion baselines at matched or smaller parameter scale, and roughly 4.9 times fewer training tokens to reach its results. Those are the authors' measurements, run on their own hardware, yet to be verified, yet these results serve as an intriguing look at what is possible as this content generation approach matures.

The mechanism is what makes it interesting, because it is not just faster. Generating fewer, larger steps in parallel means less sequential decode and less retained state per output. That is the part that touches the hosting question: the same VRAM budget buys more capability per step, or the same capability with fewer resources. And it is not trading quality for speed. On Sudoku it beats both autoregressive and diffusion baselines at every difficulty..

Why this matters to a deployment decision

Two consequences follow, and both are about the bill.

The first is hosting economics. If capability per unit cost moves, you can host a bigger model in the same budget, or a smaller one that does the same job. For anyone serving at scale, that is not a footnote. It is the difference between a product that is economic and one that is not, changing what you can afford to ship.

The second is that it moves the local case. The footprint that fits on your own hardware starts to hold usable quality. That matters for anyone who cannot or will not route the whole workload through a cloud. It is a business case that flips before the benchmarks notice, because it depends on the shape of the model, not the score.

Current limitations

The obvious pushback is fair, one 100M-parameter prototype is difficult to extrapolate into future capability and deployment economics. Hardware gets cheaper every year, and quantization keeps improving, so the cost curve was already falling. Why should a single paper with a rough architecture be the thing that moves it?

The reply is that current innovations may run into future limitations. Quantization and better serving squeeze the same sequential, token-at-a-time loop. They make a fixed loop cheaper. They do not change the loop. The two things that actually set the bill, that generation is sequential and that a token is a poor unit, are properties of the representation and the schedule. NCM is one instance of changing those. Whether the hierarchy holds at 10B or 100B is genuinely unknown, and it is a thing worth watching.

The pace is the point

The reason to care is not that this specific paper will turn out to be the winner. It is the rate at which ideas like it get taken up.

A first paper is a seed, not a product. In open-source, a promising architecture gets forked, varied and re-implemented within days. We have seen this with our own Jev work. Jev is a class of model that returns typed decisions instead of prose. We adopted it for routing, and it collapsed a routing decision from roughly 6 to 10 seconds of sequential text-LLM calls to a few hundred milliseconds in one parallel pass, about 12 to 20 times faster and at a fraction of the cost. It is now doing evidence reranking too. That is not a hypothesis. It is running in production, and it is published.

There is no read on how fast an architecture like NCM evolves beyond its current form, and judging the idea on its first instantiation is a category error. Previous iterations of new model architectures get adopted fast. So the signal is not the v1 result. It is the rate at which the class gets varied and scaled.

What I would watch

If I were making a deployment decision, I would not bet the stack on NCM as it stands. I would watch whether the coarse-to-fine structure holds as it scales, and whether any variant of it gets adopted in a serving stack. The brittleness is real. Compressing language into discrete codes is less forgiving than images, because one bad code can break meaning outright. That is where the bet gets tested.

But the direction, that the unit and schedule of generation set the cost of deployment, is the part I would plan around. The cost of generative AI is not a constant. It is a variable that can move fast, and it does not announce itself.

References

1. Kumbong, Khare, Mlauzi, Woen, Ding, Buchanan, Arora, Mirhoseini, Ré. "Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation." arXiv 2610.14162, October 2026. https://arxiv.org/abs/2610.14162
2. Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention." 2023. https://arxiv.org/abs/2309.06180
3. Pope, Douglas, Chowdhery, Devlin, Bradbury, Levskaya, Heek, Xiao, Agrawal, Dean. "Efficiently Scaling Transformer Inference." 2022. https://arxiv.org/abs/2211.05102
4. Sahoo, Arriola, Schiff, Gokaslan, Marroquin, Chiu, Rush, Kuleshov. "Simple and Effective Masked Diffusion Language Models." 2024. https://arxiv.org/abs/2406.07524
5. Gu, et al. "Scaling Diffusion Language Models via Adaptation from Autoregressive Models (DUO)." 2024. https://arxiv.org/abs/2410.17891
6. Khanna, et al. "Mercury: Ultra-Fast Language Models Based on Diffusion." 2025. https://arxiv.org/abs/2506.17298
7. DiffusionGemma Team. "DiffusionGemma Technical Report." 2026. https://arxiv.org/abs/2608.00146
8. Leviathan, Kalman, Matias. "Fast Inference from Transformers via Speculative Decoding." 2023. https://arxiv.org/abs/2211.17192
9. Cai, et al. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads." 2024. https://arxiv.org/abs/2401.10774
10. Yu, et al. "Large Concept Models: Language Modeling in a Sentence Representation Space." 2024. https://arxiv.org/abs/2412.08821
11. "Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models." 2026. https://arxiv.org/abs/2602.08984
12. Qu, et al. "Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space." 2025. https://arxiv.org/abs/2512.24617
13. van den Oord, Vinyals, Kavukcuoglu. "Neural Discrete Representation Learning." 2017. https://arxiv.org/abs/1711.00937
14. Lee, Kim, Kim, Cho, Han. "Autoregressive Image Generation using Residual Quantization." 2022. https://arxiv.org/abs/2203.01941

Internal sources (published): How We Built Jev Into Our AI Routing (20 Oct 2026); How We Built Jev Into Our Evidence Reranking (3 Nov 2026). Both on the Mirror Ai Knowledge Hub.

Disclaimer: Not financial advice. This article was produced by Mirror Ai's AI Writer.