Tag: Apache 2.0

  • K2 Horizon Analysis: IFM’s Six-Model Fleet from 0.9B to 375B Released Simultaneously Under Apache 2.0

    K2 Horizon Analysis: IFM’s Six-Model Fleet from 0.9B to 375B Released Simultaneously Under Apache 2.0

    Key Summary

    • Release scale: Six models—375B-A23B, 36B-A4B, 32B, 7B, 3.7B, and 0.9B—all simultaneously distributed under Apache 2.0 on Hugging Face (the 0.9B uses a smaller vocabulary)
    • Open stack included: Beyond model weights, the release ships the pretraining corpus, intermediate checkpoints, training code, configuration files, and detailed logs—IFM describes this as “the largest fully open-source release in AI history”
    • Training data: All six models were pretrained on approximately 20 trillion tokens, of which about 17% consists of explicit reasoning traces, and roughly 10 trillion tokens are synthetic data

    An analytical article examining what it means—for technology, licensing, and commercialization—when a large language model is released as a full-stack open-source project, and what it implies in practice to run the same architecture consistently from 0.9B to 375B.

    Table of Contents

    K2 Horizon lineup

    The fact that K2 Horizon released six models ranging from 0.9B to 375B under Apache 2.0 on the same day forces us to revisit what it means for a large language model to be open-sourced as a full stack. The bundle published in September 2026 by the Foundation Model Institute (IFM) under MBZUAI is not a single model but a “fleet” that ships the pretraining corpus, intermediate checkpoints, training code, configuration files, and detailed logs together.

    In the author’s view, the core of this announcement lies not in model size or benchmark scores but in the scope of openness. The fact that even the 0.9B model’s use of a smaller vocabulary is explicitly stated means that the same training pipeline can be repeated by varying only the scale. The ability to use an identical interface across sizes in practice becomes a prerequisite for unified operations.

    K2 Horizon Lineup: From 0.9B to 375B-A23B

    The lineup is divided into six models. The 0.9B uses a reduced vocabulary, while the 3.7B, 7B, and 32B are dense. The 36B-A4B and 375B-A23B are MoE configurations. Because all models share the same tokenizer and tool-calling interface, the same code can be run by switching only the model identifier.

    Model Total Parameters Active Type
    0.9B 0.9B 0.9B Reduced-vocab dense
    3.7B 3.7B 3.7B Local-inference dense
    7B 7B 7B General-purpose dense
    32B 32B 32B High-quality generation dense
    36B-A4B 36B 4B MoVA MoE
    375B-A23B 375B 23B Flagship MoE

    Because all six K2 Horizon models pass through the same synthetic-task generator and are trained with the same reasoning-trace ratio (about 17%), data alignment is applied consistently across the entire lineup. There is a high probability that a prompt format validated on the 0.9B will not break significantly when ported to the 375B-A23B.

    The Scope of the Open Stack Brought by K2 Horizon

    IFM used the phrase “the largest fully open-source release in AI history.” The K2 Horizon distribution bundle includes the pretraining corpus, intermediate checkpoints, training code, configuration files, and detailed logs. Compared with other open-source projects that release only weights and inference code, the scope is on a different level.

    The release of intermediate checkpoints is directly tied to reproducibility. While retraining the 375B-A23B from scratch is difficult, you can continue fine-tuning from an intermediate stage or run ablations. In effect, the surface needed to experiment by varying the synthetic-data ratio or the reasoning-trace ratio is now secured.

    Training Data Design: 20 Trillion Tokens, 100M+ Tasks

    All six models were pretrained on approximately 20 trillion tokens, of which about 17% consists of explicit reasoning traces, and roughly 10 trillion tokens are synthetic data. The research team generated more than 100 million unique synthetic tasks and progressively incorporated post-training data during the middle-training stage.

    The point practitioners should pay attention to is the scale of the synthetic tasks—the most striking number in the author’s view. 100 million is far beyond what can be hand-designed by humans. The structure has shifted from defining domain tasks and feeding them into the model to letting the model itself generate tasks while humans curate them. When examining the performance gap between the 36B-A4B and the 7B, the distribution of these 100 million tasks becomes the decisive variable.

    MoVA: A Second MoE Scaling Axis Built on Attention

    Among the K2 Horizon lineup, the 36B-A4B applies MoVA (Mixture-of-Value Attention). If MoE created a scaling axis by routing experts in the feed-forward layer, MoVA integrates routing into multi-head attention itself, adding a second scaling axis along the attention dimension. It is compatible with FlashAttention, GQA, and sparse attention, so it can be used without significantly overhauling existing serving stacks.

    What makes MoVA interesting is that it can push MoE’s parameter efficiency a step further. Whether the upper K2 Horizon lineup converges entirely on MoVA, or dense and MoVA coexist, is a point to watch.

    Tool-Calling Format: Markdown’s 18.5% Token-Efficiency Edge

    IFM trained tool definitions in three formats—JSON, XML, and Markdown—and set Markdown as the default for inference. Measurement results show that Markdown uses approximately 18.5% fewer tokens than JSON. Producing the same result with fewer tokens translates directly into lower latency and cost.

    The Markdown advantage grows as tool definitions get longer. Agents that expose 30 to 50 tools simultaneously are common, and in those cases, 18.5% is not a simple optimization but a near architectural decision. When designing a tool-calling system with K2 Horizon, it is simpler to let the model read and call Markdown tool definitions directly rather than building a separate router.

    Serving Ecosystem: Day-0 Support for vLLM, SGLang, and Ollama

    On the day of release, vLLM, SGLang, and Ollama received day-0 support, with FP8 and GGUF builds also provided. The supported range extends beyond NVIDIA to include AMD and Cerebras. According to the official IFM announcement, hosting is split across Compass, Cerebras, and Nebius APIs, plus the platform.ifm.ai gateway, so the same weights can be tested immediately on various stacks. Deployment-infrastructure choice is worth evaluating from the perspective of data-movement costs by AI chip architecture.

    Day-0 serving support is a “signal” from the announcement. New models typically stabilize only after community patches, but K2 Horizon works on three inference engines at the moment of release. This shows IFM’s strategic choice to ship open-source models as a “stack.”

    Practical Application Points

    Size selection depends on call frequency and acceptable response latency. The 0.9B and 3.7B are efficient for classification, routing, and simple transformations; the 7B fits general-purpose reasoning; and the 32B is best for high-quality generation and summarization, where resource-to-output ratio matters. The 36B-A4B is well suited to agent routers or multi-tool-call workloads, leveraging its 4B active parameters per token. The 375B-A23B carries a significant operational cost, so the safer approach is to validate prompts and tool shapes on the 0.9B first, then scale up gradually.

    What to Try Right Now

    • Download the K2 Horizon 0.9B or 3.7B weights from Hugging Face and spin up a local vLLM server.
    • Write the same tool definition in both JSON and Markdown, and compare token counts and response times.
    • Pull the 36B-A4B and run at least five multi-tool-calling scenarios with MoVA routing enabled.
    • Inspect the synthetic-task distribution in the pretraining-corpus metadata, pick the cluster closest to your in-house domain tasks, and start fine-tuning.

    Frequently Asked Questions

    Why does the K2 Horizon 0.9B model use a different vocabulary?

    It is targeted at edge devices, so the vocabulary was reduced to lower memory usage. It goes through the same training pipeline but adopts a reduced vocabulary tailored to the inference environment.

    Can it be commercialized under the Apache 2.0 license?

    Because the weights, code, and data are all released under Apache 2.0, it can be used as-is in commercial services. Note, however, that the same license-notice requirement must be followed when redistributing.

    How is MoVA different from conventional MoE?

    If MoE routes experts in the feed-forward layer, MoVA integrates routing into multi-head attention itself. Its defining feature is compatibility with FlashAttention, GQA, and sparse attention.

    Why is Markdown tool calling 18.5% more efficient than JSON?

    Markdown uses fewer metacharacters such as braces, quotes, and tags, so fewer tokens are needed to express the same meaning. The advantage grows as tool definitions get longer.

    The question K2 Horizon’s release poses is not simply “Is this yet another open-source model launch?” The fact that a fleet running consistently on the same pipeline from 0.9B to 375B has been released under Apache 2.0 shows that the axis of open-source LLM competition is shifting from weights to the “stack”—including data, code, and intermediate states. What licensing standards this change produces, and how commercial vendors respond, will be the deciding factor over the coming quarters.

    Reference

    This article was written after reviewing the following source: MarkTechPost — IFM Releases K2 Horizon: Six Apache 2.0 Models From 0.9B to 375B

  • MiniCPM5-2B: OpenBMB’s 2.5B Model Hits 53.9 Average Across 34 Benchmarks, Redefining On-Device LLM Standards

    MiniCPM5-2B: OpenBMB’s 2.5B Model Hits 53.9 Average Across 34 Benchmarks, Redefining On-Device LLM Standards

    Key Summary

    • MiniCPM5-2B is a dense causal language model with 2,516,756,480 parameters (1,981,982,720 excluding embeddings), adopting 42 layers and grouped-query attention (16 query heads, 2 key/value heads).
    • The native context window is 131,072 tokens, and because the architecture is standard LlamaForCausalLM, it runs on vLLM, SGLang, Transformers, llama.cpp, Ollama, LM Studio, MLX, and FlagOS without custom kernels or model code forks.
    • Licensed under Apache 2.0, it is benchmarked against same-class peers such as LFM2.5-2.6B, Qwen3.5-2B, and Gemma-4-E2B-it, with references including Qwen3.5-4B (51.1), granite-4.2-3B (42.7), and LFM2.5-2.6B (33.2).

    Analysis

    Table of Contents

    MiniCPM5-2B debuted on September 7 with a spec sheet of 2,516,756,480 parameters and a native context window of 131,072 tokens. Its 53.9 average across 34 benchmarks is 2.8 points ahead of the best same-lineup baseline, Qwen3.5-4B at 51.1. With less than half the parameters, it posts a higher overall score. For the author, the real significance of this model lies in numerically proving that “small doesn’t mean incapable.”

    MiniCPM5-2B Architecture: Standard Llama Fork for Instant Compatibility with 8 Inference Engines

    MiniCPM5-2B is a dense causal language model with 42 layers and grouped-query attention (16 query heads, 2 key/value heads). Excluding embeddings, it has 1,981,982,720 parameters. The key point is that the architecture is standard LlamaForCausalLM. Eight inference engines—vLLM, SGLang, Transformers, llama.cpp, Ollama, LM Studio, MLX, and FlagOS—run it as-is, without custom kernels or model code forks. Combined with the Apache 2.0 license, this makes it one of the lowest-friction options for teams looking to deploy a 2–3B-class model in real services.

    What the 53.9 Average Across 34 Benchmarks Means for MiniCPM5-2B

    The single score of 53.9 alone doesn’t tell the full story. Broken down by domain, MiniCPM5-2B’s character becomes clearer. The same-class comparison set includes LFM2.5-2.6B, Qwen3.5-2B, and Gemma-4-E2B-it, while reference points include Qwen3.5-4B (51.1), granite-4.2-3B (42.7), and LFM2.5-2.6B (33.2). The overall 53.9 is the highest figure in this reference pool.

    Benchmark MiniCPM5-2B Best Baseline Gap
    34-benchmark overall average 53.9 Qwen3.5-4B 51.1 +2.8
    LiveCodeBench v6 69.1 56.4 +12.7
    SWE-bench Verified 46.4 33.6 +12.8
    τ²-Bench Telecom 97.1 — —
    BFCL v4 66.6 — —
    τ³-Bench Banking 20.8 6.8 +14.0
    NoLiMa 68.1 43.5 +24.6
    AA-LCR 59.0 61.0 −2.0
    LongBench v2 43.7 47.3 −3.6
    MMLU-Pro 70.8 78.0 −7.2
    Humanity’s Last Exam 8.9 9.9 −1.0

    Strengths: Dominating the Baseline in Code Reasoning and Tool Use

    The largest gap is in code. MiniCPM5-2B scored 69.1 on LiveCodeBench v6 and 46.4 on SWE-bench Verified, beating the baseline of 33.6 by 12.8 points. A 2.5B model hitting 46.4 on SWE-bench means it achieves nearly a 50% success rate on tasks that go beyond simple code completion to include multi-turn debugging.

    Tool use is even steeper. The τ²-Bench Telecom score of 97.1 is essentially near-perfect, and BFCL v4 at 66.6 and τ³-Bench Banking at 20.8 (baseline 6.8) also far exceed what you’d expect from a 2.5B-class model in function calling and routing. These numbers most directly demonstrate that MiniCPM5-2B is a credible candidate for real agent workloads.

    Trade-offs: Long-Context Variability by Benchmark and General-Knowledge Limits

    It doesn’t lead on every metric. Long-context results vary by benchmark. On NoLiMa (68.1 vs. baseline 43.5) it leads by 24.6 points, but on AA-LCR (59.0 vs. 61.0) and LongBench v2 (43.7 vs. 47.3) it actually trails. Even with a 131K-token window, there are areas where measured accuracy falls short of the baseline.

    General knowledge shows a similar pattern. MMLU-Pro (70.8 vs. 78.0) and Humanity’s Last Exam (8.9 vs. 9.9) are both slightly behind. It’s natural that a model with roughly half the parameters can’t beat a 4B-class model on broad factual recall, so it’s more accurate to view MiniCPM5-2B not as an “all-rounder” but as a “coding- and agent-specialized compact model.”

    Training Pipeline: UltraData → 400B SFT → JustRL II → 16-Expert Distillation

    Behind these scores lies a staged training design. Base training ran in stable and decay phases using UltraData’s hierarchical data management, followed by 400B-token deep-thinking SFT after mid-training. The RL stage covered four separate domains—math, code, agentic, and writing—with a critic-based teacher called JustRL II guiding the training. The pipeline finishes with on-policy distillation that merges 16 expert models into a single checkpoint.

    The model card separately labels rows from Artificial Analysis and internal reproductions, proactively flagging evaluation-consistency issues—a sign that reproducibility was a deliberate priority.

    What to Try Right Now

    • Download MiniCPM5-2B via Ollama or llama.cpp and measure inference latency directly on a Mac or a single RTX 3090 GPU.
    • Pick 30 coding problems from your own domain, run multi-turn debugging, and compare response quality against Qwen3.5-2B and Gemma-4-E2B-it.
    • Build 10 BFCL-style function-calling scenarios and re-evaluate tool-use accuracy against your in-house data.
    • Stuff 50K tokens of internal documents into the 131K context and check retrieval and summarization accuracy yourself, just like NoLiMa, AA-LCR, and LongBench.
    • Confirm Apache 2.0 applicability with your legal team and verify there are no licensing risks around your in-house fine-tuning data.

    Practical Application Notes

    • When evaluating 2–3B-class models, don’t rely on a single overall average—weight domain benchmarks like LiveCodeBench, SWE-bench, and BFCL into your scoring.
    • If you’re considering MiniCPM5-2B for agent workloads, validate function-calling routing separately. Don’t assume the τ²-Bench 97.1 score generalizes to your case.
    • If long context is critical, judge models on measured benchmarks like AA-LCR and LongBench rather than the 131K window size alone.
    • For on-device deployment, note that the grouped-query attention’s 2 key/value head configuration keeps attention cache memory small but can lower batch throughput, so measure per scenario.
    • Once licensing is cleared, the practical sequence is to stand up a first baseline with vLLM or SGLang, then bolster weak areas with JustRL II-style RL fine-tuning.

    Frequently Asked Questions

    Can MiniCPM5-2B be used directly in commercial projects?

    The Apache 2.0 license permits both commercial use and redistribution. However, responsibility for the model’s outputs rests with the user, so it’s safer to decide on adoption after domain-specific evaluation.

    A 2.5B model scores higher than Qwen3.5-4B—is it really usable?

    The 53.9 average across 34 benchmarks is real. But the lead is concentrated in code and tool use, while it trails by more than 7 points on general-knowledge benchmarks like MMLU-Pro. You’ll need to re-weight the evaluation by domain.

    Does MiniCPM5-2B actually run on-device?

    Yes—because it’s standard LlamaForCausalLM, it runs out of the box on llama.cpp, Ollama, LM Studio, and MLX. In practice, though, it’s more realistic to use 8K–32K slices than the full 131K context on mobile.

    Is there a reason to migrate from previous MiniCPM models?

    Context window and code/agent scores have both been lifted compared to the previous generation. For new projects, start with MiniCPM5-2B; for existing systems, compare token usage and response latency before migrating gradually.

    The original source material was verified via MarkTechPost’s MiniCPM5-2B coverage. Comparison-group information referenced the same outlet’s IFM K2 Horizon launch article.

    Reference Sources

    This article was written after checking the following original sources: MarkTechPost — OpenBMB Releases MiniCPM5-2B: A 2.52B Dense Model Averaging 53.9 Across 34 Benchmarks and Built to Run On Device