technical-term · Models & inference · foundational · Reviewed

Mixture of experts

A neural architecture that routes each input token or example through a selected subset of learned expert subnetworks.

Definition

A mixture-of-experts model contains multiple learned subnetworks and a router that activates only some of them for a given input, enabling large parameter capacity with sparse computation. The “experts” are mathematical components inside a model—not agents, people, independently verified specialists, or separately governed services.

Sparse mixture architectures are established in research and production models; implementation details and tradeoffs vary.

Why it matters

The router and experts are trained together as model components.

MoE architecture does not imply multiple perspectives, independent judgment, or a multi-agent workflow.

System anatomy

Experts
Learned neural subnetworks with no independent agency.
Router
A learned gate selecting which experts process an input.
Sparsity
Only a subset of total parameters is active for each input.

Important distinctions

Multi-agent system
Agents have software roles, state, tools, identities, and coordination; MoE experts are internal neural layers.
Model routing
External routing selects among deployed models; an MoE router selects internal subnetworks.

Implementation signals

  • Treat architecture labels as model metadata, not governance claims
  • Evaluate the complete endpoint on the intended workload

Failure modes

  • Marketing describes internal experts as collaborating agents
  • Parameter count is used as a proxy for application quality

Sources and further study

  1. Google Research / ICLR — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Introduces a sparsely gated neural layer that routes each example to a subset of learned expert subnetworks.

    Use in this library: Primary definition. “Experts” are learned subnetworks, not autonomous agents, accountable specialists, or independently governed services.

    paper · paper · Published 2017
  2. Artificial Analysis — Artificial Analysis Intelligence Benchmarking Methodology

    A transparent methodology for comparing model quality, price, latency, throughput, and cost per task. Its composite indices are useful evidence, not a substitute for use-case-specific evaluation.

    Use in this library: Independent benchmark. Composite benchmark results change over time and do not establish enterprise fit, safety, controllability, or acceptance for a specific workload. No leaderboard values are reproduced here.

    research · research · Published 2026-06