Microsoft launches Maia 200: AI chip for inferences workload

Microsoft has launched Maia 200, a new custom AI accelerator chip designed specifically for inference workloads. This chip focuses on generating AI tokens more efficiently, which could lower the cost of running large language models at scale.

The Maia 200 is built on TSMC’s advanced 3-nanometer process node. It contains over 140 billion transistors. The chip delivers more than 10 petaFLOPS of performance in 4-bit floating-point precision (FP4) and more than 5 petaFLOPS in 8-bit floating-point precision (FP8). It operates within a 750W TDP envelope for the system-on-chip.

Key memory features include 216 GB of HBM3e memory running at 7 TB/s bandwidth, along with 272 MB of on-chip SRAM. The design includes native tensor cores for FP8 and FP4 computations. It also has a redesigned memory subsystem optimized for narrow-precision data types, a specialized DMA engine, and a high-bandwidth network-on-chip (NoC) fabric. These elements improve data movement, keeping large models active and achieving high utilization.

The chip uses a novel two-tier scale-up network based on standard Ethernet. It provides 2.8 TB/s of bidirectional dedicated bandwidth per accelerator. Within each tray, four Maia 200 chips connect directly without switches, using a custom Maia AI transport protocol. This setup supports clusters of up to 6,144 accelerators with reliable, high-performance collective operations.

Microsoft states that Maia 200 is the most performant first-party AI silicon from any hyperscaler. It offers three times the FP4 performance of Amazon’s third-generation Trainium and higher FP8 performance than Google’s seventh-generation TPU. It also provides 30% better performance per dollar compared to the latest hardware in Microsoft’s current fleet.

Deployment uses second-generation closed-loop liquid cooling with a Heat Exchanger Unit (HXU). The chip integrates natively with Azure’s control plane for security, telemetry, diagnostics, and management at both chip and rack levels.

Initial rollout has started in the US Central datacenter region near Des Moines, Iowa. The US West 3 region near Phoenix, Arizona, is next, with more regions planned. Microsoft reduced the time from first silicon to datacenter rack deployment to less than half of similar past programs. AI models ran on the silicon within days of receiving the first packaged parts.

Developers, startups, and academics can access a preview of the Maia AI SDK. It includes PyTorch integration, the Triton compiler, an optimized kernel library, and a low-level programming language called NPL. A pre-silicon simulator and cost calculator are also available.

Maia 200 supports multiple models, including the latest from OpenAI, such as GPT-5.2 variants. It powers services like Microsoft Foundry and Microsoft 365 Copilot. The Microsoft Superintelligence team uses it for synthetic data generation and reinforcement learning to improve future in-house models.

(source)