The Open Weights Frontier Advances to Llama 4
Meta AI has released architecture previews for Llama 4, the forthcoming generation of its open-weights foundation model family. Mark Zuckerberg confirmed that Llama 4 features a completely redesigned sparse Mixture-of-Experts (MoE) routing topology that activates only 40 billion parameters per token out of hundreds of billions total.
Native Multimodal Tokenization
Unlike previous hybrid designs that attached separate vision adapters to an existing language backbone, Llama 4 unifies video, audio, code, and text tokens into a shared continuous embedding space from step zero of pre-training.
