sparse-moe

Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)

LightningLM 0.1V — Reference training pipeline for the LightningLM family. 2B dense seed → 5B MoE → 9B MoE → 120B sparse MoE through TurboQuant-PreTraining on a single eight-GPU node. Companion code for *Reversible Foundations*.

hummingbird is a lightweight, zero dependency runtime for massive open source Mixture of Experts (MoE) language models. It unifies SSD, RAM, and VRAM into a single intelligent memory hierarchy, enabling inference of models like GPT-OSS 120B, GLM, DeepSeek, Qwen, and more on consumer hardware

ASAN: A conceptual architecture for a self-creating (autopoietic), energy-efficient, and governable multi-agent AI system.

Let's activate Net://Anchor

Ecosystem-Specialised Sparse Mixture-of-Experts for Physics-Informed Carbon Flux Estimation

Low-cost AMD/ROCm stationary-expert fabric for configurable trillion-parameter sparse MoE inference, with packed-INT4 HIP kernels, activation-only RoCE transport, hardware/BOM design, release gates, and reproducible roofline tooling.

Unlimited-OCR 的 OpenVINO 本地适配：显式 KV、R-SWA cache、12 层 sparse MoE decode、FP4/INT8 mixed experts

Persian Sentiment Analysis using a custom Sparse Mixture-of-Experts (MoE) architecture from scratch in PyTorch.

Object A: MoE orbit prefetch — source+math+215KB trajectories+plots+sparse DeepSeek runtime (Apache-2.0). Analyze without weights; credit on substantial use.

A Sparse MoE Transformer built from scratch in PyTorch for abstractive text summarization. Features custom token routing, GQA, and LoRA, fully benchmarked against industry baselines (BART, FLAN-T5, LLaMA-3.2) using the XSum dataset.