Get ready to meet Inkling by Thinking Machines, a groundbreaking open model that’s set to redefine how we interact with AI. Freshly released on Hugging Face, Inkling isn’t just another large language model; it’s a colossal, multimodal powerhouse designed to understand and process image, text, and audio inputs natively.
Imagine an AI that doesn’t just read your words but also sees your images and hears your voice, all while understanding the intricate relationships between them. That’s Inkling, boasting an astounding ~1 trillion parameters and a massive 1 million context window, trained on an incredible 45 trillion tokens of diverse data.
What Makes Inkling Truly Special?
Inkling is the first large open model of its kind to truly excel at reasoning across multiple modalities. While other models might handle different data types separately, Inkling integrates them from the ground up, making it ideal for sophisticated applications that require deep cross-modal understanding.
Its core focus is on intricate reasoning across audio, images, and text, making it a fantastic foundation for developers looking to fine-tune it for specific domains. We’ve explored Inkling’s architecture and built some initial demos, and we’re convinced it’s poised to power a new generation of intelligent, multimodal reasoning applications.
- Native Multimodality: Understands image, text, and audio inputs seamlessly.
- Massive Scale: Features ~1 trillion parameters and a 1 million context window.
- Diverse Training: Trained on an immense 45 trillion tokens of text, images, audio, and video.
- Reasoning Powerhouse: Designed for complex cross-modal reasoning and domain adaptation.
- Efficient Inference: Available in full BF16 and an optimized NVFP4 variant, with speculative MTP layers for speed.
Under the Hood: Inkling’s Architecture
Inkling is built as a decoder-only multimodal Mixture-of-Experts (MoE) model, with 975 billion total parameters and 41 billion active parameters. This sophisticated design allows it to handle complex tasks with remarkable efficiency and depth. Let’s peel back the layers and explore some of its architectural innovations.
One key innovation is its use of relative attention instead of the more common RoPE (Rotary Positional Embeddings). Each attention layer directly learns positional information, with a unique fourth projection generating a per-token, per-head relative feature that’s tweaked with distance data and integrated into the attention module.
The model also employs a clever hybrid attention scheme, alternating between global attention (processing the full context) and sliding window attention (focusing on fixed local segments). This 5:1 ratio of sliding window to global attention layers significantly boosts computational efficiency, with a final global attention layer ensuring rich, comprehensive representations.
Adding to its unique structure is a distinctive short 1D convolution (SConv) over hidden states. SConv processes the current token and the previous W-1 hidden states, where W is the sliding window size. This helps Inkling handle local dependencies efficiently, freeing up the attention and MoE modules to focus on broader representations.
Inkling’s MoE implementation includes a “shared experts sink” where the router evaluates both routed experts and always-active shared experts. This approach performs top-k selection over 6 experts, supplemented by 2 shared experts that are continuously engaged, enhancing the model’s robustness and performance.
For understanding visual inputs, Inkling uses a straightforward hierarchical MLP patchifier. This module progressively merges pixels through several linear layers, ultimately producing a single embedding per image patch. Audio understanding is handled via a discretized mel spectrogram, converting 100ms audio chunks into the mel scale and classifying them into precise mel spectrogram bins.
Unlike models that use complex, separate encoders for each modality, Inkling’s multimodal towers are elegantly simple. Image patches and audio chunks pass through their respective embedding towers to generate media embeddings. The image input also incorporates a temporal dimension, opening doors for future video processing capabilities, though out-of-the-box video performance is still under evaluation.
Inference and Deployment: Getting Started with Inkling
Inkling offers day-0 support in major inference engines, making it accessible for a wide range of deployments. Whether you’re a developer eager to experiment or an enterprise looking for robust solutions, Inkling has you covered.
For those diving into inference, be aware that this model is substantial. The BF16 checkpoint demands 2 TB of VRAM, while the more optimized NVFP4 version still requires a hefty 600 GB of VRAM. For smaller-scale deployments or local experimentation, consider using serverless inference routers or ggml quants with llama.cpp.
Transformers Integration
The easiest way to get started is with the transformers library, which provides an intuitive any-to-any pipeline. Simply install or update transformers to version 5.14.0 or later, and you’re ready to go. You can choose between the full 16-bit model ("thinkingmachines/Inkling") for Hopper GPUs or later, or the quantized NVFP4 checkpoint ("thinkingmachines/Inkling-NVFP4") for Blackwell Nvidia GPUs.
The AutoModelForMultimodalLM and AutoProcessor classes offer a more granular control for various reasoning tasks. You can also specify a reasoning_effort argument in the tokenizer, with options ranging from “none” to “max,” allowing you to fine-tune the model’s processing intensity for different use cases.
Advanced Deployment Options: SGLang and vLLM
For high-performance serving, Inkling is supported by frameworks like SGLang and vLLM. SGLang, known for its speed, offers a custom model implementation, allowing you to shard the model across multiple GPUs and serve an OpenAI-compatible API. vLLM is excellent for production serving, enabling seamless deployment with tensor parallelism and an OpenAI-compatible server.
These frameworks are crucial for managing Inkling’s substantial resource requirements, often needing multiple nodes and distribution tools like SLURM for realistic parallel deployment. Parameters like --tensor-parallel-size and --max-model-len can be adjusted to optimize performance and manage KV-cache memory limits.
Remote and Local Inference
Hugging Face Inference Providers offer a convenient way to interact with Inkling remotely, supporting integration with the OpenAI client. You can specify routing preferences like ":auto", ":cheapest", or ":fastest". Notably, Inkling’s audio support with Inference Providers is actively being developed.
For those seeking local deployment on more constrained hardware, llama.cpp combined with Unsloth offers a powerful solution. Unsloth has quantized Inkling down to 1-bit precision, drastically reducing VRAM consumption by 95% compared to the original model. This allows you to run an OpenAI-compatible server locally and even leverage llama.cpp‘s built-in UI for agentic workloads and multimodal interactions.
Source: Hugging Face Blog
