How to run Google's Gemma 4 12B on your own machine
Google DeepMind released Gemma 4 12B on 3 June 2026, an Apache 2.0 open-weight model that takes text, image, audio and video. Here is what hardware it needs and the officially documented ways to run it locally.

Google DeepMind released Gemma 4 12B on 3 June 2026, a new mid-sized member of its Gemma 4 family of open-weight models. It accepts text, images, audio and video, produces text, and is published under the Apache 2.0 licence. Google says it is “small enough to run locally” on machines with 16GB of VRAM or unified memory. This guide sets out what the model is, what hardware it needs, and the routes Google documents for running it yourself.
What Gemma 4 12B is
Google calls this model “Gemma 4 12B Unified” because it has no separate vision or audio encoders. Image patches and audio waveforms are projected directly into the language model’s input through lightweight linear layers, rather than passing through dedicated encoder networks first. According to the model card:
| Property | Gemma 4 12B |
|---|---|
| Parameters | 11.95 billion (dense) |
| Layers | 48 |
| Context window | 256K tokens |
| Inputs | Text, image, audio, video |
| Audio / video limits | 30 seconds of audio; 60 seconds of video at one frame per second |
| Licence | Apache 2.0 |
| Hugging Face IDs | google/gemma-4-12B-it (instruction-tuned), google/gemma-4-12B (pre-trained) |
Google’s vendor-reported results for the instruction-tuned model include 77.2% on MMLU Pro, 72.0% on LiveCodeBench v6 and 78.8% on GPQA Diamond. Weights are available from Hugging Face and Kaggle.
Hardware: what 16GB actually means
Google’s Gemma 4 documentation gives approximate memory needed just to load the 12B model’s weights:
- BF16 (16-bit): 26.7GB
- 8-bit: 13.4GB
- 4-bit: 6.7GB
Google notes these figures include an allowance for loading overhead but exclude supporting software and the context window, and that the key-value cache grows as prompts and responses get longer. In other words, the full-precision weights alone exceed 16GB, so on a 16GB laptop you will be running a lower-precision build, and very long contexts will need extra headroom.
Option 1: a desktop app or Ollama
The quickest routes are the ones Google lists for trying the model “with a couple of clicks”: LM Studio, Ollama, the Google AI Edge Gallery desktop app for Apple Silicon Macs, and the LiteRT-LM command-line tool. Ollama’s library lists a gemma4:12b tag, so with Ollama installed the command is:
ollama run gemma4:12b
Option 2: a local OpenAI-compatible server with LiteRT-LM
Google’s launch materials highlight a new litert-lm serve command that runs the model behind an OpenAI-compatible API, so existing tools that speak that API can point at it. The LiteRT-LM CLI requires Python 3.10 or later and installs with pip:
pip install --upgrade litert-lm
Import the model into the local registry, then start the server:
litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm gemma-4-12B-it.litertlm gemma4-12b
litert-lm serve
According to the LiteRT-LM documentation, the server listens on port 9379 by default and exposes /v1/models and /v1/chat/completions, with the model addressed as gemma4-12b. The --host and --port options change where it listens.
Option 3: Hugging Face Transformers
For Python developers, the model card’s quick start uses the latest Transformers release:
pip install -U transformers torch accelerate
from transformers import AutoProcessor, AutoModelForMultimodalLM
MODEL_ID = "google/gemma-4-12B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL_ID,
dtype="auto",
device_map="auto"
)
You then build a list of messages, pass it through processor.apply_chat_template(...) with add_generation_prompt=True, call model.generate(...), and use processor.parse_response(...) to separate any reasoning from the answer. Setting enable_thinking=True in the chat template turns on the model’s reasoning mode. For images, the card adds torchvision to the install; for audio and video it adds torchvision and librosa. The full examples are on the model card.
Google’s developer guide also links resources for running the model with llama.cpp, MLX, vLLM and SGLang, and for fine-tuning it with Unsloth.
Settings worth getting right
The model card recommends a few defaults that are easy to miss:
- Sampling:
temperature=1.0,top_p=0.95,top_k=64for all use cases. - Thinking: reasoning is enabled by placing the
<|think|>token at the start of the system prompt. Libraries such as Transformers and llama.cpp handle the chat template for you. - Multi-turn chats: do not feed earlier reasoning back into the conversation history; keep only the final answers, except on tool-call turns.
- Input order: put images before the text of your prompt and audio after it.
- Image detail: visual token budgets of 70, 140, 280, 560 or 1,120 trade speed for detail. Use higher budgets for OCR and documents.
What it means
Gemma 4 12B is, according to Google, the first medium-sized Gemma model to accept audio input, and it is designed to run on a single consumer machine. For developers, the practical question is precision: the 16GB figure depends on quantised weights, so it is worth checking memory headroom against the context length your application needs before committing. Google’s Gemma documentation says the models permit responsible commercial use, and Apache 2.0 is a widely used permissive open-source licence.
Sources
- Google: Introducing Gemma 4 12B (launch post, 3 June 2026)
- Google Developers Blog: Gemma 4 12B, the developer guide (3 June 2026)
- Hugging Face: google/gemma-4-12B-it model card
- Google: Gemma 4 model overview and memory requirements
- Google AI Edge: LiteRT-LM CLI documentation
- Google AI Edge: LiteRT-LM OpenAI-compatible server
- Ollama: gemma4 library page




