If you are looking to harness the power of Google’s Gemma models for creating high-quality text embeddings, you have come to the right place. In this guide, we will walk you through the process of setting up a containerized environment to run these models efficiently. We know that setting up machine learning environments can be a headache due to conflicting dependencies, which is why we recommend using Docker. By using the method we are about to show you, you can ensure that your environment is isolated, reproducible, and ready to handle heavy computational tasks.
First, let’s understand what we are doing. We are essentially building a specialized container based on an NVIDIA CUDA image. This container will act as a self-contained laboratory where we install Python, Git, and a high-performance inference engine called vLLM. vLLM is a crucial component here because it allows us to serve the Gemma embedding model with extremely high throughput, making it perfect for real-world applications.
Before we dive into the command, you need to make sure your system is prepared. You must have the NVIDIA Container Toolkit installed on your host machine so that Docker can communicate with your GPU. Without this, the container won’t be able to access your hardware, and the model will either fail to run or run painfully slowly on your CPU.
Now, let’s look at the specific command we will use to launch this environment. You can copy this directly into your terminal, but make sure you have your Hugging Face token ready.
docker run --rm -it \
--gpus '"device=1"' \
--ipc=host \
-p 8000:8000 \
-v "$(pwd)/run.sh:/run.sh" \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN=${HF_TOKEN:-} \
-e CUDA_VISIBLE_DEVICES=0 \
nvidia/cuda:12.6.0-devel-ubuntu22.04 \
bash -c "
apt-get update && apt-get install -y python3-pip git &&
pip3 install vllm --extra-index-url https://download.pytorch.org/whl/cu126 &&
bash /run.sh
"We should break down this command so you understand exactly what each part does. When we use the –rm flag, we are telling Docker to automatically clean up and remove the container once we stop it. This keeps your system tidy. The -it flag allows us to interact with the container in real-time, which is helpful for debugging.
One of the most important parts is the –gpus ‘”device=1″‘ flag. This tells Docker exactly which GPU you want to use. If you have multiple cards, you can specify the index here. We also use –ipc=host, which is a technical necessity for high-performance deep learning. It allows the container to share memory more efficiently with the host, preventing “out of memory” errors during heavy data transfers.
We are also mapping ports using -p 8000:8000. This means that once the embedding server is running inside the container, you can access it from your web browser or your local code via localhost:8000.
To make this work, we use two volume mounts (-v). The first one, -v “$(pwd)/run.sh:/run.sh”, takes a script named run.sh from your current folder and puts it inside the container. You must create this script in your local directory to tell the container how to start the Gemma model. The second one, -v ~/.cache/huggingface:/root/.cache/huggingface, is a lifesaver. It maps your local Hugging Face cache to the container. This means that once you download the Gemma model, it stays on your hard drive. You won’t have to re-download several gigabytes of data every time you restart the container.
We also pass an environment variable, HF_TOKEN. Since Gemma is a gated model, you need to provide your Hugging Face authorization token to download it. We use the syntax ${HF_TOKEN:-} so that if you haven’t set this variable in your terminal, it won’t crash immediately, though the model download will eventually fail without it.
Finally, the long string at the end is the setup script. We start with a base NVIDIA image (version 12.6.0), update the package manager, install Python and Git, and then use pip to install vLLM specifically optimized for CUDA 12.6. Once everything is installed, we execute your custom /run.sh script.
By following this workflow, you are setting up a professional-grade inference server. You get the speed of vLLM, the isolation of Docker, and the convenience of persistent model caching. It is a robust way to integrate Gemma embeddings into your AI applications.