Use vLLM to deploy GLM-4.5 and GLM-4.5V model

Use vLLM to deploy GLM-4.5 and GLM-4.5V model

Model Introduction

The GLM-4.5 series models are foundation models designed for intelligent agents. GLM-4.5 has 355 billion total parameters with 32 billion active parameters, while GLM-4.5-Air adopts a more compact design with 106 billion total parameters and 12 billion active parameters. GLM-4.5 models unify reasoning, coding, and intelligent agent capabilities to meet the complex demands of intelligent agent applications.

Both GLM-4.5 and GLM-4.5-Air are hybrid reasoning models that provide two modes: thinking mode for complex reasoning and tool usage, and non-thinking mode for immediate responses.

As demonstrated in our comprehensive evaluation across 12 industry-standard benchmarks, GLM-4.5 achieves exceptional performance with a score of 63.2, in the 3rd place among all the proprietary and open-source models. Notably, GLM-4.5-Air delivers competitive results at 59.8 while maintaining superior efficiency.

GLM-4.5V is based on GLM-4.5-Air. It continues the technical approach of GLM-4.1V-Thinking, achieving SOTA performance among models of the same scale on 42 public vision-language benchmarks.

To get more information about GLM-4.5 and GLM-4.5V, please refer to the GLM-4.5 and GLM-V.

this blog will guide users on how to use vLLM to accelerate inference for the GLM-4.5V and GLM-4.5 model series on NVIDIA Blackwell and Hopper GPUs.

Installation

In the latest vLLM main branch, both the GLM-4.5V and GLM-4.5 model series are supported. You can install the nightly version and manually update transformers to enable model support.

pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install transformers-v4.55.0-GLM-4.5V-preview

Usage

GLM-4.5 and GLM-4.5V both offer FP8 and BF16 precision models. In vLLM, you can use the same command to run inference for either precision.

For the GLM-4.5 model, you can start the service with the following command:

vllm serve zai-org/GLM-4.5-Air \
    --tensor-parallel-size 4 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice

For the GLM-4.5V model, you can start the service with the following command:

vllm serve zai-org/GLM-4.5V \
     --tensor-parallel-size 4   \
     --tool-call-parser glm45   \
     --reasoning-parser glm45   \
     --enable-auto-tool-choice  \
     --allowed-local-media-path / \
     --media-io-kwargs '{"video": {"num_frames": -1}}'

Important Notes

• The reasoning part of the model output will be wrapped in reasoning_content. content will only contain the final answer. To disable reasoning, add the following parameter: extra_body={"chat_template_kwargs": {"enable_thinking": False}}
• If you’re using 8x H100 GPUs and encounter insufficient memory when running the GLM-4.5 model, you’ll need --cpu-offload-gb 16.
• If you encounter flash infer issues, use VLLM_ATTENTION_BACKEND=XFORMERS as a temporary replacement. You can also specify TORCH_CUDA_ARCH_LIST='9.0+PTX' to use flash infer, different GPUs have different TORCH_CUDA_ARCH_LIST values, please check accordingly.
• vllm v0 is not support our model.

Grounding in GLM-4.5V

GLM-4.5V equips precise grounding capabilities. Given a prompt that requests the location of a specific object, GLM-4.5V is able to reasoning step-by-step and identify the bounding boxes of the target object. The query prompt supports complex descriptions of the target object as well as specified output formats, for example:

• Help me to locate in the image and give me its bounding boxes.
• Please pinpoint the bounding box [[x1,y1,x2,y2], …] in the image as per the given description.

Here, <expr> is the description of the target object. The output bounding box is a quadruple \([x_1,y_1,x_2,y_2]\) composed of the coordinates of the top-left and bottom-right corners, where each value is normalized by the image width (for x) or height (for y) and scaled by 1000.

In the response, the special tokens <|begin_of_box|> and <|end_of_box|> are used to mark the image bounding box in the answer. The bracket style may vary ([], [[]], (), <>, etc.), but the meaning is the same: to enclose the coordinates of the box.

Cooperation with vLLM and Z.ai Team

During the release of the GLM-4.5 and GLM-4.5V models, the vLLM team worked closely with the Z.ai team, providing extensive support in addressing issues related to the model launch.
The GLM-4.5 and GLM-4.5V models provided by the Z.ai team were modified in the vLLM implementation PR, including (but not limited to) resolving CUDA Core Dump debugging issues and FP8 model accuracy alignment problems.
They also ensured that the vLLM main branch had full support for the open-source GLM-4.5 series before the models were released.

Acknowledgement

We would like to thank the vLLM team members who contributed to this effort are: Simon Mo, Kaichao You.