Engine Arguments¶

Engine arguments control the behavior of the vLLM engine.

• For offline inference, they are part of the arguments to LLM class.
• For online serving, they are part of the arguments to vllm serve.

The engine argument classes, EngineArgs and AsyncEngineArgs, are a combination of the configuration classes defined in vllm.config. Therefore, if you are interested in developer documentation, we recommend looking at these configuration classes as they are the source of truth for types, defaults and docstrings.

When passing JSON CLI arguments, the following sets of arguments are equivalent:

• --json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'
• --json-arg.key1 value1 --json-arg.key2.key3 value2

Additionally, list elements can be passed individually using +:

• --json-arg '{"key4": ["value3", "value4", "value5"]}'
• --json-arg.key4+ value3 --json-arg.key4+='value4,value5'

EngineArgs¶

--disable-log-stats¶

Disable logging statistics.

Default: False

ModelConfig¶

Configuration for the model.

--model¶

Name or path of the Hugging Face model to use. It is also used as the content for model_name tag in metrics output when served_model_name is not specified.

Default: Qwen/Qwen3-0.6B

--runner¶

Possible choices: auto, draft, generate, pooling

The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types.

Default: auto

--convert¶

Possible choices: auto, classify, embed, none, reward

Convert the model using adapters defined in vllm.model_executor.models.adapters. The most common use case is to adapt a text generation model to be used for pooling tasks.

Default: auto

--task¶

Possible choices: auto, classify, draft, embed, embedding, generate, reward, score, transcription, None

[DEPRECATED] The task to use the model for. If the model supports more than one model runner, this is used to select which model runner to run.

Note that the model may support other tasks using the same model runner.

Default: None

--tokenizer¶

Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used.

Default: None

--tokenizer-mode¶

Possible choices: auto, custom, mistral, slow

Tokenizer mode:

• "auto" will use the fast tokenizer if available.

• "slow" will always use the slow tokenizer.

• "mistral" will always use the tokenizer from mistral_common.

• "custom" will use --tokenizer to select the preregistered tokenizer.

Default: auto

--trust-remote-code, --no-trust-remote-code¶

Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer.

Default: False

--dtype¶

Possible choices: auto, bfloat16, float, float16, float32, half

Data type for model weights and activations:

• "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.

• "half" for FP16. Recommended for AWQ quantization.

• "float16" is the same as "half".

• "bfloat16" for a balance between precision and range.

• "float" is shorthand for FP32 precision.

• "float32" for FP32 precision.

Default: auto

--seed¶

Random seed for reproducibility. Initialized to None in V0, but initialized to 0 in V1.

Default: None

--hf-config-path¶

Name or path of the Hugging Face config to use. If unspecified, model name or path will be used.

Default: None

--allowed-local-media-path¶

Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments.

Default: ``

--allowed-media-domains¶

If set, only media URLs that belong to this domain can be used for multi-modal inputs.

Default: None

--revision¶

The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

Default: None

--code-revision¶

The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

Default: None

--rope-scaling¶

RoPE scaling configuration. For example, {"rope_type":"dynamic","factor":2.0}.

Should either be a valid JSON string or JSON keys passed individually.

Default: {}

--rope-theta¶

RoPE theta. Use with rope_scaling. In some cases, changing the RoPE theta improves the performance of the scaled model.

Default: None

--tokenizer-revision¶

The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

Default: None

--max-model-len¶

Model context length (prompt and output). If unspecified, will be automatically derived from the model config.

When passing via --max-model-len, supports k/m/g/K/M/G in human-readable format. Examples:

• 1k -> 1000

• 1K -> 1024

• 25.6k -> 25,600

Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers.

Examples:
- '1k' -> 1,000
- '1K' -> 1,024
- '25.6k' -> 25,600

Default: None

--quantization, -q¶

Method used to quantize the weights. If None, we first check the quantization_config attribute in the model config file. If that is None, we assume the model weights are not quantized and use dtype to determine the data type of the weights.

Default: None

--enforce-eager, --no-enforce-eager¶

Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility.

Default: False

--max-logprobs¶

Maximum number of log probabilities to return when logprobs is specified in SamplingParams. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output_length * vocab_size) logprobs are allowed to be returned and it may cause OOM.

Default: 20

--logprobs-mode¶

Possible choices: processed_logits, processed_logprobs, raw_logits, raw_logprobs

Indicates the content returned in the logprobs and prompt_logprobs. Supported mode: 1) raw_logprobs, 2) processed_logprobs, 3) raw_logits, 4) processed_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top_k/top_p.

Default: raw_logprobs

--disable-sliding-window, --no-disable-sliding-window¶

Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored.

Default: False

--disable-cascade-attn, --no-disable-cascade-attn¶

Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. Note that even if this is set to False, cascade attention will be only used when the heuristic tells that it's beneficial.

Default: False

--skip-tokenizer-init, --no-skip-tokenizer-init¶

Skip initialization of tokenizer and detokenizer. Expects valid prompt_token_ids and None for prompt from the input. The generated output will contain token ids.

Default: False

--enable-prompt-embeds, --no-enable-prompt-embeds¶

If True, enables passing text embeddings as inputs via the prompt_embeds key. Note that enabling this will double the time required for graph compilation.

Default: False

--served-model-name¶

The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the --model argument. Noted that this name(s) will also be used in model_name tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one.

Default: None

--config-format¶

Possible choices: auto, hf, mistral

The format of the model config to load:

• "auto" will try to load the config in hf format if available else it will try to load in mistral format.

• "hf" will load the config in hf format.

• "mistral" will load the config in mistral format.

Default: auto

--hf-token¶

The token to use as HTTP bearer authorization for remote files . If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).

Default: None

--hf-overrides¶

If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config.

Default: {}

--pooler-config¶

Pooler config which controls the behaviour of output pooling in pooling models.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--override-pooler-config¶

[DEPRECATED] Use pooler_config instead. This field will be removed in v0.12.0 or v1.0.0, whichever is sooner.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--logits-processor-pattern¶

Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra completion argument. Defaults to None, which allows no processors.

Default: None

--generation-config¶

The folder path to the generation config. Defaults to "auto", the generation config will be loaded from model path. If set to "vllm", no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests.

Default: auto

--override-generation-config¶

Overrides or sets generation config. e.g. {"temperature": 0.5}. If used with --generation-config auto, the override parameters will be merged with the default config from the model. If used with --generation-config vllm, only the override parameters are used.

Should either be a valid JSON string or JSON keys passed individually.

Default: {}

--enable-sleep-mode, --no-enable-sleep-mode¶

Enable sleep mode for the engine (only cuda platform is supported).

Default: False

--model-impl¶

Possible choices: auto, terratorch, transformers, vllm

Which implementation of the model to use:

• "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available.

• "vllm" will use the vLLM model implementation.

• "transformers" will use the Transformers model implementation.

• "terratorch" will use the TerraTorch model implementation.

Default: auto

--override-attention-dtype¶

Override dtype for attention

Default: None

--logits-processors¶

One or more logits processors' fully-qualified class names or class definitions

Default: None

--io-processor-plugin¶

IOProcessor plugin name to load at model startup

Default: None

LoadConfig¶

Configuration for loading the model weights.

--load-format¶

The format of the model weights to load:

• "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available.

• "pt" will load the weights in the pytorch bin format.

• "safetensors" will load the weights in the safetensors format.

• "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading.

• "dummy" will initialize the weights with random values, which is mainly for profiling.

• "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information.

• "runai_streamer" will load the Safetensors weights using Run:ai Model Streamer.

• "bitsandbytes" will load the weights using bitsandbytes quantization.

• "sharded_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models.

• "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md).

• "mistral" will load weights from consolidated safetensors files used by Mistral models.

• Other custom values can be supported via plugins.

Default: auto

--download-dir¶

Directory to download and load the weights, default to the default cache directory of Hugging Face.

Default: None

--safetensors-load-strategy¶

Specifies the loading strategy for safetensors weights. - "lazy" (default): Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM.

Default: lazy

--model-loader-extra-config¶

Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format.

Default: {}

--ignore-patterns¶

The list of patterns to ignore when loading the model. Default to "original/*/" to avoid repeated loading of llama's checkpoints.

Default: None

--use-tqdm-on-load, --no-use-tqdm-on-load¶

Whether to enable tqdm for showing progress bar when loading model weights.

Default: True

--pt-load-map-location¶

pt_load_map_location: the map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to {"": "cuda"}. Another supported format is mapping from different devices like from GPU 1 to GPU 0: {"cuda:1": "cuda:0"}. Note that when passed from command line, the strings in dictionary needs to be double quoted for json parsing. For more details, see original doc for map_location in https://pytorch.org/docs/stable/generated/torch.load.html

Default: cpu

StructuredOutputsConfig¶

Dataclass which contains structured outputs config for the engine.

--reasoning-parser¶

Possible choices: deepseek_r1, glm45, openai_gptoss, granite, hunyuan_a13b, mistral, qwen3, seed_oss, step3

Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format.

Default: ``

--guided-decoding-backend¶

[DEPRECATED] --guided-decoding-backend will be removed in v0.12.0.

Default: None

--guided-decoding-disable-fallback¶

[DEPRECATED] --guided-decoding-disable-fallback will be removed in v0.12.0.

Default: None

--guided-decoding-disable-any-whitespace¶

[DEPRECATED] --guided-decoding-disable-any-whitespace will be removed in v0.12.0.

Default: None

--guided-decoding-disable-additional-properties¶

[DEPRECATED] --guided-decoding-disable-additional-properties will be removed in v0.12.0.

Default: None

ParallelConfig¶

Configuration for the distributed execution.

--distributed-executor-backend¶

Possible choices: external_launcher, mp, ray, uni

Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, this will default to "ray" if Ray is installed and fail otherwise. Note that tpu only support Ray for distributed inference.

Default: None

--pipeline-parallel-size, -pp¶

Number of pipeline parallel groups.

Default: 1

--tensor-parallel-size, -tp¶

Number of tensor parallel groups.

Default: 1

--decode-context-parallel-size, -dcp¶

Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp_size needs to be divisible by dcp_size.

Default: 1

--data-parallel-size, -dp¶

Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size.

Default: 1

--data-parallel-rank, -dpn¶

Data parallel rank of this instance. When set, enables external load balancer mode.

Default: None

--data-parallel-start-rank, -dpr¶

Starting data parallel rank for secondary nodes.

Default: None

--data-parallel-size-local, -dpl¶

Number of data parallel replicas to run on this node.

Default: None

--data-parallel-address, -dpa¶

Address of data parallel cluster head-node.

Default: None

--data-parallel-rpc-port, -dpp¶

Port for data parallel RPC communication.

Default: None

--data-parallel-backend, -dpb¶

Backend for data parallel, either "mp" or "ray".

Default: mp

--data-parallel-hybrid-lb, --no-data-parallel-hybrid-lb¶

Whether to use "hybrid" DP LB mode. Applies only to online serving and when data_parallel_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank.

Default: False

--enable-expert-parallel, --no-enable-expert-parallel¶

Use expert parallelism instead of tensor parallelism for MoE layers.

Default: False

--enable-dbo, --no-enable-dbo¶

Enable dual batch overlap for the model executor.

Default: False

--dbo-decode-token-threshold¶

The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.

Default: 32

--dbo-prefill-token-threshold¶

The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.

Default: 512

--enable-eplb, --no-enable-eplb¶

Enable expert parallelism load balancing for MoE layers.

Default: False

--eplb-config¶

Expert parallelism configuration.

Should either be a valid JSON string or JSON keys passed individually.

Default: EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False)

--expert-placement-strategy¶

Possible choices: linear, round_robin

The expert placement strategy for MoE layers:

• "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts [0, 1] and rank 1 will have experts [2, 3].

• "round_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts [0, 2] and rank 1 will have experts [1, 3]. This strategy can help improve load balancing for grouped expert models with no redundant experts.

Default: linear

--num-redundant-experts¶

[DEPRECATED] --num-redundant-experts will be removed in v0.12.0.

Default: None

--eplb-window-size¶

[DEPRECATED] --eplb-window-size will be removed in v0.12.0.

Default: None

--eplb-step-interval¶

[DEPRECATED] --eplb-step-interval will be removed in v0.12.0.

Default: None

--eplb-log-balancedness, --no-eplb-log-balancedness¶

[DEPRECATED] --eplb-log-balancedness will be removed in v0.12.0.

Default: None

--max-parallel-loading-workers¶

Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models.

Default: None

--ray-workers-use-nsight, --no-ray-workers-use-nsight¶

Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler.

Default: False

--disable-custom-all-reduce, --no-disable-custom-all-reduce¶

Disable the custom all-reduce kernel and fall back to NCCL.

Default: False

--worker-cls¶

The full name of the worker class to use. If "auto", the worker class will be determined based on the platform.

Default: auto

--worker-extension-cls¶

The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective_rpc calls.

Default: ``

--enable-multimodal-encoder-data-parallel¶

Default: False

CacheConfig¶

Configuration for the KV cache.

--block-size¶

Possible choices: 1, 8, 16, 32, 64, 128

Size of a contiguous cache block in number of tokens. On CUDA devices, only block sizes up to 32 are supported.

This config has no static default. If left unspecified by the user, it will be set in Platform.check_and_update_config() based on the current platform.

Default: None

--gpu-memory-utilization¶

The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.9. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance.

Default: 0.9

--kv-cache-memory-bytes¶

Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu_memory_utilization. However, users may want to manually specify the kv cache memory size. kv_cache_memory_bytes allows more fine-grain control of how much memory gets used when compared with using gpu_memory_memory_utilization. Note that kv_cache_memory_bytes (when not-None) ignores gpu_memory_utilization

Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers.

Examples:
- '1k' -> 1,000
- '1K' -> 1,024
- '25.6k' -> 25,600

Default: None

--swap-space¶

Size of the CPU swap space per GPU (in GiB).

Default: 4

--kv-cache-dtype¶

Possible choices: auto, bfloat16, fp8, fp8_e4m3, fp8_e5m2, fp8_inc

Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8.

Default: auto

--num-gpu-blocks-override¶

Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None. Used for testing preemption.

Default: None

--enable-prefix-caching, --no-enable-prefix-caching¶

Whether to enable prefix caching. Enabled by default for V1.

Default: None

--prefix-caching-hash-algo¶

Possible choices: sha256, sha256_cbor

Set the hash algorithm for prefix caching:

• "sha256" uses Pickle for object serialization before hashing.

• "sha256_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256.

Default: sha256

--cpu-offload-gb¶

The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass.

Default: 0

--calculate-kv-scales, --no-calculate-kv-scales¶

This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. If False, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0.

Default: False

--kv-sharing-fast-prefill, --no-kv-sharing-fast-prefill¶

This feature is work in progress and no prefill optimization takes place with this flag enabled currently.

In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n)

Default: False

--mamba-cache-dtype¶

Possible choices: auto, float32

The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config.

Default: auto

--mamba-ssm-cache-dtype¶

Possible choices: auto, float32

The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba_cache_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba_cache_dtype.

Default: auto

MultiModalConfig¶

Controls the behavior of multimodal models.

--limit-mm-per-prompt¶

The maximum number of input items allowed per prompt for each modality. Defaults to 1 (V0) or 999 (V1) for each modality.

For example, to allow up to 16 images and 2 videos per prompt: {"image": 16, "video": 2}

Should either be a valid JSON string or JSON keys passed individually.

Default: {}

--media-io-kwargs¶

Additional args passed to process media inputs, keyed by modalities. For example, to set num_frames for video, set --media-io-kwargs '{"video": {"num_frames": 40} }'

Should either be a valid JSON string or JSON keys passed individually.

Default: {}

--mm-processor-kwargs¶

Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from transformers.AutoProcessor.from_pretrained.

The available overrides depend on the model that is being run.

For example, for Phi-3-Vision: {"num_crops": 4}.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--mm-processor-cache-gb¶

The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs.

This cache is duplicated for each API process and engine core process, resulting in a total memory usage of mm_processor_cache_gb * (api_server_count + data_parallel_size).

Set to 0 to disable this cache completely (not recommended).

Default: 4

--disable-mm-preprocessor-cache¶

Default: False

--mm-processor-cache-type¶

Possible choices: lru, shm

Type of cache to use for the multi-modal preprocessor/mapper. If shm, use shared memory FIFO cache. If lru, use mirrored LRU cache.

Default: lru

--mm-shm-cache-max-object-size-mb¶

Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when mm_processor_cache_type is "shm".

Default: 128

--mm-encoder-tp-mode¶

Possible choices: data, weights

Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP).

• "weights": Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior)

• "data": Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by --data-parallel-size). This is only supported on a per-model basis and falls back to "weights" if the encoder does not support DP.

Default: weights

--interleave-mm-strings, --no-interleave-mm-strings¶

Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string.

Default: False

--skip-mm-profiling, --no-skip-mm-profiling¶

When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization.

This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache.

Default: False

--video-pruning-rate¶

Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range [0;1) and determines fraction of media tokens from each video to be pruned.

Default: None

LoRAConfig¶

Configuration for LoRA.

--enable-lora, --no-enable-lora¶

If True, enable handling of LoRA adapters.

Default: None

--enable-lora-bias, --no-enable-lora-bias¶

[DEPRECATED] Enable bias for LoRA adapters. This option will be removed in v0.12.0.

Default: False

--max-loras¶

Max number of LoRAs in a single batch.

Default: 1

--max-lora-rank¶

Max LoRA rank.

Default: 16

--lora-extra-vocab-size¶

(Deprecated) Maximum size of extra vocabulary that can be present in a LoRA adapter. Will be removed in v0.12.0.

Default: 256

--lora-dtype¶

Possible choices: auto, bfloat16, float16

Data type for LoRA. If auto, will default to base model dtype.

Default: auto

--max-cpu-loras¶

Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras.

Default: None

--fully-sharded-loras, --no-fully-sharded-loras¶

By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster.

Default: False

--default-mm-loras¶

Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default_mm_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

ObservabilityConfig¶

Configuration for observability - metrics and tracing.

--show-hidden-metrics-for-version¶

Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use --show-hidden-metrics-for-version=0.7 as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release.

Default: None

--otlp-traces-endpoint¶

Target URL to which OpenTelemetry traces will be sent.

Default: None

--collect-detailed-traces¶

Possible choices: all, model, worker, None, model,worker, model,all, worker,model, worker,all, all,model, all,worker

It makes sense to set this only if --otlp-traces-endpoint is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact.

Note that collecting detailed timing information for each request can be expensive.

Default: None

SchedulerConfig¶

Scheduler configuration.

--max-num-batched-tokens¶

Maximum number of tokens to be processed in a single iteration.

This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context.

Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers.

Examples:
- '1k' -> 1,000
- '1K' -> 1,024
- '25.6k' -> 25,600

Default: None

--max-num-seqs¶

Maximum number of sequences to be processed in a single iteration.

This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context.

Default: None

--max-num-partial-prefills¶

For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently.

Default: 1

--max-long-partial-prefills¶

For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. Setting this less than max_num_partial_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency.

Default: 1

--cuda-graph-sizes¶

Cuda graph capture sizes 1. if none provided, then default set to [min(max_num_seqs * 2, 512)] 2. if one value is provided, then the capture list would follow the pattern: [1, 2, 4] + [i for i in range(8, cuda_graph_sizes + 1, 8)] 3. more than one value (e.g. 1 2 128) is provided, then the capture list will follow the provided list.

Default: []

--long-prefill-token-threshold¶

For chunked prefill, a request is considered long if the prompt is longer than this number of tokens.

Default: 0

--num-lookahead-slots¶

The number of slots to allocate per sequence per step, beyond the known token ids. This is used in speculative decoding to store KV activations of tokens which may or may not be accepted.

NOTE: This will be replaced by speculative config in the future; it is present to enable correctness tests until then.

Default: 0

--scheduling-policy¶

Possible choices: fcfs, priority

The scheduling policy to use:

• "fcfs" means first come first served, i.e. requests are handled in order of arrival.

• "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties).

Default: fcfs

--enable-chunked-prefill, --no-enable-chunked-prefill¶

If True, prefill requests can be chunked based on the remaining max_num_batched_tokens.

Default: None

--disable-chunked-mm-input, --no-disable-chunked-mm-input¶

If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next.

Default: False

--scheduler-cls¶

The scheduler class to use. "vllm.core.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom_class".

Default: vllm.core.scheduler.Scheduler

--disable-hybrid-kv-cache-manager, --no-disable-hybrid-kv-cache-manager¶

If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention.

Default: False

--async-scheduling, --no-async-scheduling¶

EXPERIMENTAL: If set to True, perform async scheduling. This may help reduce the CPU overheads, leading to better latency and throughput. However, async scheduling is currently not supported with some features such as structured outputs, speculative decoding, and pipeline parallelism.

Default: False

VllmConfig¶

Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase.

--speculative-config¶

Speculative decoding configuration.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--kv-transfer-config¶

The configurations for distributed KV cache transfer.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--kv-events-config¶

The configurations for event publishing.

Should either be a valid JSON string or JSON keys passed individually.

Default: None

--compilation-config, -O¶

torch.compile and cudagraph capture configuration for the model.

As a shorthand, -O<n> can be used to directly specify the compilation level n: -O3 is equivalent to -O.level=3 (same as -O='{"level":3}'). Currently, -O and -O= are supported as well but this will likely be removed in favor of clearer -O syntax in the future.

NOTE: level 0 is the default level without any optimization. level 1 and 2 are for internal testing only. level 3 is the recommended level for production, also default in V1.

You can specify the full compilation config like so: {"level": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}

Should either be a valid JSON string or JSON keys passed individually.

Default: {'level': None, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': True, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': None, 'local_cache_dir': None}

--additional-config¶

Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable.

Default: {}

--structured-outputs-config¶

Structured outputs configuration.

Should either be a valid JSON string or JSON keys passed individually.

Default: StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='')

AsyncEngineArgs¶

--enable-log-requests, --no-enable-log-requests¶

Enable logging requests.

Default: False

--disable-log-requests, --no-disable-log-requests¶

[DEPRECATED] Disable logging requests.

Default: True