vllm.transformers_utils.configs.qwen3_next ¶
Qwen3-Next model configuration
Qwen3NextConfig ¶
Bases: PretrainedConfig
This is the configuration class to store the configuration of a [Qwen3NextModel
]. It is used to instantiate a Qwen3-Next model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of Qwen3-Next-80B-A3B-Instruct Qwen/Qwen3-Next-80B-A3B-Instruct.
Configuration objects inherit from [PretrainedConfig
] and can be used to control the model outputs. Read the documentation from [PretrainedConfig
] for more information.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
vocab_size | `int`, *optional*, defaults to 151936 | Vocabulary size of the model. Defines the number of different tokens that can be represented by the | 151936 |
hidden_size | `int`, *optional*, defaults to 2048 | Dimension of the hidden representations. | 2048 |
intermediate_size | `int`, *optional*, defaults to 5632 | Dimension of the MLP representations. | 5632 |
num_hidden_layers | `int`, *optional*, defaults to 48 | Number of hidden layers in the Transformer encoder. | 48 |
num_attention_heads | `int`, *optional*, defaults to 16 | Number of attention heads for each attention layer in the Transformer encoder. | 16 |
num_key_value_heads | `int`, *optional*, defaults to 2 | This is the number of key_value heads that should be used to implement Grouped Query Attention. If | 2 |
hidden_act | `str`, *optional*, defaults to `"silu"` | The non-linear activation function in the decoder. | 'silu' |
max_position_embeddings | `int`, *optional*, defaults to 32768 | The maximum sequence length that this model might ever be used with. | 32768 |
initializer_range | `float`, *optional*, defaults to 0.02 | The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | 0.02 |
rms_norm_eps | `float`, *optional*, defaults to 1e-06 | The epsilon used by the rms normalization layers. | 1e-06 |
use_cache | `bool`, *optional*, defaults to `True` | Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if | True |
tie_word_embeddings | `bool`, *optional*, defaults to `False` | Whether the model's input and output word embeddings should be tied. | False |
rope_theta | `float`, *optional*, defaults to 10000.0 | The base period of the RoPE embeddings. | 10000.0 |
rope_scaling | `Dict`, *optional* | Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer | None |
partial_rotary_factor | `float`, *optional*, defaults to 0.25 | Percentage of the query and keys which will have rotary embedding. | 0.25 |
attention_bias | `bool`, *optional*, defaults to `False` | Whether to use a bias in the query, key, value and output projection layers during self-attention. | False |
attention_dropout | `float`, *optional*, defaults to 0.0 | The dropout ratio for the attention probabilities. | 0.0 |
head_dim | `int`, *optional*, defaults to 256 | Projection weights dimension in multi-head attention. | 256 |
linear_conv_kernel_dim | `int`, *optional*, defaults to 4 | Kernel size of the convolution used in linear attention layers. | 4 |
linear_key_head_dim | `int`, *optional*, defaults to 128 | Dimension of each key head in linear attention. | 128 |
linear_value_head_dim | `int`, *optional*, defaults to 128 | Dimension of each value head in linear attention. | 128 |
linear_num_key_heads | `int`, *optional*, defaults to 16 | Number of key heads used in linear attention layers. | 16 |
linear_num_value_heads | `int`, *optional*, defaults to 32 | Number of value heads used in linear attention layers. | 32 |
decoder_sparse_step | `int`, *optional*, defaults to 1 | The frequency of the MoE layer. | 1 |
moe_intermediate_size | `int`, *optional*, defaults to 512 | Intermediate size of the routed expert. | 512 |
shared_expert_intermediate_size | `int`, *optional*, defaults to 512 | Intermediate size of the shared expert. | 512 |
num_experts_per_tok | `int`, *optional*, defaults to 10 | Number of selected experts. | 10 |
num_experts | `int`, *optional*, defaults to 512 | Number of routed experts. | 512 |
norm_topk_prob | `bool`, *optional*, defaults to `True` | Whether to normalize the topk probabilities. | True |
output_router_logits | `bool`, *optional*, defaults to `False` | Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss, including load balancing loss and router z-loss. | False |
router_aux_loss_coef | `float`, *optional*, defaults to 0.001 | The aux loss factor for the total loss. | 0.001 |
mlp_only_layers | `list[int]`, *optional*, defaults to `[]` | Indicate which layers use Qwen3NextMLP rather than Qwen3NextSparseMoeBlock The list contains layer index, from 0 to num_layers-1 if we have num_layers layers If | None |
layer_types | `list[str]`, *optional* | Types of each layer (attention or linear). | None |
>>> from transformers import Qwen3NextModel, Qwen3NextConfig
>>> # Initializing a Qwen3Next style configuration
>>> configuration = Qwen3NextConfig()
>>> # Initializing a model from the Qwen3-Next-80B-A3B style configuration
>>> model = Qwen3NextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
Source code in vllm/transformers_utils/configs/qwen3_next.py
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 |
|
base_model_pp_plan class-attribute
instance-attribute
¶
base_model_pp_plan = {
"embed_tokens": (["input_ids"], ["inputs_embeds"]),
"layers": (
["hidden_states", "attention_mask"],
["hidden_states"],
),
"norm": (["hidden_states"], ["hidden_states"]),
}
base_model_tp_plan class-attribute
instance-attribute
¶
base_model_tp_plan = {
"layers.*.self_attn.q_proj": "colwise",
"layers.*.self_attn.k_proj": "colwise",
"layers.*.self_attn.v_proj": "colwise",
"layers.*.self_attn.o_proj": "rowwise",
"layers.*.mlp.experts.*.gate_proj": "colwise",
"layers.*.mlp.experts.*.up_proj": "colwise",
"layers.*.mlp.experts.*.down_proj": "rowwise",
"layers.*.mlp.shared_experts.gate_proj": "colwise",
"layers.*.mlp.shared_experts.up_proj": "colwise",
"layers.*.mlp.shared_experts.down_proj": "rowwise",
"layers.*.mlp.gate_proj": "colwise",
"layers.*.mlp.up_proj": "colwise",
"layers.*.mlp.down_proj": "rowwise",
}
keys_to_ignore_at_inference class-attribute
instance-attribute
¶
shared_expert_intermediate_size instance-attribute
¶
__init__ ¶
__init__(
vocab_size=151936,
hidden_size=2048,
intermediate_size=5632,
num_hidden_layers=48,
num_attention_heads=16,
num_key_value_heads=2,
hidden_act="silu",
max_position_embeddings=32768,
initializer_range=0.02,
rms_norm_eps=1e-06,
use_cache=True,
tie_word_embeddings=False,
rope_theta=10000.0,
rope_scaling=None,
partial_rotary_factor=0.25,
attention_bias=False,
attention_dropout=0.0,
head_dim=256,
linear_conv_kernel_dim=4,
linear_key_head_dim=128,
linear_value_head_dim=128,
linear_num_key_heads=16,
linear_num_value_heads=32,
decoder_sparse_step=1,
moe_intermediate_size=512,
shared_expert_intermediate_size=512,
num_experts_per_tok=10,
num_experts=512,
norm_topk_prob=True,
output_router_logits=False,
router_aux_loss_coef=0.001,
mlp_only_layers=None,
layer_types=None,
**kwargs,
)
Source code in vllm/transformers_utils/configs/qwen3_next.py
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 |
|