[Bug]: Unable to run Phi4 with tensor-parallel-size 4 torch.compile compatiblity

[roguetech]

Your current environment

INFO 04-03 15:28:10 [init.py:256] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.8.0-1024-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.5.82
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A10G
GPU 1: NVIDIA A10G
GPU 2: NVIDIA A10G
GPU 3: NVIDIA A10G

Nvidia driver version: 555.42.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7R32
CPU family: 23
Model: 49
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 1
Stepping: 0
BogoMIPS: 5599.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 768 KiB (24 instances)
L1i cache: 768 KiB (24 instances)
L2 cache: 12 MiB (24 instances)
L3 cache: 96 MiB (6 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.50.0.dev0
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB PHB PHB 0-47 0 N/A
GPU1 PHB X PHB PHB 0-47 0 N/A
GPU2 PHB PHB X PHB 0-47 0 N/A
GPU3 PHB PHB PHB X 0-47 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NCCL_P2P_DISABLE=1
LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

running the following results in the below:

python3 -m vllm.entrypoints.openai.api_server --model microsoft/phi-4 --gpu-memory-utilization 0.90 --max-model-len 16000 --disable-log-requests --tensor-parallel-size 4

INFO 04-03 15:23:00 [init.py:256] Automatically detected platform cuda.
INFO 04-03 15:23:02 [api_server.py:977] vLLM API server version 0.8.1
INFO 04-03 15:23:02 [api_server.py:978] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='microsoft/phi-4', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16000, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=4, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 802/802 [00:00<00:00, 12.3MB/s]
INFO 04-03 15:23:08 [config.py:583] This model supports multiple tasks: {'reward', 'classify', 'score', 'embed', 'generate'}. Defaulting to 'generate'.
INFO 04-03 15:23:09 [config.py:1515] Defaulting to use mp for distributed inference
INFO 04-03 15:23:09 [config.py:1693] Chunked prefill is enabled with max_num_batched_tokens=2048.
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.7k/17.7k [00:00<00:00, 141MB/s]
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.61M/1.61M [00:00<00:00, 65.7MB/s]
merges.txt: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 917k/917k [00:00<00:00, 35.6MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.25M/4.25M [00:00<00:00, 79.1MB/s]
added_tokens.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.50k/2.50k [00:00<00:00, 49.1MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 95.0/95.0 [00:00<00:00, 1.95MB/s]
generation_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 156/156 [00:00<00:00, 2.97MB/s]
INFO 04-03 15:23:11 [core.py:53] Initializing a V1 LLM engine (v0.8.1) with config: model='microsoft/phi-4', speculative_config=None, tokenizer='microsoft/phi-4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=microsoft/phi-4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 04-03 15:23:11 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 04-03 15:23:11 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 10485760, 10, 'psm_fe6243c6'), local_subscribe_addr='ipc:///tmp/a1df557c-8758-424f-86bf-0388b32e971e', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-03 15:23:12 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x79bf1037ec20>
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:12 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_15babbc7'), local_subscribe_addr='ipc:///tmp/3f914a99-dbac-4af6-8281-84404d56958d', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-03 15:23:12 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x79bf1037ec50>
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:12 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c58392db'), local_subscribe_addr='ipc:///tmp/9dff411b-97fa-4ca7-8cef-d8c24d835315', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-03 15:23:13 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x79bf1037e2c0>
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:13 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_9cb45ffa'), local_subscribe_addr='ipc:///tmp/00224184-d3bf-4270-a4f8-80914d393503', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-03 15:23:13 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x79bf1037ec50>
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:13 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_28187e6a'), local_subscribe_addr='ipc:///tmp/af52adbf-b1d1-430d-8824-65322c95998d', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:14 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:14 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:14 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:14 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:14 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:14 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=2 pid=4396) WARNING 04-03 15:23:14 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=3 pid=4411) WARNING 04-03 15:23:14 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=1 pid=4372) WARNING 04-03 15:23:14 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=4360) WARNING 04-03 15:23:14 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_6de8e61d'), local_subscribe_addr='ipc:///tmp/1fab129a-7306-4549-83dc-7ed7ac0e8597', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:14 [parallel_state.py:967] rank 3 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 3
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:14 [parallel_state.py:967] rank 2 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 2
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [parallel_state.py:967] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:14 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:14 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:14 [parallel_state.py:967] rank 1 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:14 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=2 pid=4396) INFO 04-03 15:23:14 [gpu_model_runner.py:1164] Starting to load model microsoft/phi-4...
(VllmWorker rank=3 pid=4411) INFO 04-03 15:23:14 [gpu_model_runner.py:1164] Starting to load model microsoft/phi-4...
(VllmWorker rank=1 pid=4372) INFO 04-03 15:23:14 [gpu_model_runner.py:1164] Starting to load model microsoft/phi-4...
(VllmWorker rank=0 pid=4360) INFO 04-03 15:23:14 [gpu_model_runner.py:1164] Starting to load model microsoft/phi-4...
(VllmWorker rank=1 pid=4372) WARNING 04-03 15:23:14 [config.py:3670] torch.compile is turned on, but the model microsoft/phi-4 does not support it. Please open an issue on GitHub if you want it to be supported.
(VllmWorker rank=3 pid=4411) WARNING 04-03 15:23:14 [config.py:3670] torch.compile is turned on, but the model microsoft/phi-4 does not support it. Please open an issue on GitHub if you want it to be supported.
(VllmWorker rank=2 pid=4396) WARNING 04-03 15:23:14 [config.py:3670] torch.compile is turned on, but the model microsoft/phi-4 does not support it. Please open an issue on GitHub if you want it to be supported.
(VllmWorker rank=0 pid=4360) WARNING 04-03 15:23:14 [config.py:3670] torch.compile is turned on, but the model microsoft/phi-4 does not support it. Please open an issue on GitHub if you want it to be supported.
(VllmWorker rank=2 pid=4396) Process ForkProcess-1:3:
CRITICAL 04-03 15:23:14 [multiproc_executor.py:48] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=0 pid=4360) Process ForkProcess-1:1:
CRITICAL 04-03 15:23:14 [multiproc_executor.py:48] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
CRITICAL 04-03 15:23:14 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=2 pid=4396) Traceback (most recent call last):
(VllmWorker rank=2 pid=4396) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(VllmWorker rank=2 pid=4396) self.run()
(VllmWorker rank=2 pid=4396) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(VllmWorker rank=2 pid=4396) self._target(*self._args, **self._kwargs)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 311, in worker_main
(VllmWorker rank=2 pid=4396) worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 244, in init
(VllmWorker rank=2 pid=4396) self.worker.load_model()
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 136, in load_model
(VllmWorker rank=2 pid=4396) self.model_runner.load_model()
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1167, in load_model
(VllmWorker rank=2 pid=4396) self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorker rank=2 pid=4396) return loader.load_model(vllm_config=vllm_config)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 423, in load_model
(VllmWorker rank=2 pid=4396) model = _initialize_model(vllm_config=vllm_config)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 126, in _initialize_model
(VllmWorker rank=3 pid=4411) Process ForkProcess-1:4:
(VllmWorker rank=2 pid=4396) return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 480, in init
CRITICAL 04-03 15:23:14 [multiproc_executor.py:48] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=2 pid=4396) self.model = self._init_model(vllm_config=vllm_config,
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 517, in _init_model
(VllmWorker rank=2 pid=4396) return LlamaModel(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 151, in init
(VllmWorker rank=2 pid=4396) old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 319, in init
(VllmWorker rank=2 pid=4396) self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 557, in make_layers
(VllmWorker rank=2 pid=4396) [PPMissingLayer() for _ in range(start_layer)] + [
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 558, in
(VllmWorker rank=2 pid=4396) maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 321, in
(VllmWorker rank=2 pid=4396) lambda prefix: layer_type(config=config,
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 237, in init
(VllmWorker rank=2 pid=4396) self.self_attn = LlamaAttention(
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 122, in init
(VllmWorker rank=1 pid=4372) Process ForkProcess-1:2:
(VllmWorker rank=2 pid=4396) assert self.total_num_kv_heads % tp_size == 0
(VllmWorker rank=2 pid=4396) AssertionError
CRITICAL 04-03 15:23:14 [multiproc_executor.py:48] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=0 pid=4360) Traceback (most recent call last):
(VllmWorker rank=0 pid=4360) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(VllmWorker rank=0 pid=4360) self.run()
(VllmWorker rank=0 pid=4360) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(VllmWorker rank=0 pid=4360) self._target(*self._args, **self._kwargs)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 311, in worker_main
(VllmWorker rank=0 pid=4360) worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 244, in init
(VllmWorker rank=0 pid=4360) self.worker.load_model()
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 136, in load_model
(VllmWorker rank=0 pid=4360) self.model_runner.load_model()
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1167, in load_model
(VllmWorker rank=0 pid=4360) self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorker rank=0 pid=4360) return loader.load_model(vllm_config=vllm_config)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 423, in load_model
(VllmWorker rank=0 pid=4360) model = _initialize_model(vllm_config=vllm_config)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 126, in _initialize_model
(VllmWorker rank=0 pid=4360) return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 480, in init
(VllmWorker rank=0 pid=4360) self.model = self._init_model(vllm_config=vllm_config,
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 517, in _init_model
(VllmWorker rank=0 pid=4360) return LlamaModel(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 151, in init
(VllmWorker rank=0 pid=4360) old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 319, in init
(VllmWorker rank=0 pid=4360) self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 557, in make_layers
(VllmWorker rank=0 pid=4360) [PPMissingLayer() for _ in range(start_layer)] + [
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 558, in
(VllmWorker rank=0 pid=4360) maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 321, in
(VllmWorker rank=0 pid=4360) lambda prefix: layer_type(config=config,
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 237, in init
(VllmWorker rank=0 pid=4360) self.self_attn = LlamaAttention(
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 122, in init
(VllmWorker rank=0 pid=4360) assert self.total_num_kv_heads % tp_size == 0
(VllmWorker rank=0 pid=4360) AssertionError
CRITICAL 04-03 15:23:14 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=3 pid=4411) Traceback (most recent call last):
(VllmWorker rank=3 pid=4411) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(VllmWorker rank=3 pid=4411) self.run()
(VllmWorker rank=3 pid=4411) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(VllmWorker rank=3 pid=4411) self._target(*self._args, **self._kwargs)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 311, in worker_main
(VllmWorker rank=3 pid=4411) worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 244, in init
(VllmWorker rank=3 pid=4411) self.worker.load_model()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 136, in load_model
(VllmWorker rank=3 pid=4411) self.model_runner.load_model()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1167, in load_model
(VllmWorker rank=3 pid=4411) self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorker rank=3 pid=4411) return loader.load_model(vllm_config=vllm_config)
(VllmWorker rank=1 pid=4372) Traceback (most recent call last):
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 423, in load_model
(VllmWorker rank=3 pid=4411) model = _initialize_model(vllm_config=vllm_config)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 126, in _initialize_model
(VllmWorker rank=3 pid=4411) return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 480, in init
(VllmWorker rank=3 pid=4411) self.model = self._init_model(vllm_config=vllm_config,
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 517, in _init_model
(VllmWorker rank=1 pid=4372) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(VllmWorker rank=3 pid=4411) return LlamaModel(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=1 pid=4372) self.run()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 151, in init
(VllmWorker rank=1 pid=4372) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(VllmWorker rank=3 pid=4411) old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker rank=1 pid=4372) self._target(*self._args, **self._kwargs)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 319, in init
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 311, in worker_main
(VllmWorker rank=3 pid=4411) self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker rank=1 pid=4372) worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 557, in make_layers
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 244, in init
(VllmWorker rank=3 pid=4411) [PPMissingLayer() for _ in range(start_layer)] + [
(VllmWorker rank=1 pid=4372) self.worker.load_model()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 558, in
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 136, in load_model
(VllmWorker rank=3 pid=4411) maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker rank=1 pid=4372) self.model_runner.load_model()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 321, in
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1167, in load_model
(VllmWorker rank=3 pid=4411) lambda prefix: layer_type(config=config,
(VllmWorker rank=1 pid=4372) self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 237, in init
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorker rank=3 pid=4411) self.self_attn = LlamaAttention(
(VllmWorker rank=1 pid=4372) return loader.load_model(vllm_config=vllm_config)
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 122, in init
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 423, in load_model
(VllmWorker rank=3 pid=4411) assert self.total_num_kv_heads % tp_size == 0
(VllmWorker rank=1 pid=4372) model = _initialize_model(vllm_config=vllm_config)
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 126, in _initialize_model
(VllmWorker rank=3 pid=4411) AssertionError
(VllmWorker rank=1 pid=4372) return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 480, in init
(VllmWorker rank=1 pid=4372) self.model = self._init_model(vllm_config=vllm_config,
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 517, in _init_model
(VllmWorker rank=1 pid=4372) return LlamaModel(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 151, in init
(VllmWorker rank=1 pid=4372) old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 319, in init
(VllmWorker rank=1 pid=4372) self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 557, in make_layers
(VllmWorker rank=1 pid=4372) [PPMissingLayer() for _ in range(start_layer)] + [
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 558, in
(VllmWorker rank=1 pid=4372) maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 321, in
(VllmWorker rank=1 pid=4372) lambda prefix: layer_type(config=config,
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 237, in init
(VllmWorker rank=1 pid=4372) self.self_attn = LlamaAttention(
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 122, in init
(VllmWorker rank=1 pid=4372) assert self.total_num_kv_heads % tp_size == 0
(VllmWorker rank=1 pid=4372) AssertionError
(VllmWorker rank=1 pid=4372) Exception ignored in: <function ShmRingBuffer.del at 0x79bf1034b7f0>
(VllmWorker rank=1 pid=4372) Traceback (most recent call last):
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 148, in del
(VllmWorker rank=3 pid=4411) Exception ignored in: <function Context.del at 0x79bf970e7910>
(VllmWorker rank=3 pid=4411) Traceback (most recent call last):
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/zmq/sugar/context.py", line 140, in del
(VllmWorker rank=1 pid=4372) self.shared_memory.close()
(VllmWorker rank=1 pid=4372) File "/usr/lib/python3.10/multiprocessing/shared_memory.py", line 230, in close
(VllmWorker rank=3 pid=4411) self.destroy()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/zmq/sugar/context.py", line 322, in destroy
(VllmWorker rank=3 pid=4411) self.term()
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/zmq/sugar/context.py", line 264, in term
(VllmWorker rank=3 pid=4411) super().term()
(VllmWorker rank=3 pid=4411) File "_zmq.py", line 564, in zmq.backend.cython._zmq.Context.term
(VllmWorker rank=3 pid=4411) File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
(VllmWorker rank=3 pid=4411) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 303, in signal_handler
(VllmWorker rank=3 pid=4411) raise SystemExit()
(VllmWorker rank=3 pid=4411) SystemExit:
(VllmWorker rank=1 pid=4372) self._mmap.close()
(VllmWorker rank=1 pid=4372) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 303, in signal_handler
(VllmWorker rank=1 pid=4372) raise SystemExit()
(VllmWorker rank=1 pid=4372) SystemExit:
(VllmWorker rank=2 pid=4396) Exception ignored in: <function Socket.del at 0x79bf970e63b0>
(VllmWorker rank=2 pid=4396) Traceback (most recent call last):
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/zmq/sugar/socket.py", line 184, in del
(VllmWorker rank=2 pid=4396) def del(self):
(VllmWorker rank=2 pid=4396) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 303, in signal_handler
(VllmWorker rank=2 pid=4396) raise SystemExit()
(VllmWorker rank=2 pid=4396) SystemExit:
(VllmWorker rank=0 pid=4360) Exception ignored in: <function Socket.del at 0x79bf970e63b0>
(VllmWorker rank=0 pid=4360) Traceback (most recent call last):
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/zmq/sugar/socket.py", line 184, in del
(VllmWorker rank=0 pid=4360) def del(self):
(VllmWorker rank=0 pid=4360) File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 303, in signal_handler
(VllmWorker rank=0 pid=4360) raise SystemExit()
(VllmWorker rank=0 pid=4360) SystemExit:
Killed

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[vsenik]

Could be started with pipeline-parallel.
python3 -m vllm.entrypoints.openai.api_server --model microsoft/phi-4 --gpu-memory-utilization 0.90 --max-model-len 16000 --disable-log-requests --pipeline-parallel-size 4

[OmarMohammed88]

Could be started with pipeline-parallel. python3 -m vllm.entrypoints.openai.api_server --model microsoft/phi-4 --gpu-memory-utilization 0.90 --max-model-len 16000 --disable-log-requests --pipeline-parallel-size 4

@vsenik Do u know how to use it using the standard loading ,
This is my code and gives me same error

    llm = VLLM(
        model='microsoft/phi-4',
        tensor_parallel_size=4,
        trust_remote_code=True,  # mandatory for hf models
        max_new_tokens=16000,
    )

[vsenik]

You have problem because model couldn't be started with tensor_parallel on 4 device, you could choose 2 or 8. If you want to use exactly 4 devices you should switch to pipeline_parallel. Here difference explained

llm = VLLM(
    model='microsoft/phi-4',
    **tensor_parallel_size**=4,
    trust_remote_code=True,  # mandatory for hf models
    max_new_tokens=16000,
)