How to run multiple mixtral in one machine

[ToheartZhang]

I have an 8xA100 GPU machine. I noticed that one Mixtral instance requires at least 2 GPUs. However, when I attempt to run two Mixtral instances on the machine, each allocated 2 GPUs, the second one hangs at Started a local Ray instance. Additionally, the terminal reports fork: retry: Resource temporarily unavailable. It appears that the two Ray instances are causing a conflict. Is there any solution to resolve this issue?

[thesues]

may be you can se env variable CUDA_VISIBLE_DEVICES before starting a LLM instance?

CUDA_VISIBLE_DEVICES="0,1" python llm.py
CUDA_VISIBLE_DEVICES="2,3" python llm.py

[ToheartZhang]

Thanks for your help. I have specified CUDA_VISIBLE_DEVICES. Besides, multiple vllm process works when each model only needs one GPU, where Ray is not used. So I believe the issue is about setting Ray.

[ToheartZhang]

Problem solved by #1058