Questions About VLLM Parallel Inference

[tulvgengenr]

In evaluate_vllm.sh and evaluate_72B_vllm.sh, I see max_workers=1. This configuration fails to leverage VLLM's parallel processing advantages. However, when I set max_workers > 1, an error occurs. Have you encountered this issue?

[sabulin]

max_workers is the number of concurrent prediction requests. When testing a local VLLM, you should just set this parameter to 1. When testing via an API, you can set it to a value greater than 1 to speed up the process.

[sabulin]

Of course, if your local VLLM is not large, you can also set this parameter higher. Setting it to 1 is a cautious measure to avoid OOM errors.

[tulvgengenr]

Thank you for your response! Additionally, could you please provide the code for deploying the editscore service so we can access it via the API? We would be extremely grateful.

[sabulin]

You can refer to the code in examples/OmniGen2-RL/reward_server. This code can deploy the editscore service, which enables EditScore to receive data from the RL process and return score

[tulvgengenr]

Hello, in examples/OmniGen2-RL path, I run the shell for starting server by bash reward_server/start_multi_machines.sh --model_name=editscore_7B_pass4 --config_path=reward_server/server_configs/editscore_7B_avg4.yml. When I run python reward_server/scripts/utils/reward_server_sanity_check.py --config_path=reward_server/server_configs/editscore_7B_avg4.yml to check the server, there is a following error:

HTTP error: 503
All retries failed, last exception: HTTP 503
Traceback (most recent call last):
  File "/apdcephfs_nj10/share_301739632/jacobjzli/workspace/Qwen3-VL/qwen-vl-finetune/data_process/test_i2i_rm_acc/EditScore/examples/OmniGen2-RL/reward_server/scripts/utils/reward_server_sanity_check.py", line 65, in <module>
    main(args)
  File "/apdcephfs_nj10/share_301739632/jacobjzli/workspace/Qwen3-VL/qwen-vl-finetune/data_process/test_i2i_rm_acc/EditScore/examples/OmniGen2-RL/reward_server/scripts/utils/reward_server_sanity_check.py", line 52, in main
    scores, rewards, reasoning, meta_data = evaluate_images(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable NoneType object

How can i debug the error? Thank you~

[Luciennnnnnn]

@tulvgengenr Could you please check the log files under reward_server/logs/? It’s usually caused by the server not being started. You can provide error information so we can help you starting the server.