About CIRCO test set results

[vecum0814]

Hi there!
First of all, thanks for your amazing work.

I noticed that your BGE-VL-MLLM made remarkable 42.2 of mAP@5 score with CIRCO test dataset.

However, I tried to reproduce the result with both of BGE-VL-MLLM-S1 and S2, but the results were not reproduced as follows:

While checking your code 'eval_Circo.py', and circo_query.jsonl, It seems like you were both using the 'relative_caption' and 'shared_concept' in your query.

Was the score of 42.2 of mAP@5 with CIRCO test set made with this combinational queries?
I feel like it should be only evaluated with the 'relative_caption'.

Thanks!

[JUNJIE99]

Hi, thank you for your interest in our work!

For the CIRCO benchmark, the text queries use both shared_concept and relative_caption. We followed the common practices adopted in most previous methods. For example, the implementation of MagicLens (link) also evaluates using combinational queries.