The project currently contains an evaluation code for 2 datasets containing hard examples for Vision language models (VLM's)
The 2 known benchmarks are
-
sugar-crepe (available at here
-
MMBenchmark (available at OpenCompass Project page)
"...the go to statement should be abolished..." [1].
according to a few executions, the prompt
"Question: The following is a multiple choice question. Choose an answer by it's number
...: \n 1.There is a tower in the image\n 2. There is a castle in the image.\n Answer:"
returns always A.
For installation and running of LLaMA2 using transformers python package (by Huggingface.co) you will need to create a read token for the LLaMA resource. More information can be found here