Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference Output Format Issues in GUI Odyssey #7

Open
WRWA opened this issue Nov 6, 2024 · 6 comments
Open

Inference Output Format Issues in GUI Odyssey #7

WRWA opened this issue Nov 6, 2024 · 6 comments

Comments

@WRWA
Copy link

WRWA commented Nov 6, 2024

I've been following the quickstart documentation closely, but I'm encountering issues with the inference output format. Specifically, instead of the expected structured outputs, the results appear in unexpected formats, such as:

Natural language text, rather than the structured format.
Output with random or nonsensical numbers.
Results in Chinese, which was not anticipated.
I would greatly appreciate any guidance on resolving these output inconsistencies to achieve the expected output format. If there are specific configurations or parameters I need to adjust, please let me know.

Thank you for your assistance!

@Lqf-HFNJU
Copy link
Collaborator

Lqf-HFNJU commented Nov 9, 2024

Hi, There could be two possible reasons:

  1. You could check if the model weights were loaded correctly. The OdysseyAgent in the quickstart only loads the weights of Qwen-VL, which lacks the ability to produce structured actions. The pretrained model we used can be found here: https://huggingface.co/collections/hflqf88888/gui-odyssey-6683bac37ad6fe37b1215c18.

  2. Ensure that the prompt is formatted correctly. You can refer to lines 206-224 in the following file for the correct format: https://github.com/OpenGVLab/GUI-Odyssey/blob/master/src/eval_mm/evaluate_GUIOdyssey.py.

@WRWA
Copy link
Author

WRWA commented Nov 13, 2024

Thank you for response. so I didn't change anything from original code and I just changed the model from OdysseyAgent to hflqf88888/OdysseyAgent-random. but when I try to evaluate the model I'm encountering a configuration class mismatch error related to QWenConfig.

Environment

  • Model: hflqf88888/OdysseyAgent-random
  • Framework: PyTorch (Distributed Training)
  • Python version: 3.10

Error Message

python
ValueError: The model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has <class 'configuration_qwen.QWenConfig'> and you passed <class 'transformers_modules.hflqf88888.OdysseyAgent-random.557606768057ff297bb76398800f936bbbd14e7c.configuration_qwen.QWenConfig'>. Fix one of those so they match!

I'm using the following eval.sh

#!/bin/bash

checkpoint=hflqf88888/OdysseyAgent-random
ds=random_split # one of "app_split", "device_split", "random_split", "task_split"
DIR=`pwd`
exp_name=OdysseyAgent_$ds
mkdir -p output/"$exp_name"

GPUS_PER_NODE=4
NNODES=1
NODE_RANK=0
MASTER_ADDR=localhost
MASTER_PORT=$((RANDOM % 30001 + 20000))
GPUS=$((GPUS_PER_NODE * NNODES))

DISTRIBUTED_ARGS="
    --nproc_per_node $GPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT "

echo $ds
echo $checkpoint

torchrun $DISTRIBUTED_ARGS eval_mm/evaluate_GUIOdyssey.py \
    --checkpoint $checkpoint --dataset $ds --batch-size 4 --his_len 4#!/bin/bash

checkpoint=hflqf88888/OdysseyAgent-random
ds=random_split # one of "app_split", "device_split", "random_split", "task_split"
DIR=`pwd`
exp_name=OdysseyAgent_$ds
mkdir -p output/"$exp_name"

GPUS_PER_NODE=4
NNODES=1
NODE_RANK=0
MASTER_ADDR=localhost
MASTER_PORT=$((RANDOM % 30001 + 20000))
GPUS=$((GPUS_PER_NODE * NNODES))

DISTRIBUTED_ARGS="
    --nproc_per_node $GPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT "

echo $ds
echo $checkpoint

torchrun $DISTRIBUTED_ARGS eval_mm/evaluate_GUIOdyssey.py \
    --checkpoint $checkpoint --dataset $ds --batch-size 4 --his_len 4 

Additional Context

Using distributed training with 4 GPUs
The error occurs consistently across all ranks (0-3)
The script fails immediately during model loading
I've tried to remove hf transformers module cache but errors keep occurring

Question

Have you encountered this QWenConfig class inconsistency issue before? What would be the recommended way to resolve this configuration mismatch?

@Lqf-HFNJU
Copy link
Collaborator

Hi,
You could try running git clone https://huggingface.co/hflqf88888/OdysseyAgent-random to download the model to your local machine. Then, update the checkpoint parameter in the script to point to your local path: checkpoint=/path/to/your/local/path
This should help solve the issue mentioned above.

@WRWA
Copy link
Author

WRWA commented Nov 13, 2024

I followed your suggestion to clone the model locally using:
git clone https://huggingface.co/hflqf88888/OdysseyAgent-random

And updated the checkpoint parameter to point to my local path. However, I'm still encountering the same AttributeError:

AttributeError: 'QWenTokenizer' object has no attribute 'IMAGE_ST'

The full traceback shows that the error occurs in tokenization_qwen.py line 227:

if surface_form not in SPECIAL_TOKENS + self.IMAGE_ST:

This error appears across all ranks (0-3) when attempting to initialize the tokenizer. Could you please provide additional guidance on resolving this IMAGE_ST attribute error? Is there perhaps a specific version of dependencies I should be using, or are there any additional setup steps I might be missing?

@Lqf-HFNJU
Copy link
Collaborator

Maybe you could check out this link: https://huggingface.co/MMInstruction/Silkie/discussions/1.
Try updating your version of Transformers to 4.32.0; I tested it and found that it works.

@ApolloRay
Copy link

Maybe you could check out this link: https://huggingface.co/MMInstruction/Silkie/discussions/1. Try updating your version of Transformers to 4.32.0; I tested it and found that it works.

I try this, it can work. But during inference, result show that tensor in different cuda devices, which is an error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants