SegmentAnything loading state_dict error

[haolapig]

I successfully installed Annolid on Windows by following the documentation:

1. Cloning the GitHub repository
2. Running pip install -e .
3. I also had to reinstall numpy version 1.25.2 because the newer 2.x versions caused size-related issues.
4. I also reinstall pytorch with cuda so that I can utilize my GPU.

The GUI runs well, and I can load videos without any problems. However, when I tried using SAM-based text prompt segmentation by following (https://www.youtube.com/watch?v=ry9bnaajKCs), I encountered an error:

Traceback (most recent call last):
File "d:\annolid\annolid\gui\app.py", line 649, in _grounding_sam
self.canvas.predictAiRectangle(prompt_text)
File "d:\annolid\annolid\gui\widgets\canvas.py", line 274, in predictAiRectangle
self.sam_hq_model = SamHQSegmenter()
^^^^^^^^^^^^^^^^
File "d:\annolid\annolid\segmentation\SAM\sam_hq\sam_hq.py", line 41, in init
self.sam = sam_model_registrymodel_type
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\miniconda3\envs\annolid-env\Lib\site-packages\segment_anything\build_sam.py", line 28, in build_sam_vit_l
return _build_sam(
^^^^^^^^^^^
File "D:\miniconda3\envs\annolid-env\Lib\site-packages\segment_anything\build_sam.py", line 106, in _build_sam
sam.load_state_dict(state_dict)
File "D:\miniconda3\envs\annolid-env\Lib\site-packages\torch\nn\modules\module.py", line 2215, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Sam:
Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

[healthonrails]

First, please ensure that your sam-hq model file is located at ./annolid/segmentation/SAM/sam_hq/sam_hq_vit_l.pth. Verify the integrity of the file by checking its MD5 hash:

md5 sam_hq_vit_l.pth

The correct MD5 hash should be:

MD5 (sam_hq_vit_l.pth) = 08947267966e4264fb39523eccc33f86

If the hash matches, the file is downloaded correctly and is not corrupted.

Next, confirm that sam-hq is installed properly in your environment. You can do this by running the following command in your Annolid environment:

pip install git+https://github.com/SysCV/sam-hq.git

This will ensure that sam-hq is installed correctly in your environment.

[haolapig]

• There are no pth file in /annolid/segmentation/SAM/sam_hq/
  So I downloaded it from https://huggingface.co/lkeab/hq-sam/tree/main/.
  MD5 hash of sam_hq_vit_l.pth: 08947267966e4264fb39523eccc33f86
  The _REMOTE_MODEL_URL = "https://huggingface.co/lkeab/hq-sam/resolve/main/" in sam_hq.py seems not valid anymore.

• I also reinstall pytorch2.4.1+cuda12.4, reinstall annolid 1.2.1(pip -install -e .) and reinstall sam-hq.

• Then I

1. Open Annolid GUI and open my video
2. Select Sam HQ in the droplist
3. In text prompt, I enter "mouse".
   I got the same errors in loading state_dict for Sam.

• I print out checkpoint location and it is correct

[haolapig]

I think I find the solution:

• I reinstall sam-hq but instead of pip install git+https://github.com/SysCV/sam-hq.git, I did following:
  pip install segment-anything-hq
  I check my conda list and I can see segment-anything-hq now.
• In /annolid/segmentation/SAM/sam_hq/sam_hq.py, I modified:

from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

to

from segment_anything_hq import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

Now the text prompt segmentation works.
The source of error could be the conflict between original segmenation-anything (facebook git) and segmentation-anyting-hq

[healthonrails]

I think I find the solution:

• I reinstall sam-hq but instead of pip install git+https://github.com/SysCV/sam-hq.git, I did following:
  pip install segment-anything-hq
  I check my conda list and I can see segment-anything-hq now.
• In /annolid/segmentation/SAM/sam_hq/sam_hq.py, I modified:

from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

to

from segment_anything_hq import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

Now the text prompt segmentation works. The source of error could be the conflict between original segmenation-anything (facebook git) and segmentation-anyting-hq

The setup.py file contains the line "segment-anything @ git+https://github.com/SysCV/sam-hq.git". When you run pip install -e ., it installs the package from the GitHub repository using the alias segment-anything. This line explicitly installs the SAM-HQ package from the SysCV repository but labels it as segment-anything. This prevents any conflict with Meta’s segment-anything package and eliminates the need for a separate package, such as segment-anything-hq.

[haolapig]

The setup.py file contains the line "segment-anything @ git+https://github.com/SysCV/sam-hq.git". When you run pip install -e ., it installs the package from the GitHub repository using the alias segment-anything. This line explicitly installs the SAM-HQ package from the SysCV repository but labels it as segment-anything. This prevents any conflict with Meta’s segment-anything package and eliminates the need for a separate package, such as segment-anything-hq.

As mentioned earlier, I have already tried reinstalling sam-hq using pip install git+https://github.com/SysCV/sam-hq.git, but unfortunately, it did not work for me. The load state_dict continues to reference the original Facebook SAM.

To ensure I hadn't missed anything, I repeated the entire process from scratch on a new Windows machine, and I install Segment Anything specifically from SysCV using pip install git+https://github.com/SysCV/sam-hq.git. However, I encountered the same error related to load state_dict.

I realized that the issue might have been due to the installation tutorial. As the github tutorial mentioned, before running pip install -e ., I should have used pip install git+https://github.com/facebookresearch/segment-anything.git. I had skipped this step and pip install -e . directly. Now, everything is working as expected.

In summary, here are the steps I followed to get Annolid working:

1. Follow the GitHub instructions, but skip the step: pip install git+https://github.com/facebookresearch/segment-anything.git.
2. Reinstall NumPy version 1.5.2, as NumPy 2.0 causes the following error:
   ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject.
3. If you need to use GPU instead of the default CPU, install PyTorch and then reinstall Annolid.
4. Modify /annolid/segmentation/SAM/sam_hq/sam_hq.py to update the model URL:
   _REMOTE_MODEL_URL = "https://huggingface.co/lkeab/hq-sam/tree/main/"

[healthonrails]

1. p install git+https://github.com/facebookresearch/segment-anything.git.

Thank you for sharing your experience! I’m glad to hear that you successfully installed Annolid and it’s working for you. I’m not sure about the environment you’re using to run Annolid, but we recommend using Conda, as outlined in the installation guide: Installation Instructions.