Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support selecting Object Detection models #1359

Merged
merged 13 commits into from
Sep 21, 2024

Conversation

Alextopher
Copy link
Contributor

@Alextopher Alextopher commented Jul 2, 2024

This PR is for part 1 of #1354. It focuses on adding a model selection interface for models that exist in photonvision_config/models/. Upon completion we can ship more than 1 model and users could upload their own through ssh without deleting the shipped model. This PR also adds the abstractions need to support more DNN backends (say OpenCV, or RPI AI Kit)

Up next is adding a CRUD interface for managing models through the UI.

@Alextopher Alextopher force-pushed the custom-models branch 2 times, most recently from 7a68919 to f3dbec3 Compare July 2, 2024 03:00
@mcm001
Copy link
Contributor

mcm001 commented Jul 2, 2024

Sadly I think I need to click the approve workflow button every time you push until you make at least one contribution

@Alextopher
Copy link
Contributor Author

Sadly I think I need to click the approve workflow button every time you push until you make at least one contribution

Skill issue on my part. Sorry!

@Alextopher
Copy link
Contributor Author

Alextopher commented Jul 2, 2024

What do we think about the design for defining model labels?

note-640-640-yolov5s-labels.txt
note-640-640-yolov5s.rknn

My thought process is that someday when we are able to support uploading models in different formats we might see something like:

note-640-640-yolov5s-labels.txt
note-640-640-yolov5s.onnx
note-640-640-yolov5s.rknn
note-640-640-yolov5s.tflite

@mcm001
Copy link
Contributor

mcm001 commented Jul 2, 2024

I think that potential conflict (with keeping track of label names/number of classes across different model types with the same name) makes me a little scared.

@Alextopher
Copy link
Contributor Author

different model types

What I'm going for is all those files encode the same "model" - just in different file formats. The difference between onnx and rknn isn't topological, both have the same weights & structure, it's just they use a different format for encoding.

The metadata format I'm going for is name-width-height-model.format.

The alternative is you have multiple copies of the same model (keyed by '"name"`) and you risk screwing up writing the labels file.

@Alextopher
Copy link
Contributor Author

I feel pretty good about the implementation so far. I have a question, how can I go about getting the list of available models from core to the client? I need to give the UI a List<String>.

@Alextopher Alextopher marked this pull request as ready for review July 4, 2024 16:32
@Alextopher Alextopher requested a review from a team as a code owner July 4, 2024 16:32
@Alextopher
Copy link
Contributor Author

Ready for review - if someone is able to test this code running on hardware that would be great.

@Alextopher
Copy link
Contributor Author

At the moment this is hung up on extracting models from the JAR. There is some kind of platform dependency - I'll work through it and move on to testing the actual implementation.

@Alextopher
Copy link
Contributor Author

I'm preparing a more reviewable patch set!

@Alextopher Alextopher force-pushed the custom-models branch 4 times, most recently from 47a54a9 to 865f829 Compare September 19, 2024 07:37
@Alextopher
Copy link
Contributor Author

Alextopher commented Sep 19, 2024

@mcm001 I think this is the minimum abstractions needed to start #1340 and the uploading part of #1354

@Alextopher
Copy link
Contributor Author

To have a second model to test with I just copied the existing one but slightly changed the name.

cp photon-server/src/main/resources/models/note-640-640-yolov5s.rknn photon-server/src/main/resources/models/note-640-640-yolov5m.rknn
cp photon-server/src/main/resources/models/note-640-640-yolov5s-labels.txt photon-server/src/main/resources/models/note-640-640-yolov5m-labels.txt

Copy link
Contributor

@mcm001 mcm001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ship it idk man it works on my computer

@mcm001 mcm001 merged commit 27cb69c into PhotonVision:master Sep 21, 2024
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants