Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable export for inference when eval model is loaded from buffer #21422

Closed
wants to merge 12 commits into from
Closed

Enable export for inference when eval model is loaded from buffer #21422

wants to merge 12 commits into from

Conversation

carzh
Copy link
Contributor

@carzh carzh commented Jul 20, 2024 •
edited
Loading

Description

Modifies the eval model in-place to enable exporting for inference when the eval model is provided through buffer.

Notes

  • Could not find a way to modify the eval model graph in-place with existing methods, so we create a wrapper class defined in ExportModelForInferencing that retrieves the mutable graph and the mutable model. We assume that exporting for inference is the last step in a training session's lifetime.

Motivation and Context

skottmckay
skottmckay previously approved these changes Jul 25, 2024
View reviewed changes
baijumeswani
baijumeswani previously approved these changes Jul 25, 2024
View reviewed changes
@carzh carzh mentioned this pull request Aug 2, 2024
Merged
carzh added a commit that referenced this pull request Aug 9, 2024
@carzh
Enable exporting for inference when loading from buffer without behav…
eeef0c8 
…ior changes (#21601)

### Description
Added eval model buffer as optional field in Module so that you can
export for inference using the eval model stored as a buffer.

### Motivation and Context
- Resolves #21152 
- Previous solution (PR #21422) produced an eval model that was specific
to the EP's used to train because of unavoidable runtime optimizations
that changed the graph stored with the eval session.
@carzh
Copy link
Contributor Author

carzh commented Aug 12, 2024

closing -- alternative solution was merged in

@carzh carzh closed this Aug 12, 2024
@carzh carzh deleted the carzh/export_for_inference_working branch August 12, 2024 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skottmckay skottmckay skottmckay left review comments

@baijumeswani baijumeswani baijumeswani left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Training] Cannot export model for inferencing from session created from buffers
3 participants
@carzh @skottmckay @baijumeswani