Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference register fix #114

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Inference register fix #114

wants to merge 6 commits into from

Conversation

heyjustinai
Copy link
Member

@heyjustinai heyjustinai commented Nov 12, 2024

Do not land - pending issue on sdk

What does this PR do?

Added register model in inference examples, modified message structure to typed dict

from SystemMessage to{"content": "You are a helpful assistant", "role": "system"}

Closes #113

Feature/Issue validation/testing/test plan

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.

Please also list any relevant details for your test configuration or test plan.

Sources

Please link relevant resources if necessary.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Thanks for contributing 🎉!

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 12, 2024
@heyjustinai heyjustinai marked this pull request as draft November 12, 2024 02:45
Copy link
Contributor

@yanxi0830 yanxi0830 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need to call register anymore. Update your run.yaml file to start the server with the pre-registred resources. E.g.

inference:
  - provider_id: fireworks-0
    provider_type: remote::fireworks
    config:
      url: https://api.fireworks.ai/inference
      api_key: null
...
models: 
  - model_id: Llama3.2-3B-Instruct
    provider_id: fireworks-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't run inference examples
3 participants