Azure OpenAI and/or LiteLLM Proxy Support

[dcieslak19973]

Would be nice to allow users to specify either a LiteLLM Proxy instance or their own Azure OpenAI instance in addition to support for direct OpenAI and AWS Claude

[emeryberger]

This is possible; we are using LiteLLM in https://github.com/plasma-umass/ChatDBG/. If there's sufficient interest, we will add it (please thumbs-up this issue).

[dcieslak19973]

Thanks; I think I see in the code that you may now be using litellm; less than clear what I would need to set (either environment variable-wise or cli-argument-wise) to connect to my LiteLLM proxy instance...

[nicovank]

I expect we also need to change the code a bit just to let that parameter pass through to the underlying litellm package. I’ll give it a look.

[nicovank]

So from what I understand, the proxy more or less gives the same functionality as having LiteLLM in the code namely abstracting out which service to request from. Do you have the possibility of directly querying the underlying service? e.g. can you specify AZURE_API_KEY, AZURE_API_BASE, and AZURE_API_VERSION directly in the environment? Or are you bound by the proxy = everything must go through the proxy? I assumed Azure OpenAI because that was in the title, but same question if it was another provider.

We currently do a hard check for either OpenAI or AWS credentials, and only pass those through to LiteLLM. We should be able to allow anything LiteLLM already allows instead.