Option to integrate claude API

[roycefanproxy]

Hi,
Since there are a lot of competing LLM API that offers similar functionality, do you guys think it's good idea to support those service like Claude as well?

[Excedrin]

I made a fork and implemented Portkey AI Gateway support, which I ran locally, but then I found litellm and started running that locally instead. Portkey AI Gateway is a bit lighter weight (about 10% the SLOC!), but I think it's also less featureful and it's definitely not as actively developed.

Anyway, litellm works great except that I'm not sure how to get function calling working and I had to edit the model_prices_and_context_window.json to support gemini-1.5-pro-latest. But it works great out of the box for Claude. Note that I'm using it as a proxy, so in my shell_gpt config I have USE_LITELLM=false like this (excerpt):

USE_LITELLM=false
API_BASE_URL=http://localhost:4000/
DEFAULT_MODEL=mixtral
OPENAI_API_KEY=unused_this_is_now_in_litellm_config

litellm_config.yaml

litellm_settings:
  drop_params: true
model_list:
  - model_name: gemini
    litellm_params:
      model: gemini/gemini-1.5-pro-latest
      api_key: os.environ/GEMINI_API_KEY
      safety_settings:
        - category: HARM_CATEGORY_HARASSMENT
          threshold: BLOCK_NONE
        - category: HARM_CATEGORY_HATE_SPEECH
          threshold: BLOCK_NONE
        - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
          threshold: BLOCK_NONE
        - category: HARM_CATEGORY_DANGEROUS_CONTENT
          threshold: BLOCK_NONE
  - model_name: mixtral
    litellm_params:
      model: groq/mixtral-8x7b-32768
      api_key: os.environ/GROQ_API_KEY
  - model_name: llama3
    litellm_params:
      model: groq/llama3-70b-8192
      api_key: os.environ/GROQ_API_KEY
  - model_name: claude
    litellm_params:
      model: anthropic/claude-3-opus-20240229
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: sonnet
    litellm_params:
      model: anthropic/claude-3-sonnet-20240229
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: haiku
    litellm_params:
      model: anthropic/claude-3-haiku-20240307
      api_key: os.environ/ANTHROPIC_API_KEY

and how to run litellm:

docker run --name litellm \
  -d \
  --rm \
  -v $(pwd)/model_prices_and_context_window.json:/usr/local/lib/python3.11/site-packages/litellm/model_prices_and_context_window_backup.json \
  -e LITELLM_LOCAL_MODEL_COST_MAP=True \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  --env-file ~/.api_keys \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

If you don't need gemini just remove the model_prices stuff.

With this setup, I can sgpt --model llama3 "Tell Claude about rasins." | sgpt --model claude "What do you think about this description of rasins?"

[krrishdholakia]

Hey @Excedrin - litellm maintainer here -

I'm not sure how to get function calling working

Where in docs did you check? We have an example here - https://docs.litellm.ai/docs/providers/anthropic#usage---function-calling

Did you want to see how to do function calling with the API?

edit the model_prices_and_context_window.json to support gemini-1.5-pro-latest

We now support this out of the box 🎉

---

Would love to learn how we can improve the proxy for you. Attaching my calendar for your convenience - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat