OpenAI

You can get an API key from the OpenAI console

Chat model

We recommend configuring GPT-4o as your chat model.

YAML

config.yaml

models:
  - name: GPT-4o
    provider: openai
    model: gpt-4o
    apiKey: <YOUR_OPENAI_API_KEY>

Autocomplete model

OpenAI currently does not offer any autocomplete models.

Click here to see a list of autocomplete model providers.

Embeddings model

We recommend configuring text-embedding-3-large as your embeddings model.

YAML

config.yaml

models:
  - name: OpenAI Embeddings
    provider: openai
    model: text-embedding-3-large
    apiKey: <YOUR_OPENAI_API_KEY>
    roles:
      - embed

Reranking model

OpenAI currently does not offer any reranking models.

Click here to see a list of reranking model providers.

OpenAI compatible servers / APIs

OpenAI compatible servers

• KoboldCpp
• text-gen-webui
• FastChat
• LocalAI
• llama-cpp-python
• TensorRT-LLM
• vLLM
• BerriAI/litellm

OpenAI compatible APIs

• Anyscale Endpoints
• Anyscale Private Endpoints

If you are using an OpenAI compatible server / API, you can change the apiBase like this:

YAML

config.yaml

models:
  - name: OpenAI-compatible server / API
    provider: openai
    model: MODEL_NAME 
    apiBase: http://localhost:8000/v1
    apiKey: <YOUR_CUSTOM_API_KEY>

To force usage of chat/completions instead of completions endpoint you can set:

YAML

config.yaml

models:
  - name: OpenAI-compatible server / API
    provider: openai
    model: MODEL_NAME 
    apiBase: http://localhost:8000/v1
    useLegacyCompletionsEndpoint: true