Introduction

FloTorch Gateway is a lightweight application that allows you to connect your AI resources to the FloTorch Console. Gateway exposes a OpenAI Compatible API, that you can use to interact with your AI resources. Being OpenAI Compatible, you can use it as a drop-in replacement for any OpenAI API client and start using FloTorch with minimal changes.

Currently, Gateway supports following OpenAI Compatible APIs:

• Chat Completions - Allowing you to interact with your AI resources using natural language.
• Embeddings - Allowing you to generate embeddings for your text.
• Vector Store - Allowing you to search your Vector Database.

Beyond OpenAI Compatible APIs, Gateway also supports:

• RAG Endpoints - Allowing you to integrate a RAG workflow in your application with the help of your own Vector Store.

Apart from unified API layer, Gateway also provides a set of tools to help you manage your AI resources.

• Guardrails - Allowing you to control the behavior of your AI resources with the help of your own guardrails.
• Model Registry - Allowing you to manage your AI models and their versions.
• Model Routing - Allowing you to route your requests based on your own custom logic.
• Prompts - Allowing you to manage your prompts and their versions.

With robust model registry and routing, you can easily scale and test your AI applications with minimal effort

Using Gateway with OpenAI API Client

To use Gateway with OpenAI API Client, you need to set OPENAI_BASE_URL to the base URL of Gateway and OPENAI_API_KEY to the secret key of the API Key you created in the FloTorch Console.

export OPENAI_BASE_URL=https://<gateway-url>/api/openai/v1
export OPENAI_API_KEY=<secret-key>

Once the environment variables are set, you can use the OpenAI API Client to interact with Gateway.

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4.1",
    input="Write a one-sentence bedtime story about a unicorn."
)
print(response.output_text)