Skip to content
FloTorch

Model Registry

Model Registry is a central repository for all your AI models. It allows you to manage your models, their versions, and their configurations. FloTorch gateway enables you to interact with multiple providers with a single OpenAI compatible API.

When a request is made to the gateway, the gateway will look for the model in the model registry and route the request to the appropriate provider.When configured with guardrails, the gateway will also check if the model is compliant with the guardrails

A FloTorch model can be configured with

  • Input Guardrails - A set of rules or policies to control the input of the model.
  • Provider Configuration - A set of configuration that you can configure to control the behavior of your AI resources.
  • Output Guardrails - A set of rules or policies to control the output of the model.

It is possible to configure a model with multiple providers and guardrails. The gateway will then route the request to the appropriate provider and apply the guardrails.

Following Routing Strategies are supported:

  • Round Robin - The gateway will route the request to the next provider in the list.
  • Weighted Round Robin - The gateway will route the request to the provider based on the weight assigned to each provider.
  • Fallback - The gateway will route the request to the fallback provider if the request to primary provider fails.

Routing strategy is applied only if more than one provider is configured for a model.