Models

Creating a Model

To create a model, you need to click on New Model button in the top right corner of the FloTorch Console. Provide a name and description for the model. Model name should be unique. Once a model is created, you will be navigated to version configuration page, meanwhile your model will appear in Models Registry.

When a model is created, a default version 1 is created automatically and is set in Draft state. You need to configure the model version and publish it to make it available for routing requests through Gateway.

Model Versions

Model versions are the individual configurations of a model. Each model can have multiple versions. Each version can have a different provider, guardrails, and configuration. Once a model version is published, it is immutable and cannot be changed.

A new version can be created by clicking on New Version button in the top right corner of the FloTorch Console.

Configuring a Model

Model versions can be configured using below instructions.

1. Identify the model you want to configure from the Model Registry table.
2. Click on Actions dropdown button and select Versions.
3. You will be redirected to Model details table.
4. Again here, click on Actions dropdown button and select Configure Model, to open the model version page.

You will be presented with a model configuration form. You can configure the following:

• Input Guardrails - A set of rules or policies to control the input of the model.
• Model Configuration - A set of providers to route the request to
• Output Guardrails - A set of rules or policies to control the output of the model.

Once the model version is configured, you can publish it to make it available for routing requests through Gateway.

Publishing a Model Version

Model versions can be published using the Actions dropdown in the Model Details table.

1. Identify the model version you want to publish.
2. Click on Actions dropdown and select Publish to publish the model version.

Once a model version is published, it is immutable and cannot be changed.

The published model can be used in OpenAI API as flotorch/<model-name>:<version>

Publishing Latest Version of a Model

You can publish the latest version of a model by doing this a model can be used in OpenAI API as flotorch/<model-name> without specifying the version. Internally it is tagged as latest.

1. Identify the model you want to publish the latest version of.
2. Click on Actions dropdown and select Publish Latest to publish the latest version of the model.
3. You will be presented with a confirmation dialog. Select the version you want to publish as latest and click on Publish to publish the latest version of the model.

Once the latest version is published, it is tagged as published and can be used in OpenAI API as flotorch/<model-name> without specifying the version.