Models

Creating a Model

To create a model, you need to click on New Model button in the top right corner of the FloTorch Console. Provide a name and description for the model. Model name should be unique. Once a model is created, you will be navigated to version configuration page, meanwhile your model will appear in Models Registry.

When a model is created, a default version 1 is created automatically and is set in Draft state. You need to configure the model version and publish it to make it available for routing requests through Gateway.

Model Versions

Model versions are the individual configurations of a model. Each model can have multiple versions. Each version can have a different provider models, guardrails, router and cache configuration. Once a model version is published, it is immutable and cannot be changed.

A new version can be created by clicking on New Version button in the top right corner of the FloTorch Console or on Model version configure page by clicking on three dots button beside publish button which will open a dropdown which contains Create new version.

A new copy of a any published or draft version (make a revision) can be created.

To create a copy of published version, click on Make a revision button which is located top right of the page.

To create a copy of draft version, click on three dots button, which will open dropdown, select make a revision option.

Configuring a Model Version

Model versions can be configured using below instructions.

1. Identify the model you want to configure from the Model Registry table.
2. Click on Actions dropdown button and select View Versions.
3. You will be see list of Model versions in slideover table.
4. Again here, click on any table versions table row whcih has version number to open model version configuration page or click on a paritcular row whcih will navigate to latest model version configure page of that particular model.

You will be presented with a model configuration canvas. You can configure the following:

• Input Guardrails - A set of rules or policies to control the input of the model.
• Router - Directs the processed input to the appropriate model based on defined logic. Routing strategy is applied only if more than one provider is configured for a model. Following Routing Strategies are supported:
  • Round Robin - The gateway will route the request to the next provider in the list.
  • Weighted - The gateway will route the request to the provider based on the weight assigned to each provider.
  • Fallback - The gateway will route the request to the fallback provider if the request to primary provider fails.
• Cache - Manages caching mechanisms for common responses. Right now we can use Simple and Semantic cache types.
• Output Guardrails - A set of rules or policies to control the output of the model.

Once the model version is configured, you can publish it to make it available for routing requests through Gateway.

Adding Keyword and Schedule Based Routing.

• Goto to Router configuration on the canvas, click on routing card, it will open slideover with routing strategies and added models.
• Click on settings icon beside model name, the new modal will be opened with addition configuration.
• Here you can enable the Keyword based routing and Schedule based routing.
• Enable / Disable any routing and click Save Configuration.

Publishing a Model Version

Model versions can be published using the Publish button which is located top right corner of the model version canvas. Once you click Publish button a new form will be opened in slideover which contains summary of version and mark as latest checkbox. If you mark the checkbox mark as latest as true, the current publshed version can be marked as latest version to the model.

To publish already created versions which are in draft mode, click on history button which is located before Publish/Make a revision button, it will be opened a slideover with model version list table. there click on publsh button for un published versions, and click on mark as latest button to mark that version as latest version to the model.

Once a model version is published, it is immutable and cannot be changed.

The published model can be used in OpenAI API as flotorch/<model-name>:<version>

Publishing Latest Version of a Model

You can publish the latest version of a model by doing this a model can be used in OpenAI API as flotorch/<model-name> without specifying the version. Internally it is tagged as latest.

1. Identify the model you want to publish the latest version of.
2. Click on Actions dropdown and select Publish Latest to publish the latest version of the model.
3. You will be presented with a slideover with versios in dropdown list. Select the version you want to publish as latest and click on Publish to publish the latest version of the model.

Similarly like above, you can publish latest by clicking on View Versions options on actions menu, the slideover will be opened with model versions list, there you can publish or mark the version as latest.

Once the latest version is published, it is tagged as published and can be used in OpenAI API as flotorch/<model-name> without specifying the version.

Interacting with the model

To interact with the model and test it, the model should be published.

1. Click on code icon in the models table row whcih is alredy published.
2. The code snippet will be opened in slideover, you can copy python, typescript or curl code and test it in your own environment.