Getting Started

This guide walks through running model2vec-serve locally, calling its endpoints, and validating the container and Helm chart.

Prerequisites

• Rust toolchain (stable, 1.85+)
• A model2vec model identifier (e.g. minishlab/potion-multilingual-128M) or a local model directory
• Docker (for container validation)
• Helm 3+ and a Kubernetes cluster or local environment such as kind/minikube (for Helm validation)

Run locally

Start the server with a Hugging Face model id:

cargo run --release -- --model minishlab/potion-multilingual-128M --port 8080

You can also set values via environment variables:

MODEL=minishlab/potion-multilingual-128M PORT=8080 cargo run --release

Serve multiple models in one process:

cargo run --release -- \
  --model minishlab/potion-multilingual-128M \
  --model minishlab/potion-code-16M-v2 \
  --default-model minishlab/potion-multilingual-128M \
  --port 8080

Verify health

curl http://localhost:8080/health

Expected response:

{
  "status": "healthy",
  "ready": true,
  "message": "1 model(s) ready",
  "models": [
    {
      "model_id": "minishlab/potion-multilingual-128M",
      "status": "ready",
      "message": "model loaded"
    }
  ]
}

Request embeddings

OpenAI-compatible endpoint

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"Hello world"}'

With multiple models loaded, select a model explicitly:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"def hello(): pass","model":"minishlab/potion-code-16M-v2"}'

Expected response shape:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0123, -0.0456, "..."]
    }
  ],
  "model": "minishlab/potion-multilingual-128M",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

TEI-compatible endpoint

curl -X POST http://localhost:8080/embed \
  -H "Content-Type: application/json" \
  -d '{"inputs":"Hello world"}'

Expected response:

[[0.0123, -0.0456, "..."]]

Inspect metrics

curl http://localhost:8080/metrics

The response is Prometheus-compatible text with request counters and latency histograms.

Enable API key authentication

Start the service with a key:

cargo run --release -- --model minishlab/potion-multilingual-128M --api-key secret-key

An unauthenticated request is rejected:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"Hello"}'

Expected: 401 Unauthorized.

A request with the Bearer token succeeds:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer secret-key" \
  -d '{"input":"Hello"}'

Run in Docker

Build the image:

docker build -t model2vec-serve:latest .

Run it:

docker run -p 8080:8080 \
  -e MODEL=minishlab/potion-multilingual-128M \
  model2vec-serve:latest

Serve multiple models:

docker run -p 8080:8080 \
  -e MODEL=minishlab/potion-multilingual-128M,minishlab/potion-code-16M-v2 \
  -e DEFAULT_MODEL=minishlab/potion-multilingual-128M \
  model2vec-serve:latest

Run the same curl checks as in the local section.

Deploy with Helm

Install the chart:

helm install model2vec-serve ./helm/model2vec-serve \
  --set model=minishlab/potion-multilingual-128M \
  --set apiKey=secret-key \
  --set replicaCount=2

Install with multiple models:

helm install model2vec-serve ./helm/model2vec-serve \
  --set models={minishlab/potion-multilingual-128M,minishlab/potion-code-16M-v2} \
  --set defaultModel=minishlab/potion-multilingual-128M \
  --set apiKey=secret-key \
  --set replicaCount=2

Wait for pods:

kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=model2vec-serve

Port-forward and test:

kubectl port-forward svc/model2vec-serve 8080:80
curl http://localhost:8080/health

See the Helm page for volume-mounted models and all configuration options.