Inference Space Private Deployment

This guide is for customer operations teams and platform administrators. The ai-gateway/onprem/ directory in the deployment package is the source of truth for configuration: .env.example defines configurable values, and docker-compose.yml defines the service topology and startup order. Do not overwrite the delivered package with Compose snippets copied from older documentation.

Deployment scope

Compose starts services in this order: postgres healthy → migrate exits successfully → ai-gateway-app healthy → vector; Grafana starts after GreptimeDB is healthy.

Pre-deployment checks

The server must have Docker Engine and Docker Compose v2 installed:

docker --version
docker compose version

Confirm that the server can access the public image registry or that an enterprise-internal registry is ready. In production, use an enterprise domain and an HTTPS reverse proxy; keep the database, Redis, GreptimeDB, Vector, and /api/local/usage-batch inside the internal network.

Public images and architectures

The currently released Inference Space image is:

uhub.service.ucloud.cn/tensorfusion-public/ai-gateway-onprem:0.3.214

The default images provide both linux/amd64 and linux/arm64 manifests:

Offline environments or environments with an internal-only registry must mirror all six image types above, then set ONPREM_IMAGE and the corresponding *_IMAGE values in .env. Do not mirror only the Inference Space image.

docker buildx imagetools inspect \
  uhub.service.ucloud.cn/tensorfusion-public/ai-gateway-onprem:0.3.214

The output should include both linux/amd64 and linux/arm64. Verify internal images the same way.

Prepare the deployment package

Run the following in the private deployment directory of the delivery package or the mono repository:

cd ai-gateway/onprem
cp .env.example .env

.env contains keys and passwords and must not be committed to Git. In production, inject them through an enterprise secrets-management system or controlled deployment variables.

Configure .env

The following values are required:

Example:

openssl rand -base64 48
openssl rand -hex 32
openssl rand -hex 24

POSTGRES_PASSWORD is inserted into the database connection string. Do not use URL-reserved characters such as @, :, /, %, #, ?, or &.

Origin and reverse proxy

BETTER_AUTH_URL and CLIENT_ORIGIN must include the protocol, domain or IP address, and port, and must match the browser address. For example, when users access the service through an HTTPS domain, set both values to https://gateway.example.com. A mismatch causes login requests to return Invalid origin.

Keep ONPREM_TRUST_ANY_ORIGIN set to false by default. Set it to true only for temporary troubleshooting inside the internal network, and immediately restore it to false afterward.

Initial model providers

You can set DASHSCOPE_API_KEY or DEEPSEEK_API_KEY in .env; the corresponding provider is seeded on first startup. Configure other OpenAI- or Anthropic-compatible providers and self-hosted inference services in the admin console before making them available to applications.

The private deployment image does not contain our upstream keys and does not read SaaS operations variables.

Start and validate

docker compose pull
docker compose up -d
docker compose ps
curl -fsS http://localhost:3000/api/health
curl -fsS http://localhost:3030/api/health

The expected state is: PostgreSQL, Redis, GreptimeDB, Inference Space, Vector, and Grafana are healthy, and migrate is exited 0.

If ai-gateway-app does not start, inspect the migration and application logs first:

docker compose logs --tail=200 migrate
docker compose logs --tail=200 ai-gateway-app

First login and application integration

Open BETTER_AUTH_URL and sign in with ONPREM_ADMIN_EMAIL and ONPREM_ADMIN_PASSWORD from .env. The first super administrator is created only when the database is empty; changing .env later does not reset an existing account or password.

After signing in:

1. Add or verify available models in provider settings.
2. Create a separate gk_ Key for each application in API Key management.
3. Maintain moderation and redaction rules by organization in security settings.

Applications call the service with the gateway address, an enabled model ID from the admin console, and a dedicated gk_ Key:

curl "https://gateway.example.com/v1/chat/completions" \
  -H "Authorization: Bearer gk_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-plus",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Use the model IDs enabled in the admin console. Application code should treat the Base URL, model ID, and Key name as configuration rather than hard-coding them.

Default security behavior

The private instance handles PII redaction separately from input and output moderation:

Administrators can view, add, and remove input-moderation keywords in security settings. After a rule is cleared, default values no longer restore it.

Monitoring and security baseline

Grafana is available at http://<host>:3030; sign in with admin and GRAFANA_ADMIN_PASSWORD from .env. The default dashboards include gateway health, usage ingestion, and basic metrics.

• Applications should use different gk_ Keys so that rate limits, audits, and individual revocation are manageable.
• Expose the gateway through an enterprise HTTPS reverse proxy; do not expose the database or internal ingestion ports directly.
• The reverse proxy must reject external access to /api/local/usage-batch; this endpoint is for the local Vector service only.
• Store BETTER_AUTH_SECRET, the database password, administrator password, Grafana password, ingestion token, and provider Keys in the enterprise secrets-management system.

Upgrade, rollback, and backup

Create a PostgreSQL logical backup before upgrading:

docker compose exec -T postgres sh -lc \
  'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
  | gzip > ai-gateway-before-upgrade-$(date +%F-%H%M).sql.gz

Update ONPREM_IMAGE in .env to a verified new version, then run:

docker compose pull ai-gateway-app migrate
docker compose up -d
docker compose ps

The new application version starts only after migrate succeeds. Rolling back an image does not roll back the database schema. For an incompatible upgrade, first validate recovery in an isolated environment, then follow the backup restoration procedure.

Use the same logical-backup command for routine backups. Stop the application and Vector before restoring the backup:

docker compose stop ai-gateway-app vector
gzip -dc ai-gateway-backup-YYYY-MM-DD.sql.gz \
  | docker compose exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" "$POSTGRES_DB"'
docker compose up -d

In production, also back up greptime-data, image-data, and grafana-data as required by your compliance policies. Do not rely on fixed Docker volume names when removing volumes; different project names and deployment environments produce different volume names.

Common issues

Go-live checklist

• docker compose ps shows the application and dependent services as healthy, with migrate at exited 0.
• An administrator can sign in and create a dedicated gk_ Key.
• An application can complete a real /v1/chat/completions request.
• ONPREM_TRUST_ANY_ORIGIN=false, and the browser Origin configuration is correct.
• /api/local/usage-batch is not exposed to the public internet.
• A PostgreSQL backup is complete, and at least one restoration drill has been performed.