Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 106 additions & 19 deletions skills/a7-plugin-ai-proxy/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ name: a7-plugin-ai-proxy
description: >-
Skill for configuring the API7 Enterprise Edition ai-proxy plugin via the a7 CLI.
Covers proxying requests to LLM providers (OpenAI, Azure OpenAI, DeepSeek,
Anthropic, Gemini, Vertex AI, and more), authentication per provider,
model configuration, streaming, logging, and route/service usage.
Anthropic, Gemini, Vertex AI, Amazon Bedrock, and more), authentication per
provider, model configuration, streaming, logging, and route/service usage.
version: "1.0.0"
author: API7.ai Contributors
license: Apache-2.0
Expand All @@ -23,51 +23,70 @@ metadata:

## Overview

The `ai-proxy` plugin turns API7 Enterprise Edition (API7 EE) into an AI gateway. It proxies requests in
OpenAI-compatible format to LLM providers, handling authentication, endpoint
routing, and response streaming. Clients send a standard chat-completion
request; the plugin translates and forwards it to the configured provider.
The `ai-proxy` plugin turns API7 Enterprise Edition (API7 EE) into an AI
gateway. Clients can send requests in supported protocols to API7 EE instead
of handling provider authentication and endpoint selection themselves. The
plugin detects the client protocol, selects a compatible provider endpoint,
forwards the native format or converts it when an adapter is available, and
handles response streaming.

## When to Use

- Proxy chat-completion or embedding requests to any supported LLM provider
- Proxy Chat Completions, Responses API, Embeddings, Anthropic Messages, or
Bedrock Converse requests to a compatible provider
- Centralize API keys at the gateway instead of distributing to clients
- Add observability (token counts, latency) to LLM calls
- Combine with `ai-prompt-template`, `ai-prompt-decorator`, or content
moderation plugins for a full AI gateway pipeline
- Apply consistent AI proxy configurations directly on services or routes

## Protocol Detection

API7 Gateway uses the request URI as part of protocol detection. Anthropic
Messages requests must use a URI ending in `/v1/messages`, and Bedrock Converse
requests must use a URI ending in `/converse`. Without these suffixes, a request
body can match another protocol, such as OpenAI Chat.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

OpenAI Responses requests with an `input` field must use a URI ending in
`/v1/responses`. Otherwise, API7 Gateway detects the body as OpenAI Embeddings;
use a URI ending in `/v1/embeddings` for embedding routes.

For Bedrock streaming, keep the client-facing URI ending in `/converse` and set
`stream: true` in the request body. API7 Gateway then selects the upstream
`/model/{modelId}/converse-stream` endpoint.

## Supported Providers

| Provider | Value | Default Endpoint |
|----------|-------|------------------|
| OpenAI | `openai` | `https://api.openai.com/v1/chat/completions` |
| Provider | Value | Endpoint Behavior |
|----------|-------|-------------------|
| OpenAI | `openai` | Automatically selects `/v1/chat/completions`, `/v1/responses`, or `/v1/embeddings` on `https://api.openai.com` |
| DeepSeek | `deepseek` | `https://api.deepseek.com/chat/completions` |
| Azure OpenAI | `azure-openai` | Custom via `override.endpoint` |
| Anthropic | `anthropic` | `https://api.anthropic.com/v1/chat/completions` |
| Anthropic | `anthropic` | Automatically selects `/v1/chat/completions` or `/v1/messages` on `https://api.anthropic.com` |
| AIMLAPI | `aimlapi` | `https://api.aimlapi.com/v1/chat/completions` |
| OpenRouter | `openrouter` | `https://openrouter.ai/api/v1/chat/completions` |
| Gemini | `gemini` | `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` |
| Vertex AI | `vertex-ai` | `https://aiplatform.googleapis.com` |
| Amazon Bedrock | `bedrock` | Region- and model-specific Bedrock Runtime endpoint; available from API7 Enterprise 3.9.11 |
| OpenAI-Compatible | `openai-compatible` | Custom via `override.endpoint` |

## Plugin Configuration Reference

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `provider` | string | **Yes** | — | One of the 9 supported providers |
| `provider` | string | **Yes** | — | One of the 10 supported providers |
| `auth` | object | **Yes** | — | Authentication config (see below) |
| `options` | object | No | — | Model and generation parameters |
| `options.model` | string | No | — | Model name (provider-specific) |
| `options.temperature` | number | No | — | Sampling temperature |
| `options.top_p` | number | No | — | Nucleus sampling |
| `options.max_tokens` | integer | No | — | Maximum tokens to generate |
| `options.stream` | boolean | No | `false` | Enable SSE streaming |
| `override` | object | No | — | Override default endpoint |
| `override.endpoint` | string | No | — | Full URL for the provider API |
| `provider_conf` | object | No | — | Provider-specific config (Vertex AI) |
| `provider_conf.project_id` | string | No | — | GCP project ID (Vertex AI) |
| `provider_conf.region` | string | No | — | GCP region (Vertex AI) |
| `options.stream` | boolean | No | — | Override the outgoing `stream` field. For Bedrock Converse, `stream: true` on a `/converse` request selects `/model/{modelId}/converse-stream` and returns unmodified AWS EventStream binary frames with `Content-Type: application/vnd.amazon.eventstream`, not SSE; clients must parse EventStream responses. |
| `override` | object | No | — | Provider endpoint and request-body override settings |
| `override.endpoint` | string | No | — | Provider scheme and host, or a full URL including the path and query |
| `provider_conf` | object | No | — | Provider-specific config for Vertex AI or Amazon Bedrock |
| `provider_conf.project_id` | string | No | — | GCP project ID for Vertex AI; required with `region` unless `override.endpoint` is configured |
| `provider_conf.region` | string | No | — | GCP region for Vertex AI; required AWS region for Amazon Bedrock |
| `logging` | object | No | — | Logging options |
| `logging.summaries` | boolean | No | `false` | Log model, duration, tokens |
| `logging.payloads` | boolean | No | `false` | Log request/response bodies |
Expand All @@ -79,7 +98,7 @@ request; the plugin translates and forwards it to the configured provider.

## Authentication by Provider

### OpenAI / DeepSeek / Anthropic / AIMLAPI / OpenRouter
### OpenAI / DeepSeek / AIMLAPI / OpenRouter

```json
{
Expand All @@ -91,6 +110,22 @@ request; the plugin translates and forwards it to the configured provider.
}
```

### Anthropic

```json
{
"auth": {
"header": {
"x-api-key": "your-anthropic-api-key",
"anthropic-version": "2023-06-01"
}
}
}
```

Native Anthropic Messages requests require an `anthropic-version` header.
Configure it in `auth.header`, as shown, or require clients to send it.

### Azure OpenAI

```json
Expand All @@ -106,6 +141,18 @@ request; the plugin translates and forwards it to the configured provider.
}
```

### Gemini

```json
{
"auth": {
"header": {
"Authorization": "Bearer your-gemini-key"
}
}
}
```

### Vertex AI (GCP Service Account)

```json
Expand All @@ -124,6 +171,46 @@ request; the plugin translates and forwards it to the configured provider.
}
```

The `service_account_json` can also be set through the
`GCP_SERVICE_ACCOUNT` environment variable.

### Amazon Bedrock

```json
{
"auth": {
"aws": {
"access_key_id": "your-access-key-id",
"secret_access_key": "your-secret-access-key",
"session_token": "your-session-token"
}
},
"provider_conf": {
"region": "us-east-1"
},
"options": {
"model": "your-model-id"
}
}
```

The session token is required when you use temporary AWS credentials.

### Custom OpenAI-Compatible API

```json
{
"auth": {
"header": {
"Authorization": "Bearer your-token"
}
},
"override": {
"endpoint": "https://your-custom-llm.com/v1/chat/completions"
}
}
```

## Step-by-Step: Route to OpenAI

### 1. Create a route with ai-proxy
Expand Down
Loading