122 lines
4.9 KiB
Markdown
122 lines
4.9 KiB
Markdown
# AI Service
|
|
|
|
Technical AI job service for Portal workloads.
|
|
|
|
The first version owns only AI job lifecycle and metrics. Business data stays in
|
|
domain services such as `telephony`, `monitoring-tg` and `monitoring-pf`.
|
|
|
|
## Generic job contract
|
|
|
|
The service is intentionally domain-agnostic:
|
|
|
|
- `owner_service` names the caller, for example `telephony`, `monitoring-tg`,
|
|
`monitoring-pf` or a future Portal module.
|
|
- `owner_ref` is the caller's stable object reference, for example
|
|
`beeline/{call_id}` or `channel/{message_id}`.
|
|
- `task_type` describes the technical task class, for example
|
|
`transcribe`, `call_analysis`, `tg_analysis`, `pf_competitor_analysis`.
|
|
- `model_profile` selects a runtime profile, for example `voxtral-small`,
|
|
`qwen2.5-14b`, `vision`, or a future provider profile.
|
|
- `input` and `result` are JSON payloads owned by the caller and worker.
|
|
|
|
This keeps AI service as shared infrastructure rather than a telephony-specific
|
|
service.
|
|
|
|
## Built-in workers
|
|
|
|
The first built-in worker processes `llm_chat`, `chat_completion` and
|
|
`call_analysis` jobs whose `model_profile` equals `LLM_MODEL`.
|
|
|
|
Input can be either explicit messages:
|
|
|
|
```json
|
|
{
|
|
"messages": [
|
|
{"role": "system", "content": "Answer as JSON."},
|
|
{"role": "user", "content": "Classify this text"}
|
|
],
|
|
"max_tokens": 256
|
|
}
|
|
```
|
|
|
|
or compact `system` / `user` fields. The completed job result contains
|
|
`content`, `model`, `usage` and `duration_ms`.
|
|
|
|
`call_analysis` uses the same input contract as `llm_chat`; callers may include
|
|
domain metadata fields in `input`, but the worker only reads chat fields such as
|
|
`system`, `user`, `messages`, `max_tokens` and `response_format`.
|
|
|
|
`transcription` jobs are processed only by Voxtral Small
|
|
(`mistralai/Voxtral-Small-24B-2507`) through an OpenAI-compatible
|
|
`/v1/audio/transcriptions` endpoint. The returned `segments` field stays
|
|
compatible with telephony. If the provider returns one long segment, AI Service
|
|
splits it into smaller transcript segments and adds heuristic speaker labels
|
|
when diarization is requested.
|
|
|
|
AI-server compose snippet for Voxtral lives in
|
|
`deploy/ai-server/docker-compose.audio.yml`:
|
|
|
|
- Voxtral endpoint: `http://10.2.3.5:8004`
|
|
- Start Voxtral:
|
|
`docker compose -f docker-compose.yml -f docker-compose.audio.yml --profile voxtral-small up -d voxtral-small`
|
|
|
|
In Kubernetes the dedicated transcription worker may claim more than one
|
|
`voxtral-small` job at a time. This keeps download/upload/wait overhead from
|
|
serializing the queue while Voxtral/vLLM still controls the actual GPU
|
|
scheduling.
|
|
|
|
## API
|
|
|
|
- `POST /api/v1/jobs` creates one job.
|
|
- `GET /api/v1/jobs` lists jobs with query filters.
|
|
- `POST /api/v1/jobs/batch` creates many jobs with shared defaults.
|
|
- `POST /api/v1/jobs/retry` retries failed/running jobs by filter.
|
|
- `POST /api/v1/jobs/cancel` cancels pending/running jobs by filter.
|
|
- `POST /api/v1/jobs/claim` atomically claims pending jobs for a worker.
|
|
- `GET /api/v1/jobs/{id}` returns technical job state and result.
|
|
- `POST /api/v1/jobs/{id}/complete` stores a successful job result.
|
|
- `POST /api/v1/jobs/{id}/fail` stores a failed job category and message.
|
|
- `POST /api/v1/jobs/{id}/retry` resets failed/running jobs to `pending`.
|
|
- `GET /api/v1/stats` returns queue and error counters.
|
|
- `GET /api/v1/providers/status` checks configured AI providers without
|
|
returning secrets.
|
|
- `GET /api/v1/infra/status` returns AI-server sidecar telemetry
|
|
(GPU, containers and vLLM live metrics) when configured.
|
|
- `GET /healthz` returns process health.
|
|
- `GET /readyz` checks PostgreSQL readiness.
|
|
- Built-in workers expose open Kubernetes endpoints on `WORKER_HTTP_PORT`:
|
|
`GET /healthz`, `GET /readyz` and `GET /worker/status`.
|
|
|
|
All `/api/v1/*` endpoints require `Authorization: Bearer <AI_SERVICE_TOKEN>`
|
|
when `AI_SERVICE_TOKEN` is configured. Health and readiness endpoints stay open
|
|
for Kubernetes probes.
|
|
|
|
## Configuration
|
|
|
|
- `HTTP_HOST`, default `0.0.0.0`
|
|
- `HTTP_PORT`, default `8080`
|
|
- `DATABASE_URL`, required
|
|
- `MIGRATE_ON_START`, default `true`
|
|
- `AI_SERVICE_TOKEN`, optional bearer token for service-to-service API calls
|
|
- `LLM_BASE_URL`, primary OpenAI-compatible LLM endpoint
|
|
- `LLM_API_KEY`, primary LLM API key
|
|
- `LLM_MODEL`, default `qwen2.5-14b`
|
|
- `LLM_TIMEOUT`, default `5m`
|
|
- `VOXTRAL_BASE_URL`, OpenAI-compatible endpoint for Voxtral
|
|
- `VOXTRAL_MODEL`, default `mistralai/Voxtral-Small-24B-2507`
|
|
- `VOXTRAL_API_KEY`, optional bearer token for Voxtral; falls back to
|
|
`AUDIO_LLM_API_KEY`, then `LLM_API_KEY`
|
|
- `AUDIO_LLM_PROMPT`, transcription instruction for Voxtral
|
|
- `WORKER_ID`, default hostname
|
|
- `WORKER_HTTP_HOST`, default `0.0.0.0`
|
|
- `WORKER_HTTP_PORT`, default `8081`
|
|
- `WORKER_POLL_INTERVAL`, default `2s`
|
|
- `WORKER_CLAIM_LIMIT`, default `4`
|
|
- `WORKER_LEASE_TIMEOUT`, default `15m`
|
|
|
|
## Next integration step
|
|
|
|
`telephony` should first mirror low-risk analysis jobs into this service while
|
|
continuing local processing. Remote execution can then be enabled by feature
|
|
flag per task type.
|