Add opt-in Prometheus metrics and ServiceMonitor

[Sara-Agent]

Hello 👋

I’m Sara, Ivan Sushkov’s personal operator. I help Ivan research, implement, and verify engineering work, while keeping the evidence and the limits visible.

@GC-Elia and @hweissta, hello! Prometheus metrics support for Altinity MCP is ready for your review. Thank you for taking a look. 🙂

What this PR adds

• An opt-in Prometheus /metrics endpoint for HTTP and SSE transports, disabled by default.
• Bounded-label metrics for MCP HTTP requests, ClickHouse query count and latency, returned rows, blocked-clause rejections, ClickHouse readiness, and catalog-cache activity.
• Configuration through YAML, environment, and CLI.
• Helm values and service-port wiring, plus an optional Prometheus Operator ServiceMonitor.
• Tests and operator documentation for the endpoint, middleware, collectors, configuration, and Helm rendering.

The design avoids unbounded labels such as raw queries, paths, or error messages.

EC2 end-to-end verification

I ran the final clean test on a temporary private t4g.small ARM64 EC2 instance in AWS eu-north-1, using Amazon Linux 2023. The instance ran:

• Altinity MCP from this PR, built with Go 1.26.8.
• ClickHouse Server 26.3 in Docker.
• Prometheus 3.5.0 in Docker.

The final run completed with exit code 0 in 6 minutes 27 seconds. The instance was terminated after the report was collected.

What was tested

• The complete short Go test suite passed across all packages.
• Focused metrics, server, and command tests passed.
• MCP started with metrics enabled; /livez, /health, and /metrics responded correctly.
• A real MCP SDK client listed tools and executed SELECT 42 against ClickHouse.
• The client also executed an intentionally failing query; success and error counters both increased with the expected bounded labels.
• With metrics disabled, /metrics was not exposed. The integrated server returned HTTP 405 because the MCP catch-all handled the unregistered path; it did not return metrics.
• Prometheus scraped Altinity MCP successfully and returned up=1 for the target.
• ClickHouse was stopped deliberately: MCP health changed to HTTP 503, altinity_mcp_clickhouse_up changed to 0, and the MCP process remained alive.
• ClickHouse was restarted: MCP recovered to HTTP 200 and altinity_mcp_clickhouse_up returned to 1 without restarting MCP.
• Prometheus successfully queried the recovered metric.

Result

The measured end-to-end path works: Altinity MCP exports the metrics, Prometheus scrapes them, query outcomes are counted, and ClickHouse failure and recovery are visible without restarting MCP. No functional blocker was found in the tested path. ✅

[BorisTyshkevich]

• [P1] Metrics are not exposed in multicluster mode. The multicluster server builds its own mux but never calls registerMetricsRoute or wraps it with metrics.HTTPMiddleware. Consequently, metrics.enabled=true
  still returns no /metrics, and the newly registered catalog-cache collector is unreachable. See startMulticlusterHTTPServer (

  altinity-mcp/cmd/altinity-mcp/main.go

  Lines 1445 to 1473 in 9749c7c

  	func (a *application) startMulticlusterHTTPServer(cfg config.Config) error {
  	addr := fmt.Sprintf("%s:%d", cfg.Server.Address, cfg.Server.Port)
  	log.Info().
  	Str("address", addr).
  	Msg("Starting MCP server with multi-cluster HTTP transport")
  	authInjector := a.createMCPAuthInjector(cfg)
  	serverInjector := func(next http.Handler) http.Handler {
  	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  	ctx := context.WithValue(r.Context(), altinitymcp.CHJWEServerKey, a.mcpServer)
  	next.ServeHTTP(w, r.WithContext(ctx))
  	})
  	}
  	factory := altinitymcp.NewMulticlusterServerFactory(cfg, a.mcpServer, a.mcCache, version)
  	sdkHandler := mcp.NewStreamableHTTPHandler(factory.GetServer, statelessStreamableOptions())
  	mux := http.NewServeMux()
  	mux.HandleFunc("/health", a.healthHandler)
  	mux.HandleFunc("/livez", a.livenessHandler)
  	a.registerOAuthHTTPRoutes(mux)
  	a.registerMulticlusterPRMRoutes(mux)
  	mcpHandler := a.mcRouter.Middleware(authInjector(serverInjector(sdkHandler)))
  	mux.Handle("/mcp/{cluster}", mcpHandler)
  	mux.Handle("/mcp/{cluster}/", mcpHandler)
  	httpHandler := stripTrailingSlash(corsMiddleware(cfg.Server.CORSOrigin, mux))

  ).

• [P1] altinity_mcp_clickhouse_up falsely reports 0 with OAuth or JWE. The gauge starts at zero, while the health handler intentionally skips ClickHouse pings whenever credentials are per-request. Therefore
  healthy OAuth/JWE deployments continuously appear down. This also affects every multicluster deployment because multicluster requires OAuth. See the gauge initialization (

  altinity-mcp/pkg/metrics/metrics.go

  Lines 90 to 95 in 9749c7c

  	ClickHouseUp = promauto.NewGauge(
  	prometheus.GaugeOpts{
  	Name: "altinity_mcp_clickhouse_up",
  	Help: "Whether the most recent ClickHouse readiness ping succeeded (1) or failed (0).",
  	},
  	)

  ) and skipped readiness check (

  altinity-mcp/cmd/altinity-mcp/main.go

  Lines 719 to 755 in 9749c7c

  	// Test ClickHouse connection for readiness, unless credentials are per-request
  	credentialsArePerRequest := cfg.Server.JWE.Enabled ||
  	cfg.Server.OAuth.Enabled
  	if !credentialsArePerRequest {
  	chClient, err := clickhouse.NewClient(ctx, cfg.ClickHouse)
  	if err != nil {
  	metrics.ObserveClickHouseHealth(err)
  	log.Error().Err(err).Msg("Health check: failed to create ClickHouse client")
  	status["status"] = "unhealthy"
  	status["error"] = "ClickHouse connection failed"
  	w.Header().Set("Content-Type", "application/json")
  	w.WriteHeader(http.StatusServiceUnavailable)
  	_ = json.NewEncoder(w).Encode(status)
  	return
  	}
  	defer func() {
  	if closeErr := chClient.Close(); closeErr != nil {
  	log.Warn().Err(closeErr).Msg("Health check: failed to close ClickHouse client")
  	}
  	}()
  	if err := chClient.Ping(ctx); err != nil {
  	metrics.ObserveClickHouseHealth(err)
  	log.Error().Err(err).Msg("Health check: ClickHouse ping failed")
  	status["status"] = "unhealthy"
  	status["error"] = "ClickHouse connection failed"
  	w.Header().Set("Content-Type", "application/json")
  	w.WriteHeader(http.StatusServiceUnavailable)
  	_ = json.NewEncoder(w).Encode(status)
  	return
  	}
  	metrics.ObserveClickHouseHealth(nil)
  	status["clickhouse"] = "connected"
  	} else {
  	status["auth"] = "per_request_credentials"
  	}

  ). The metric should be omitted/marked unknown in these modes, or updated from authenticated connection attempts.

• [P2] Configuration reload cannot enable or disable /metrics. Route registration captures the startup configuration and the reload path only replaces a.config and the MCP server. Changing server.metrics.enabled therefore has no effect until restart; notably, setting it to false leaves the unauthenticated endpoint exposed. Either update routing dynamically or document/warn that this field is restart-only. See registerMetricsRoute (

  altinity-mcp/cmd/altinity-mcp/main.go

  Lines 203 to 213 in 9749c7c

  	// registerMetricsRoute adds the Prometheus scrape endpoint alongside
  	// /health and /livez when the operator opted in. Unconditional registration
  	// would expose query-shape and timing data (via the ClickHouse histograms)
  	// to anyone who can reach the port, which is why every other optional
  	// surface here (OpenAPI, OAuth) is also config-gated rather than always on.
  	func registerMetricsRoute(mux *http.ServeMux, cfg config.Config) {
  	if !cfg.Server.Metrics.Enabled {
  	return
  	}
  	mux.Handle("/metrics", promhttp.Handler())
  	}

  ) and reloadConfig (

  altinity-mcp/cmd/altinity-mcp/main.go

  Lines 1322 to 1377 in 9749c7c

  	// reloadConfig reloads configuration from file and updates the application
  	func (a *application) reloadConfig(cmd CommandInterface) error {
  	log.Debug().Str("config_file", a.configFile).Msg("Reloading configuration")
  	// Load new config from file
  	newCfg, err := config.LoadConfigFromFile(a.configFile)
  	if err != nil {
  	return fmt.Errorf("failed to load config file: %w", err)
  	}
  	for _, w := range newCfg.RemovedKeyWarnings {
  	log.Warn().Str("config_file", a.configFile).Msg(w)
  	}
  	// Override with CLI flags
  	overrideWithCLIFlags(newCfg, cmd)
  	if err := newCfg.ClickHouse.ValidateConnectHost(); err != nil {
  	return err
  	}
  	// multicluster.* fields are restart-only: the router + catalog cache
  	// were bound during newApplication and cannot be safely rebuilt
  	// mid-flight without dropping in-flight requests. Warn loudly on
  	// any change so operators see the no-op and know to roll the pod.
  	a.configMutex.RLock()
  	oldMC := a.config.Multicluster
  	a.configMutex.RUnlock()
  	if !reflect.DeepEqual(oldMC, newCfg.Multicluster) {
  	log.Warn().Msg("config reload: multicluster.* fields changed — restart required for these to take effect; routing/cache remain on the previous configuration")
  	}
  	// Update logging level if changed
  	a.configMutex.Lock()
  	oldLogLevel := a.config.Logging.Level
  	a.config = *newCfg
  	a.configMutex.Unlock()
  	if oldLogLevel != newCfg.Logging.Level {
  	if err := setupLogging(string(newCfg.Logging.Level)); err != nil {
  	log.Error().Err(err).Msg("Failed to update logging level")
  	} else {
  	log.Info().
  	Str("old_level", string(oldLogLevel)).
  	Str("new_level", string(newCfg.Logging.Level)).
  	Msg("Logging level updated")
  	}
  	}
  	// Create new MCP server with updated config
  	newMCPServer := altinitymcp.NewClickHouseMCPServer(*newCfg, version)
  	// Update the server (note: this doesn't restart HTTP servers, only updates the MCP server)
  	a.configMutex.Lock()
  	a.mcpServer = newMCPServer
  	a.configMutex.Unlock()
  	log.Info().Str("config_file", a.configFile).Msg("Configuration reloaded successfully")

  ).

[BorisTyshkevich]

Additional review findings after checking the current head (080f5f2): I don't think this is ready to merge yet.

• [P1] metrics.enabled=false is not actually disabled. HTTP middleware is installed unconditionally, ClickHouse query instrumentation runs unconditionally, successful queries are rescanned to compute metric bytes, and the Prometheus collectors are registered globally via promauto. Disabled metrics should be a true no-op: no middleware, no per-request/query instrumentation, no metric-state mutation, and ideally no collectors registered.
• [P1] method is an unbounded Prometheus label. HTTPRequestsTotal.WithLabelValues(pattern, r.Method, ...) uses a client-controlled arbitrary HTTP method directly. An attacker can create unbounded series. Normalize to a fixed set such as GET, POST, OPTIONS, other.
• [P1] Plain SSE does not expose/collect metrics. In startSSEServer, the JWE branch calls registerMetricsRoute and wraps the mux with metrics.HTTPMiddleware, but the non-JWE branch does neither. This contradicts the PR's stated HTTP + SSE support and needs an actual server-level regression test.
• [P1/P2] Metrics config reload has inconsistent effective state. reloadConfig accepts and stores a changed server.metrics.enabled, but the HTTP mux is not rebuilt. Enabled -> disabled leaves /metrics exposed; disabled -> enabled leaves it absent. Either apply the change dynamically, or make the field genuinely restart-only by retaining the old effective value and warning. Merely documenting the mismatch is not sufficient.
• [P2] ServiceMonitor is broken with server TLS. The ServiceMonitor omits scheme, so it scrapes HTTP, while MCP switches the same port to HTTPS when config.server.tls.enabled=true. Either support HTTPS/TLS config in the ServiceMonitor or reject/document that configuration combination explicitly.
• The latest GitHub Actions run for this head is action_required with no jobs, so the final SHA still needs a real green go vet / go test CI run.

Also worth fixing while touching the instrumentation: executeSelect already computes approximate returned bytes for result caps, but the metrics wrapper walks every result row again and recomputes approxRowBytes. If that is the same quantity, reuse the existing value instead of adding another O(result-size) pass.