Follow us
Tech News

Common mistakes with Kubernetes HPA custom metrics in 2026

Learn how to avoid common configuration errors with the Prometheus-adapter and Kubernetes HPA. This guide covers TLS trust issues, ETCD size limits for ConfigMaps, and why KEDA is a superior alternative for scaling to zero.

Share

Adapter installation and connectivity

I use the prometheus-community Helm chart to install the Prometheus Adapter. I configure the prometheus.url and prometheus.port in my values.yaml file to point to my Prometheus service. I find the Helm installation through the prometheus-community repository most efficient. If I use a stable URL via an Ingress, I must configure the adapter to reach that endpoint. I must also provide the host’s CA Bundle using extraVolumes and extraVolumeMounts if Prometheus uses HTTPS. I notice that the APIService shows False availability if I have TLS trust issues. I use cert-manager to handle this. The Helm chart manages the creation of a self-signed CA and the injection of the CA into the caBundle. I rely on these settings to ensure the Kubernetes API server trusts the adapter. I can enable different rules by setting rules.default, rules.custom, or rules.external. If I set rules.resource, the chart deploys a resource metrics API service. This service provides the same functionality as Metrics Server, so I cannot deploy both in the same cluster. If rbac.create is true, the chart creates RBAC resources. If I use a restricted cluster, I update the serviceAccount.name.

Connectivity fails.

Rule configuration and mapping errors

The adapter uses four parts for every rule. Discovery uses seriesQuery to find Prometheus metrics and seriesFilters to filter them. Association uses template or overrides to link metrics to resources. Naming uses the name field and a matches regex to convert names. Querying uses the metricsQuery field to turn API requests into Prometheus queries. This query uses Series, LabelMatchers, and GroupBy fields.

Rule Part Function
Discovery Finds Prometheus metrics
Association Links metrics to resources
Naming Converts metric names
Querying Fetches metric values

I must ensure my metrics include namespace and pod labels. If a user defines a rule where the metric name does not match the name in the HPA exactly, the Horizontal Pod Autoscaler will fail to fetch the required custom metrics. I also avoid using a single ConfigMap for all my rules. The Prometheus Adapter relies on a single ConfigMap for configuration. This ConfigMap can hit the 1MB size limit of ETCD. This single file makes it hard to isolate work for different teams. The configuration is cryptic and complicated. I use seriesQuery to find a set of Prometheus series, and the adapter strips the label values from this series. I use template to specify how a label name that matches a pattern refers to a group-resource. I use overrides to map a Prometheus label to a Kubernetes group-resource. I use the as field to convert a name. If there is no capture group, it defaults to $0. If there is one capture group, it defaults to $1.

Configuration remains difficult.

Scaling logic and KEDA alternatives

I suggest KEDA for many workloads. KEDA is an alternative to the Prometheus Adapter. KEDA supports over 60 predefined event sources. KEDA allows scale to zero. The Prometheus Adapter cannot scale to zero. I use KEDA because it uses CRDs like ScaledObject and ScaledJob. I use the PrometheusScaler to pull metrics. I set an activationThreshold in my KEDA configuration.

You should check if your metrics are spread over multiple Prometheus instances. The Prometheus Adapter connects to only one instance at a time. I find the Prometheus Adapter is the old timer. The HPA controller runs every 15 seconds by default. I check the http_requests_per_second metric to verify my scaling. I also look at http_server_requests_second to avoid 404 errors. I remember a user tried to use count(DCGM_FI_DEV_GPU_UTIL == 0) for GPU idle modes but struggled with the metricsQuery syntax. The HPA controller works by calculating the ratio between the desired metric value and the current metric value. If the current value is 200m and the desired value is 100m, the controller doubles the number of replicas. If the current value is 50m, it halves the replicas. The control plane skips scaling if the ratio is close to 1.0.

KEDA works well. Can the Prometheus Adapter connect to multiple instances?

Share

Technewsdaily

Senior tech writer covering AI, gadgets and cybersecurity. Breaking down the news that matters, every day.