Background
I have metrics in prometheus that I want to use to scale a hpa using external metrics. I deployed an APIService which lets me get the result of my PromQL as a value which the hpa should hopefully understand. But my issue is that the hpa is failing to find my metric. It comes out as unknown in the hpa.
Prometheus Adapter Config
I am using an external metric, my data in prometheus has a namespace and a fleet label. The metric is currently saved as lukas_test.
apiVersion: v1
data:
config.yaml: |
externalRules:
- seriesQuery: 'game_server_busy{namespace!~"", fleet!~""}'
resources:
overrides:
namespace:
resource: "namespace"
name:
as: 'lukas_test'
metricsQuery: 'sum(game_server_busy) by (fleet)'
APIService for external metrics
To my understanding this is what allows the hpa to get my metric using the External type.
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.external.metrics.k8s.io
spec:
service:
name: prometheus-adapter
namespace: default
port: 443
group: external.metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
HorizontalPodAutoscaler
This hpa is targeting an agones fleet.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: prometheus-autoscaler
namespace: default
spec:
scaleTargetRef:
apiVersion: agones.dev/v1
kind: Fleet
name: lukas-fleet-homestead
minReplicas: 1
maxReplicas: 3
metrics:
- type: External
external:
metric:
name: lukas_test
selector:
matchLabels:
fleet: "lukas-homestead-fleet"
target:
type: Value
value: 1
How it's currently (not) working
To try and pinpoint what's wrong I have tried to test each step of the pipeline.
- I have checked in the prometheus web ui that there is data with the correct labels.
- With my APIService I am able to get the current value like this
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/lukas_test" | jq
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metricName": "lukas_test",
"metricLabels": {
"fleet": "lukas-homestead-fleet"
},
"timestamp": "2025-03-21T12:19:26Z",
"value": "0"
}
]
}
To me this seems like I should be able to select an External metric with the name lukas_test and match it using the label fleet: lukas-homestead-fleet. This should give a value of 0. However, my hpa has the following status when I describe it:
Name: prometheus-autoscaler
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Fri, 21 Mar 2025 12:57:25 +0100
Reference: Fleet/lukas-fleet-homestead
Metrics: ( current / target )
"lukas_test" (target value): <unknown> / 1 Min replicas: 1 Max replicas: 3 Fleet pods: 1 current / 0 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False InvalidSelector the HPA target's scale is missing a selector Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedComputeMetricsReplicas 36m (x12 over 39m) horizontal-pod-autoscaler selector is required Warning SelectorRequired 4m26s (x141 over 39m) horizontal-pod-autoscaler selector is required
It seems like the hpa is finding Fleet/lukas-fleet-homestead, as it can detect that there is currently 1 pod. But the current is marked as <unknown> and I am getting InvalidSelector.
I have tried switching form External to Object and Pods, which required binding my metric to a resource, but I was getting the same error message. It almost feels like kubernetes isn't using /apis/external.metrics.k8s.io/v1beta1/ to get my metric. Or like it's related to insufficient rights? I changed the rights for system:controller:horizontal-pod-autoscaler to allow listing, getting and watching * resources in * apiGroups. But that didn't change anything. I am probably missing a key part in this, but I have no idea.
All advice is welcome! Have a nice day!