How service endpoint and load balancer monitoring works
The Kubernetes Service Endpoints and Load Balancer monitors: states, overrides and troubleshooting
Applies to: Opslogix Kubernetes Management Pack (V26.11.4546.0 and later)
Last updated: November 2026 · Latest version: 26.11.4546.0
See also: Upgrade guide for 26.11.4546.0 · List of rules and monitors · Release notes
A Kubernetes service is the address clients use to reach an application. When none of the pods behind a service is ready, the clients of the service get connection errors, even if no node or pod alert points to the cause. From V26.11.4546.0 the Opslogix Kubernetes Management Pack monitors the endpoints of every service, and the external address of services of type LoadBalancer.
Kubernetes Service Endpoints monitor
The endpoints of a service are the pods its selector matches. Kubernetes only routes traffic to endpoints that are ready. The monitor reads the EndpointSlices of each service every 300 seconds:
| Endpoint state | Monitor state | Meaning |
|---|---|---|
| Ready | Healthy | All endpoints are ready. Terminating endpoints, for example during a rolling update, are not counted as not ready. |
| PartiallyReady | Warning, after 3 consecutive samples | Some endpoints are not ready. The 3 samples (about 15 minutes) keep a rolling update from raising an alert. |
| NoReadyEndpoints | Critical, after 2 consecutive samples | No endpoint is ready, or the selector matches no pods, for example after the labels of a deployment were changed. |
| ScaledToZero | Healthy | The selector matches no pods because the deployments or stateful sets it selects are scaled to 0 replicas on purpose, by hand or by an autoscaler. |
| NotApplicable | Healthy | The service has no selector, so Kubernetes does not manage its endpoints, or it is an ExternalName service. |
| Unknown | Warning | The endpoint slices could not be read, for example because of a missing permission. |
The alert names the pods the selector matches and their state, for example "No ready endpoints (1 terminating). The selector app=web matches 2 pods, 0 ready: web-7d4b9 (Pending, not ready), web-5c8f2 (Running, terminating)." A monitor that is Warning or Critical returns to healthy after 2 healthy samples.
Notes:
- Workloads scaled to 0 are only recognized for deployments and stateful sets. A workload that scales its pods in another way is reported as NoReadyEndpoints when it has no pods.
- A service with publishNotReadyAddresses set reports all its endpoints as ready, so it is only reported when it has no endpoints at all.
Kubernetes Service Load Balancer monitor
A service of type LoadBalancer needs a load balancer controller, such as the cloud provider integration or MetalLB, to get an external address. Services of other types are always healthy.
| Load balancer state | Monitor state | Meaning |
|---|---|---|
| Assigned | Healthy, after 1 sample | The load balancer has an external IP address or host name. |
| Pending | Warning, after 3 consecutive samples | The load balancer has no external address yet. A cluster without a load balancer controller never assigns one. |
| Failed | Critical, after 2 consecutive samples | No external address, and a load balancer controller reported a failure: a SyncLoadBalancerFailed warning event (cloud provider integration) or an AllocationFailed warning event (MetalLB). The alert shows the message of the event. |
Kubernetes removes events after an hour, so a load balancer that has had no new failure event for an hour is reported as Pending again.
Performance collection rules and views
Service \ Endpoints \ Ready and Service \ Endpoints \ Not Ready, for services with a selector, in the new Service Performance view. The availability of the services rolls up to their namespace. The Service class now also shows the properties Namespace, Type, Cluster IP, Ports and Selector.
Changing the defaults with overrides
- Only be alerted when no endpoint is ready: override HealthyValue of the Kubernetes Service Endpoints monitor to ^(Ready|PartiallyReady|ScaledToZero|NotApplicable)$.
- Also be alerted about services of workloads scaled to 0: override HealthyValue to ^(Ready|NotApplicable)$.
- Change how long a state has to last: override HealthyNumSamples, WarningNumSamples or CriticalNumSamples.
- Clusters without a load balancer controller: disable the Kubernetes Service Load Balancer monitor for those services or that cluster.
Requirements
The account in the kube config needs list permission on services, endpointslices (API group discovery.k8s.io), pods, deployments, statefulsets and events. The read-only cluster role in How to create a Kubernetes monitoring account already grants all of these.
Troubleshooting
- No ready endpoints. The pods of the service crash, fail their readiness probe, cannot be scheduled, wait for a volume or run on a node that is down; or the selector does not match the labels of the pods. Run kubectl get endpointslices -l kubernetes.io/service-name=NAME -n NAMESPACE and kubectl describe pod for the pods in the alert.
- Load balancer pending. Check whether the cluster has a load balancer controller and run kubectl describe service NAME -n NAMESPACE to see its events. Cloud providers can fail because of a quota, missing permissions, an invalid annotation or a subnet without free addresses; MetalLB because its address pools have no free address.