New update available - Opslogix Kubernetes Management Pack (V.26.11.4546.0)

We are happy to announce that a new release of our Opslogix Kubernetes Management Pack for SCOM (V.26.11.4546.0) is now available.

You can download the latest version from our Customer Portal.

This release extends the monitoring to the places where Kubernetes outages often start unnoticed: services without a working backend, persistent volume claims that cannot bind, grow or run out of space, and node disks that are filling up before the kubelet starts evicting pods. Pod CPU and memory utilization is now measured against the pod limits, so you are warned before a container runs out of memory.

Before you upgrade

• The account in the kube config needs a few more read permissions: get on nodes/proxy, and list on persistent volume claims, storage classes, endpoint slices, stateful sets and events. The read-only cluster role from our knowledge base already grants all of them.
• After the import, open a cluster in the Kubernetes MP Configuration and click Test Configuration to upgrade the collector, and then restart the Microsoft Monitoring Agent on the members of the resource pool.
• Expect alerts right after the upgrade for problems that already exist, such as services without ready endpoints or nodes with almost full disks.

The Upgrade Guide lists all steps and what changes after the upgrade.

New updates:

• Added: Kubernetes Service Endpoints monitor. Critical when a service has no ready endpoint, also when its selector matches no pods (2 consecutive samples); Warning when some endpoints are not ready (3 consecutive samples, so a rolling update does not raise an alert). The alert lists the pods the selector matches and their state. Services without a selector, ExternalName services and services whose deployments or stateful sets are scaled to 0 replicas are healthy. See How service endpoint and load balancer monitoring works.
• Added: Kubernetes Service Load Balancer monitor. Warning when a service of type LoadBalancer has no external address after 3 consecutive samples, Critical when a load balancer controller reports a failure in a SyncLoadBalancerFailed (cloud provider integration) or AllocationFailed (MetalLB) warning event. The alert shows the event message.
• Added: performance collection rules for the ready and the not ready endpoints of a service, a Service Performance view and a rollup of the service availability to its namespace. The Service class has the new properties Namespace, Type, Cluster IP, Ports and Selector; services discovered before the upgrade get them at their next hourly discovery.
• Added: Kubernetes Persistent Volume Claim class, discovered hourly under each namespace, with state, alert and performance views in the new Persistent Volume Claims folder. See How persistent volume claim monitoring works.
• Added: Kubernetes Persistent Volume Claim State monitor. Bound is healthy, Pending is a warning and Lost is critical, after 2 consecutive samples. A pending claim of a storage class that binds on first use (WaitForFirstConsumer) and that no pod uses yet is healthy.
• Added: Kubernetes Persistent Volume Claim Resize monitor. Warning when an expansion stays in Resizing or FileSystemResizePending for 3 consecutive samples, Critical when the resize keeps failing: a ControllerResizeError or NodeResizeError condition (Kubernetes 1.31 and later), or a VolumeResizeFailed or FileSystemResizeFailed warning event while the claim is still resizing. Most storage drivers only report failures as events; because Kubernetes throttles repeated events and removes them after an hour, the collector remembers a failure until the resize completes or another size is requested. The alert shows the error message.
• Added: Kubernetes Persistent Volume Claim Used Percentage (Warning above 80%, Critical at 90%) and Inodes Used Percentage (Warning above 85%, Critical at 95%) monitors, and performance collection rules for claim capacity, used percentage, available bytes and inodes used percentage. Usage is only reported while a running pod has the volume mounted and the storage driver supports volume stats.
• Added: node disk usage monitoring from the kubelet Summary API. New monitors: Kubernetes Node Filesystem Used Percentage (Warning above 80%, Critical at 90%), Kubernetes Node Filesystem Inodes Used Percentage (Warning above 85%, Critical at 95%) and Kubernetes Node Image Filesystem Used Percentage (Warning above 75%, Critical at 85%), in line with the default kubelet eviction thresholds. New performance collection rules: node filesystem used percentage, available bytes, capacity, inodes used percentage and image filesystem used percentage. See How disk usage monitoring works for Kubernetes nodes and pods.
• Added: Kubernetes Pod Ephemeral Storage Used Percentage of Limit monitor (Warning above 80%, Critical at 95%), which warns before the kubelet evicts a pod for exceeding its ephemeral-storage limit. Only pods with an ephemeral-storage limit are monitored. New performance collection rule: Kubernetes Pod Ephemeral Storage Used (bytes, all pods).
• Added: monitor type OpsLogix.IMP.Kubernetes.Library.MonitorType.3Stage.Obj.Components.PropertyValue.PerStateSamples in the Library management pack, a property value monitor type with a separate number of samples for the healthy, the warning and the critical state.
• Changed: permissions. The account in the kube config needs get permission on nodes/proxy for disk usage, and list permission on persistentvolumeclaims, storageclasses, endpointslices (API group discovery.k8s.io), pods, deployments, statefulsets and events for the new claim and service monitors. The documented read-only ClusterRole already grants all of these. Without nodes/proxy, or when the kubelet of a node cannot be reached, no disk data is collected for that node and the collector logs one warning per interval.
• Changed: the collector has new endpoints for persistent volume claims and services, and new settings in appsettings.json: KubeletStats:Enabled, CacheSeconds, RequestTimeoutSeconds and MaxConcurrentRequests. The collector makes one kubelet call per node per interval, shared by the node, pod and claim metrics.
• Changed: the metrics data source (OpsLogix.IMP.Kubernetes.Library.DataSource.Metrics.Components) drops samples whose value is missing from the property bag instead of passing an empty value to the performance mapper, so a value that is unknown, such as disk usage when the kubelet cannot be reached, collects no data.
• Fixed: Pod CPU and Memory Utilization Percentage rules and monitors calculated the percentage against the node capacity instead of the CPU and memory limits of the pod, so a pod close to its limit never showed a high percentage and SCOM could not warn before the container was killed for running out of memory. The percentage is now calculated against the sum of the container limits of the pod, and against the node capacity only when the pod has no limit.
• Fixed: pod state monitors (Pod Phase and the pod condition monitors) received no data for any pod in a cluster while one pod was not scheduled to a node, for example a pod waiting for resources. Pods that are not scheduled yet are now included.

Haven't had the chance to try the Opslogix Kubernetes Management Pack yet?

Start your 30-day free and non-binding trial below!

Are you unable to find the new version in the portal? Contact us to try the newest version or to get assistance updating your license key.