Список алертов

На странице приведен список алертов мониторинга в Deckhouse Platform.

Алерты сгруппированы по модулям. Справа от названия алерта указаны иконки минимальной редакции DP в которой доступен алерт и уровня критичности алерта (severity).

Для каждого алерта приведено краткое описание (summary), раскрыв которое можно увидеть подробное описание алерта (description), при их наличии.

Критичность алерта

В описании алертов присутствует параметр Severity (S), означающий уровень критичности. Его значение варьируется от S1 до S9 и может расцениваться следующим образом:

• S1 — максимальный уровень, критический сбой/авария (требуются незамедлительные действия);
• S2 — высокий уровень, близкий к максимальному, возможная авария (необходимо быстрое реагирование);
• S3 — средний уровень, потенциально серьёзная проблема (необходима проверка);
• S4-S9 — низкий уровень. Есть проблема, но в целом работоспособность не нарушена.

Модуль admission-policy-engine

• ControllerPolicyViolation CE S4
  Pod-creating controllers in the XXX namespace violate the XXX policy, so their Pods are denied.

  The XXX policy (XXX) denies the Pods that violate it. For a transition period, it only warns about a violation in a pod-creating controller (Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicationController) instead of denying the controller. The controller is accepted, but it cannot create its Pods: the denial appears only in the events of the ReplicaSet or the controller.

  Controllers that violate the policy in the namespace: VALUE.

  The d8-pod-security-baseline and d8-pod-security-restricted policies are the Pod Security Standards.

  To list the violating controllers and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="XXX",
      name=~"d8-controllers-XXXXXX-deny(-d8)?(-default)?XXX",
      violating_namespace="XXX"
    }
  )
  

  Bring the pod template of each controller in line with the policy. To deny such controllers instead of warning about them, set spec.controllerValidation.mode: Enforce in the policy, or the podSecurityStandards.controllerValidationEnforcement module parameter to Policy. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• D8AdmissionPolicyEngineAuditUnavailable CE S4
  The audit controller of admission-policy-engine has no available replicas.

  The gatekeeper-audit Deployment in the d8-admission-policy-engine namespace has no available replicas.

  Admission of new objects is not affected, but the module stops reporting and stops picking up new policies:

  • Violations among the objects that already run are not collected, so the PodSecurityStandardsViolation, OperationPolicyViolation and SecurityPolicyViolation alerts go silent instead of reporting that all is well.
  • The list of resources the webhook intercepts is not updated, so a policy added while the audit is down is not enforced on admission.

  Find out why the replica is gone:

  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=audit-controller
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=audit-controller
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c manager --tail=200
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c constraint-exporter --tail=200
  

• D8AdmissionPolicyEngineNotBootstrapped CE S7
  Not every ConstraintTemplate of the admission-policy-engine module is ready for 10 minutes.

  Some ConstraintTemplates of the admission-policy-engine module are missing or not processed by Gatekeeper. The constraints of a template are applied only once Gatekeeper has established its CRD.

  On a new installation, the module also renders no webhooks and no policies until every template is ready. In a cluster where the module has already worked, the webhooks and the constraints of the other templates stay in effect.

  Steps to troubleshoot:

  1. Verify that the module’s components are up and running:

     d8 k get pods -n d8-admission-policy-engine
     

  2. Check logs for issues, such as missing constraint templates or incomplete CRD creation:

     d8 k logs -n d8-system -lapp=deckhouse --tail=1000 | grep admission-policy-engine
     

• D8AdmissionPolicyEngineWebhookDegraded CE S3
  The admission webhook of admission-policy-engine has unavailable replicas.

  The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has unavailable replicas, and the remaining ones still serve admission requests.

  The webhook has no redundancy left. Losing the last replica makes the API server reject every request the webhook intercepts, because the webhook has failurePolicy: Fail.

  A rollout of the Deployment raises this alert for a short time and needs no action. If the alert holds, find out why the replica does not start:

  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager
  d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30
  

• D8AdmissionPolicyEngineWebhookNotServing CE S2
  The API server cannot call the admission webhook of admission-policy-engine.

  The API server is rejecting requests because it fails to call a webhook of the admission-policy-engine module. The webhook has failurePolicy: Fail, so the cluster is blocked for everything the webhook intercepts.

  The alert is independent of the replica counters: it fires while the pods are Ready as well. A replica that is running but unreachable to the API server produces the same outage, and only this alert reports it. Typical causes are a gatekeeper-webhook-server-cert that no longer matches the caBundle of the ValidatingWebhookConfiguration, an empty gatekeeper-webhook-service endpoint list, and a network policy that blocks the API server.

  The API server counts a webhook that exceeds its timeoutSeconds of 15 the same way it counts one it cannot reach, so a few slow evaluations of a large object raise the metric as well. They do not raise the alert: for: 15m requires the failures to continue, while a burst leaves the rate elevated for little more than its own duration plus the five-minute window.

  Read the reason the API server reports, then check the certificate and the endpoints:

  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200
  d8 k -n d8-admission-policy-engine get endpoints gatekeeper-webhook-service
  d8 k get validatingwebhookconfiguration d8-admission-policy-engine-config -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -dates
  d8 k -n d8-admission-policy-engine get secret gatekeeper-webhook-server-cert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -dates
  

  The two certificates must match. Deckhouse Platform reissues them through the module’s hook, so deleting the secret makes it regenerate the pair.

• D8AdmissionPolicyEngineWebhookUnavailable CE S2
  The admission webhook of admission-policy-engine has no available replicas.

  The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has no available replicas.

  The Deployment serves the admission webhook of the module, and the webhook has failurePolicy: Fail. While no replica is available, the API server rejects every request the webhook intercepts, cluster-wide:

  • Pods and the controllers that create them are not created or updated.
  • Deletion of those resources is rejected as well.
  • Roles, RoleBindings and Gatekeeper constraints are not applied.
  • The d8 k exec, d8 k attach and d8 k debug requests intercepted by the webhooks are rejected. This includes d8 k exec and d8 k attach in the d8-* and kube-* namespaces.

  Find out why the replicas are gone:

  d8 k -n d8-admission-policy-engine get deploy gatekeeper-controller-manager
  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200
  

  If a pod stays Pending, look at the events of the namespace and at the nodes the replicas can run on:

  d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30
  d8 k get nodes -l node-role.deckhouse.io/system
  d8 k get nodes -l node-role.kubernetes.io/control-plane
  

• OperationPolicyViolation CE S3
  XXXObjects in the XXX namespaceXXXCluster-scoped objectsXXX violate the XXX OperationPolicy.

  Existing objects XXXin the XXX namespace XXXof cluster-scoped kinds XXXviolate the XXX OperationPolicy. An object is checked when it is created or updated, so these objects were created before the policy applied to them or before its rules changed.

  Objects that violate the policy: VALUE.

  To list the violating objects and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="OperationPolicy",
      name="XXX",
      violation_enforcement="deny",
      violating_namespace="XXX"
    }
  )
  

  Bring the objects in line with the policy. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• PodSecurityStandardsViolation CE S3
  Pods in the XXX namespace violate the XXX Pod Security Standard.

  Running Pods in the XXX namespace violate the Pod Security Standard published as the XXX SecurityPolicy. A Pod is checked when it is created, so these Pods were created before the standard applied to them or before its rules changed. They keep running until they are restarted, and are denied then.

  Pods that violate the standard in the namespace: VALUE.

  To list the violating Pods and the rules they break, run the following Prometheus query:

  count by (violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="PSS",
      name=~"XXX-.+",
      violation_enforcement="deny",
      violating_kind="Pod",
      violating_namespace="XXX"
    }
  )
  

  Bring the workloads in line with the standard, or exempt them with a SecurityPolicyException. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• SecurityPolicyViolation CE S3
  XXXObjects in the XXX namespaceXXXCluster-scoped objectsXXX violate the XXX SecurityPolicy.

  Existing objects XXXin the XXX namespace XXXof cluster-scoped kinds XXXviolate the XXX SecurityPolicy. An object is checked when it is created or updated, so these objects were created before the policy applied to them or before its rules changed.

  Objects that violate the policy: VALUE.

  To list the violating objects and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="SecurityPolicy",
      name="XXX",
      violation_enforcement="deny",
      violating_namespace="XXX"
    }
  )
  

  Bring the objects in line with the policy, or exempt them with a SecurityPolicyException. You can also check the admission-policy-engine reports in the Deckhouse web interface.

Модуль cert-manager

• CertmanagerCertificateChallengePending CE S4
  ACME Challenge for domain "XXX" is pending for more than 1 hour.

  The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 1 hour and requires investigation.

  Common causes:

  • DNS misconfiguration (for DNS-01 challenges).
  • Ingress or firewall issues blocking HTTP-01 validation.
  • ACME provider rate limiting.
  • Network connectivity issues.

  Resolution steps:

  1. Check the Challenge status and reason:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Check cert-manager logs for errors:

     d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
     

  3. If the challenge is stuck, delete the Order to trigger a retry:

     d8 k -n NAMESPACE get orders
     d8 k -n NAMESPACE delete order ORDER_NAME
     

• CertmanagerCertificateChallengePending CE S6
  ACME Challenge for domain "XXX" is pending for more than 10 minutes.

  The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 10 minutes.

  This may indicate issues with the ACME challenge validation process.

  Common causes:

  • DNS propagation delays (for DNS-01 challenges).
  • Ingress or firewall issues blocking HTTP-01 validation.
  • ACME provider rate limiting or temporary unavailability.

  Diagnostic steps:

  1. Check the Challenge status:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Check cert-manager logs for errors:

     d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
     

• CertmanagerCertificateChallengeStuck CE S4
  ACME Challenge for domain "XXX" is stuck in "STATUS_REFERENCE" state.

  The ACME Challenge NAMESPACE/NAME for domain XXX is in STATUS_REFERENCE state and requires manual intervention.

  Reason: XXX

  The certificate will not be issued or renewed until the issue is resolved.

  Common causes:

  • DNS misconfiguration (for DNS-01 challenges).
  • Ingress or firewall issues (for HTTP-01 challenges).
  • Rate limiting by the ACME provider.
  • Authorization expired before challenge completion.

  Resolution steps:

  1. Check the Challenge status and reason:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Delete the failed Order to trigger a new certificate request:

     d8 k -n NAMESPACE get orders
     d8 k -n NAMESPACE delete order ORDER_NAME
     

• CertmanagerCertificateExpired CE S4
  Certificate NAMESPACE/NAME is not provided.

  Certificate is not provided.

  To check the certificate details, run the following command:

  d8 k -n NAMESPACE describe certificate NAME
  

• CertmanagerCertificateExpiredSoon CE S4
  Certificate will expire soon.

  The certificate NAMESPACE/NAME will expire in less than two weeks.

  To check the certificate details, run the following command:

  d8 k -n NAMESPACE describe certificate CERTIFICATE-NAME
  

• CertmanagerCertificateOrderErrors CE S5
  Cert-manager couldn't order a certificate.

  Cert-manager received responses with the status code STATUS_REFERENCE when requesting SCHEME://HOST_NAMEPATH.

  This can affect certificate ordering and prolongation in the future. For details, check the cert-manager logs using the following command:

  d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager
  

Модуль chrony

• NodeTimeOutOfSync CE S5
  Clock on the node NODE_NAME is drifting.

  Time on the node NODE_NAME is out of sync and drifts apart from the NTP server clock by VALUE seconds.

  To resolve the time synchronization issues:

  • Fix network errors:
    • Ensure the upstream time synchronization servers defined in the chrony configuration are available.
    • Eliminate large packet loss and excessive latency to upstream time synchronization servers.
  • Modify the NTP servers list defined in the chrony configuration.
• NTPDaemonOnNodeDoesNotSynchronizeTime CE S5
  NTP daemon on the node NODE_NAME haven't synchronized time for too long.

  Steps to troubleshoot:

  1. Check if the chrony pod is running on the node by executing the following command:

     d8 k -n d8-chrony --field-selector spec.nodeName="NODE_NAME" get pods
     

  2. Verify the chrony daemon’s status by executing the following command:

     d8 k -n d8-chrony exec POD_NAME -- /opt/chrony-static/bin/chronyc sources
     

  3. Resolve the time synchronization issues:

     • Fix network errors:
       • Ensure the upstream time synchronization servers defined in the chrony configuration are available.
       • Eliminate large packet loss and excessive latency to upstream time synchronization servers.
     • Modify the NTP servers list defined in the chrony configuration.

Модуль cloud-provider-aws

• D8CloudProviderAWSIMDSv2NotConverged CE S6
  Some EC2 instances of the cluster still allow IMDSv1.

  imdsv2: true is set in AWSClusterConfiguration, but VALUE EC2 instances of the cluster still accept metadata requests without a session token, which leaves them more exposed to unauthorized metadata access, including through SSRF attacks.

  Existing instances are not reconfigured by the setting alone:

  1. Master, static and bastion instances are updated by a converge run. Start the DKP installer and run dhctl converge.
  2. Ephemeral instances are re-created when their nodes are rolled, which happens on its own. Check that the nodes of every CloudEphemeral NodeGroup are being replaced: d8 k get nodegroups.

  The instance state is re-read once per discovery period of the cloud data discoverer (one hour by default), so the alert can lag behind the last node being replaced.

Модуль cloud-provider-dvp

• D8CloudProviderDVPMigrationPending CE S6
  The cloud-provider-dvp module is waiting for migration resources to be applied.

  The cloud-provider-dvp module has detected the legacy DVPProviderClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

  To complete the migration, apply the resources from the d8-migration-resources Secret:

  d8 k -n d8-cloud-provider-dvp get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
  

  After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

Модуль cloud-provider-huaweicloud

• D8HuaweiCloudMachineStuckInDeleting EE S6
  HuaweiCloudMachine NAME is stuck in Deleting for more than 24h.

  The HuaweiCloudMachine NAME in namespace NAMESPACE has been stuck in Deleting state for more than 24 hours.

  This usually means the deletion job in HuaweiCloud is failing or the API is unavailable.

  To inspect the machine status, run:

  d8 k get huaweicloudmachine -n NAMESPACE NAME -o yaml
  

  Check the controller logs for errors:

  d8 k logs -n d8-cloud-provider-huaweicloud -l app=caphc-controller-manager
  

Модуль cloud-provider-openstack

• OpenStackPreemptibleUnsupported EE S8
  NodeGroup NODE_GROUP_NAME requests preemptible instances, but this cluster cannot deliver them (reason: XXX).

  The OpenStackInstanceClass referenced by NodeGroup NODE_GROUP_NAME has spec.preemptible: true, but the raw Nova preemptible tag will not reach the virtual machines: nodes are created as regular (non-preemptible) VMs, and Selectel-side preemption never kicks in.

  XXX Reason: the NodeGroup runs on the MCM engine, which has no way to emit raw Nova tags — machine-controller-manager’s OpenStackMachineClass CRD does not expose the field. Only the CAPI engine (capi-controller-manager + capo-controller-manager) renders raw Nova tags via OpenStackMachineTemplate.spec.template.spec.tags.

  How to fix:

  1. Confirm the current engine and offending InstanceClass:

     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.status.engine}{"\n"}'
     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.spec.cloudInstances.classReference}{"\n"}'
     

  2. Choose one of the following actions:

     • Recommended. Remove the node.deckhouse.io/use-mcm annotation from the NodeGroup so it migrates to CAPI. Nodes will be recreated on the CAPI engine, this time carrying the preemptible tag:

       d8 k annotate ng NODE_GROUP_NAME node.deckhouse.io/use-mcm-
       

     • Alternatively, if this NodeGroup must stay on MCM, set spec.preemptible: false on the OpenStackInstanceClass (nodes stay regular, the alert clears):

       d8 k patch openstackinstanceclass CLASS_NAME --type=merge -p '{"spec":{"preemptible":false}}'
       

       Note: a single OpenStackInstanceClass shared between a CAPI NodeGroup (where the field works) and an MCM one (where it doesn’t) will keep firing this alert on the MCM side. Use separate InstanceClasses per engine when the classes differ in preemptible.

  XXX Reason: the cluster’s connection.authURL does not point at Selectel (selcloud.ru / selectel). The raw Nova preemptible tag is a Selectel-specific mechanism; on other OpenStack providers Nova accepts the tag but ignores it, so instances are billed and behave as regular non-preemptible VMs.

  How to fix:

  1. Confirm the offending InstanceClass and the cluster’s authURL:

     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.spec.cloudInstances.classReference}{"\n"}'
     d8 k -n kube-system get secret d8-node-manager-cloud-provider -o jsonpath='{.data.openstack}' | base64 -d | jq -r '.connection.authURL'
     

  2. Set spec.preemptible: false on the OpenStackInstanceClass — the field only makes sense on Selectel-hosted clusters:

     d8 k patch openstackinstanceclass CLASS_NAME --type=merge -p '{"spec":{"preemptible":false}}'
     

     The alert clears on the next scrape after the change; existing nodes stay running (they are already regular VMs — the tag was never emitted).

  XXX Reason: unknown (XXX). This alert should surface only two well-known cases (mcm, non-selectel). A new reason implies node-controller has been extended without this runbook — check the release notes for cloud-provider-openstack before deciding what to do. XXX

Модуль cloud-provider-yandex

• D8CloudProviderYandexCloudMigrationPending CE S6
  The cloud-provider-yandex module is waiting for migration resources to be applied.

  The cloud-provider-yandex module has detected the legacy YandexProviderClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

  To complete the migration, apply the resources from the d8-migration-resources Secret:

  d8 k -n d8-cloud-provider-yandex get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
  

  After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

  After the migration is complete, the legacy YandexProviderClusterConfiguration secret is no longer needed and can be deleted manually:

  1. Make sure the cloud discovery data is stored outside the legacy secret. The module reads it from the d8-candi-cloud-provider-discovery-data Secret first and uses the legacy secret only as a fallback, so the following command must print a non-empty value:

     d8 k -n d8-cloud-provider-yandex get secret d8-candi-cloud-provider-discovery-data -o jsonpath='{.data.cloud-provider-discovery-data\.json}'
     

     If it prints nothing, the discovery data exists only in the legacy secret, and deleting that secret would lose it. Copy the payload over — the key holds exactly the same discovery data dhctl recorded from the infrastructure run:

     d8 k -n kube-system get secret d8-provider-cluster-configuration \
       -o jsonpath='{.data.cloud-provider-discovery-data\.json}' | base64 -d > /tmp/yandex-discovery.json
     
     d8 k -n d8-cloud-provider-yandex create secret generic d8-candi-cloud-provider-discovery-data \
       --from-file=cloud-provider-discovery-data.json=/tmp/yandex-discovery.json \
       --dry-run=client -o yaml | d8 k apply -f -
     

  2. Back up the secret. It contains cloud credentials and discovery data, so store the file securely:

     d8 k -n kube-system get secret d8-provider-cluster-configuration -o yaml > d8-provider-cluster-configuration-backup.yaml
     

  3. Allow the deletion. The secret is protected by an admission webhook and cannot be deleted without the annotation:

     d8 k -n kube-system annotate secret d8-provider-cluster-configuration deckhouse.io/allow-delete=true
     

  4. Delete the secret:

     d8 k -n kube-system delete secret d8-provider-cluster-configuration
     

• D8YandexNatInstanceConnectionsQuotaUtilization CE S4
  Connection quota utilization of the Yandex NAT instance exceeds 85% over the last 5 minutes.

  The connection quota for the Yandex NAT instance has exceeded 85% utilization over the past 5 minutes.

  To prevent potential issues, contact Yandex technical support and request an increase in the connection quota.

• NATInstanceWithDeprecatedAvailabilityZone CE S9
  NAT instance NAME is in a deprecated availability zone.

  The NAT instance NAME is located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud. To resolve this issue, migrate the NAT instance to either ru-central1-a or ru-central1-b by following the instructions below.

  The migration process involves irreversible changes and may result in a significant downtime. The duration typically depends on Yandex Cloud’s response time and can last several tens of minutes.

  1. Migrate the NAT instance. To get providerClusterConfiguration.withNATInstance, run the following command:

     d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerClusterConfiguration.withNATInstance'
     

     • If you specified withNATInstance.natInstanceInternalAddress and/or withNATInstance.internalSubnetID in providerClusterConfiguration, remove them using the following command:

       d8 system edit provider-cluster-configuration
       

     • If you specified withNATInstance.externalSubnetID and/or withNATInstance.natInstanceExternalAddress in providerClusterConfiguration, change them to the appropriate values.

       To get the address and subnet ID, use the Yandex Cloud Console or CLI.

       To change withNATInstance.externalSubnetID and withNATInstance.natInstanceExternalAddress, run the following command:

       d8 system edit provider-cluster-configuration
       

  2. Run the appropriate edition and version of the Deckhouse installer container on the local machine. You may have to change the container registry address to do that. After that, perform the converge.

     • Get the appropriate edition and version of Deckhouse:

       DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}')
       DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]')
       echo "DH_VERSION=$DH_VERSION DH_EDITION=$DH_EDITION"
       

     • Run the installer:

       docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
       

     • Perform the converge:

       dhctl converge --ssh-agent-private-keys=/tmp/.ssh/SSH_KEY_FILENAME --ssh-user=USERNAME --ssh-host MASTER-NODE-0-HOST
       

  3. Update the route table.

     • Get the route table name:

       d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.global.prefix // .global.clusterConfiguration.cloud.prefix'
       

     • Get the NAT instance name:

       d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerDiscoveryData.natInstanceName'
       

     • Get the NAT instance internal IP address:

       yc compute instance list | grep -e "INTERNAL IP" -e NAT_INSTANCE_NAME_FROM_PREVIOUS_STEP
       

     • Update the route:

       yc vpc route-table update --name ROUTE_TABLE_NAME_FROM_PREVIOUS_STEP --route "destination=0.0.0.0/0,next-hop=NAT_INSTANCE_INTERNAL_IP_FROM_PREVIOUS_STEP"
       

• NodeGroupNodeWithDeprecatedAvailabilityZone CE S9
  NodeGroup NODE_GROUP_NAME contains nodes in a deprecated availability zone.

  Certain nodes in the node group NODE_GROUP_NAME are located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud.

  Steps to troubleshoot:

  1. Identify the nodes that need to be migrated by running the following command:

     d8 k get node -l "topology.kubernetes.io/zone=ru-central1-c"
     

  2. Migrate your nodes, disks, and load balancers to one of the supported zones: ru-central1-a, ru-central1-b, or ru-central1-d. Refer to the Yandex migration guide for detailed instructions.

     You can’t migrate public IP addresses between zones. For details, refer to the migration guide.

Модуль cloud-provider-zvirt

• D8CloudProviderZvirtMigrationPending SE-PLUS S6
  The cloud-provider-zvirt module is waiting for migration resources to be applied.

  The cloud-provider-zvirt module has detected the legacy ZvirtClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

  To complete the migration, apply the resources from the d8-migration-resources Secret:

  d8 k -n d8-cloud-provider-zvirt get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
  

  After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

  After the migration is complete, the legacy ZvirtProviderClusterConfiguration secret is no longer needed and can be deleted manually:

  1. Make sure the cloud discovery data is stored outside the legacy secret. The module reads it from the d8-candi-cloud-provider-discovery-data Secret first and uses the legacy secret only as a fallback, so the following command must print a non-empty value:

     d8 k -n d8-cloud-provider-zvirt get secret d8-candi-cloud-provider-discovery-data -o jsonpath='{.data.cloud-provider-discovery-data\.json}'
     

     If it prints nothing, the discovery data exists only in the legacy secret, and deleting that secret would lose it. Copy the payload over — the key holds exactly the same discovery data dhctl recorded from the infrastructure run:

     d8 k -n kube-system get secret d8-provider-cluster-configuration \
       -o jsonpath='{.data.cloud-provider-discovery-data\.json}' | base64 -d > /tmp/zvirt-discovery.json
     
     d8 k -n d8-cloud-provider-zvirt create secret generic d8-candi-cloud-provider-discovery-data \
       --from-file=cloud-provider-discovery-data.json=/tmp/zvirt-discovery.json \
       --dry-run=client -o yaml | d8 k apply -f -
     

  2. Back up the secret. It contains cloud credentials and discovery data, so store the file securely:

     d8 k -n kube-system get secret d8-provider-cluster-configuration -o yaml > d8-provider-cluster-configuration-backup.yaml
     

  3. Allow the deletion. The secret is protected by an admission webhook and cannot be deleted without the annotation:

     d8 k -n kube-system annotate secret d8-provider-cluster-configuration deckhouse.io/allow-delete=true
     

  4. Delete the secret:

     d8 k -n kube-system delete secret d8-provider-cluster-configuration
     

Модуль cni-cilium

• CiliumAgentEndpointsNotReady CE S4
  Over 50% of all known endpoints aren't ready in agent NAMESPACE/POD_NAME.

  For details, refer to the logs of the agent:

  d8 k -n NAMESPACE logs POD_NAME
  

• CiliumAgentMapPressureCritical CE S4
  eBPF map MAP_NAME exceeds 90% utilization in agent NAMESPACE/POD_NAME.

  The eBPF map resource utilization limit has almost been reached.

  Check with the vendor for potential remediation steps.

• CiliumAgentMetricNotFound CE S4
  Agent NAMESPACE/POD_NAME isn't sending some metrics.

  Steps to troubleshoot:

  1. Check the logs of the agent:

     d8 k -n NAMESPACE logs POD_NAME
     

  2. Verify the agent’s health status:

     d8 k -n NAMESPACE exec -ti POD_NAME cilium-health status
     

  3. Compare the metrics with those of a neighboring agent.

  Note that the absence of metrics can indirectly indicate that new pods can’t be created on the node due to connectivity issues with the agent.

• CiliumAgentPolicyImportErrors CE S4
  Agent NAMESPACE/POD_NAME fails to import policies.

  For details, refer to the logs of the agent:

  d8 k -n NAMESPACE logs POD_NAME
  

• CiliumAgentUnreachableHealthEndpoints CE S4
  Agent NAMESPACE/POD_NAME can't reach some of the node health endpoints.

  For details, query Cilium health status API:

  d8 k -n NAMESPACE exec POD_NAME -- cilium-health status
  

  You can also refer to the agent logs:

  d8 k -n NAMESPACE logs POD_NAME
  

• CniCiliumNonStandardVXLANPortFound CE S4
  Cilium configuration uses a non-standard VXLAN port.

  The Cilium configuration specifies a non-standard VXLAN port XXX. The recommended port number is XXX.

  To resolve this issue, update the tunnel-port parameter in the cilium-configmap ConfigMap located in the d8-cni-cilium namespace to match the recommended port.

  If you configured the non-standard port on purpose, ignore this alert.

• CniCiliumOrphanEgressGatewayPolicyFound SE-PLUS S4
  Orphaned EgressGatewayPolicy with an irrelevant EgressGateway name has been found.

  The cluster contains an orphaned EgressGatewayPolicy named NAME with an irrelevant EgressGateway name.

  To resolve this issue, verify the EgressGateway name specified in the EgressGatewayPolicy resource EGRESSGATEWAY and update it as needed.

• CNICiliumPolicyWithRequiresFieldsFound CE S4
  The XXX XXXNAMESPACE/XXXNAME uses the fromRequires or toRequires fields.

  The XXX NAMEXXX in the NAMESPACE namespaceXXX uses the fromRequires or toRequires fields.

  Cilium 1.19 and later doesn’t support these fields. The Cilium agent of these versions can’t read such policies and fails on all nodes. Deckhouse won’t deploy a release with such a Cilium version until all these policies are rewritten.

  To resolve this issue, rewrite each fromRequires constraint as an ingressDeny rule with an inverted selector. Don’t move the selector into the deny rule as is: this inverts the policy. A deny rule wins over allow rules of any policy, so it keeps the original scope: fromRequires applied to all policies selecting the endpoint.

  For example, fromRequires with matchLabels: {env: prod} becomes:

  ingressDeny:
  - fromEndpoints:
    - matchExpressions:
      - {key: env, operator: NotIn, values: [prod]}
  XXX
    - matchExpressions:
      - {key: k8s:io.kubernetes.pod.namespace, operator: Exists}
      - {key: k8s:io.kubernetes.pod.namespace, operator: NotIn, values: [NAMESPACE]}
  XXX
  

  • Add a separate NotIn selector for each label of the constraint. XXX
  • Keep the second selector: in a CiliumNetworkPolicy, fromRequires also implied the policy namespace, so sources from other namespaces were rejected. The Exists condition is required, otherwise the rule also blocks the host, the world, and other sources without a namespace. XXX
  • Rewrite toRequires the same way, as an egressDeny rule with toEndpoints.
  • Add the deny rule first, check the traffic, and only then remove fromRequires and toRequires.

  To list all affected policies, use the d8_cni_cilium_policy_with_requires_fields == 1 expression in Prometheus.

• D8NodeHasUnmetKernelRequirements CE S4
  Nodes have unmet kernel requirements.

  Deckhouse has detected that some nodes don’t meet the required kernel constraints. As a result, certain modules can’t run on these nodes.

  Kernel requirements for each module:

  • Cilium module: Kernel version must be >= 5.8.

  To list all affected nodes, use the d8_node_kernel_does_not_satisfy_requirements == 1 expression in Prometheus.

Модуль control-plane-manager

• D8ControlPlaneComponentsResourcesInsufficientCapacity CE S5
  Automatic control-plane resource tuning cannot increase XXX requests due to lack of available capacity.

  Automatic control-plane resource tuning attempted to increase XXX requests for one or more control-plane components (kube-apiserver, etcd, kube-controller-manager, kube-scheduler), but the calculated requests exceed the available capacity of at least one master node after accounting for the kubelet reservation and resource requests of other (non-control-plane) pods already scheduled on the node.

  XXX Additional memory required: XXXB. XXX Additional CPU required: VALUEm CPU. XXX

  To resolve this, free up resources on master nodes by moving or reducing other workloads or increase the CPU or RAM available on the nodes.

  To disable automatic tuning for this resource and manage requests manually, set spec.settings.resourcesRequests.XXX in the ModuleConfig of control-plane-manager.

• D8ControlPlaneDefaultVersionDrift CE S5
  Control plane Kubernetes version rollback is blocked (Default soft-guard freeze).

  The cluster tracks the Deckhouse default Kubernetes version (kubernetesVersion: Default in ModuleConfig control-plane-manager, or unset ModuleConfig with no ClusterConfiguration pin). That default is now more than 1 minor below the highest version this cluster has ever run (maxUsed).

  Rollback blocked: Deckhouse freezes the published target (holds it at the current control-plane version, or at the last desired version when the current one is not published yet) and does not roll Default down toward the lower default. Control plane and kubelet stay on the frozen target until the Deckhouse default catches up within the maxUsed−1 window.

  Check the frozen label on the metric: frozen="true" means the digit is held as described above. frozen="false" means the window is violated but nothing was left to hold the digit at (for example the d8-cluster-kubernetes ConfigMap was deleted), so the published target has moved down to the Deckhouse default — pin the intended version explicitly.

  To resolve this alert:

  • Do nothing and wait, if you are happy staying on the version the cluster runs now. This is a valid resolution only while frozen="true": the cluster is not degraded, the version is held, and the alert clears by itself once the Deckhouse default rises back into the maxUsed−1 window. How long that takes depends on why the default dropped — after a Deckhouse rollback, until you upgrade Deckhouse again; when the release default is simply older than the version this cluster runs, until a Deckhouse release ships a high enough default, which may be several releases away.

  • To keep the current (higher) version and stop the alert now, pin it explicitly:

    d8 k patch mc control-plane-manager --type merge -p '{"spec":{"settings":{"kubernetesVersion":"current version"}}}'
    

  • To downgrade deliberately, pin one minor at a time (more than 1 minor below maxUsed is rejected):

    d8 k patch mc control-plane-manager --type merge -p '{"spec":{"settings":{"kubernetesVersion":"current version - 1 minor"}}}'
    

• D8ControlPlaneManagerObsoleteMonitoringKubernetesControlPlaneModuleConfig CE S9
  Obsolete monitoring-kubernetes-control-plane ModuleConfig should be removed.

  The monitoring-kubernetes-control-plane module was merged into the control-plane-manager module, but its ModuleConfig still exists in the cluster and is no longer used.

  Remove it manually:

  d8 k delete moduleconfig monitoring-kubernetes-control-plane
  

  If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.

• D8ControlPlaneManagerPodNotRunning CE S6
  Controller pod isn't running on node NODE_NAME.

  The d8-control-plane-manager pod is either failing or hasn’t been scheduled on node NODE_NAME.

  To resolve this issue, check the status of the kube-system/d8-control-plane-manager DaemonSet and its pods by running the following command:

  d8 k -n kube-system get daemonset,pod --selector=app=d8-control-plane-manager
  

• D8ControlPlaneNodeMaintenanceModeEnabled CE S4
  Maintenance mode is enabled for control-plane node NODE_NAME.

  Control-plane node NODE_NAME has maintenance mode enabled with label control-plane-manager.deckhouse.io/maintenance. While this label is present, the control plane is not managed by control-plane-manager.

  If maintenance is complete, remove the label:

  d8 k label cpn NODE_NAME control-plane-manager.deckhouse.io/maintenance-
  

• D8ControlPlaneOperationRunningTooLong CE S4
  Control-plane operation XXX for COMPONENT_NAME on node NODE_NAME runs or retries longer than 10 minutes.

  Control-plane operation XXX for component COMPONENT_NAME on node NODE_NAME has been running or retrying for more than 10 minutes.

  Check the corresponding ControlPlaneOperation and logs of the d8-control-plane-manager Pod on that node:

  d8 k get cpo XXX -o yaml
  d8 k -n kube-system logs -l app=d8-control-plane-manager --field-selector spec.nodeName=NODE_NAME
  

• D8EtcdDatabaseHighFragmentationRatio CE S7
  etcd database size in use is less than 50% of the allocated storage.

  The etcd database size in use on instance INSTANCE_NAME is less than 50% of the allocated disk space, indicating potential fragmentation. Additionally, the total storage size exceeds 75% of the configured quota.

  To resolve this issue, defragment the etcd database by running the following command:

  d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert /etc/kubernetes/pki/etcd/ca.crt \
    --key /etc/kubernetes/pki/etcd/ca.key \
    --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s
  

• D8EtcdExcessiveDatabaseGrowth CE S4
  etcd cluster database is growing rapidly.

  Based on the growth rate observed over the last six hours, Deckhouse predicts that the etcd database will run out of disk space within one day on instance INSTANCE_NAME.

  To prevent disruptions, investigate the cause and take necessary action.

• D8EtcdHighRequestsLatency CE S5
  Etcd request latency (p99) has exceeded 800 ms multiple times in the past hour.

  The 99th percentile of etcd request duration on INSTANCE_NAME has exceeded 800 ms more than twice in the last hour.

  High etcd latency can cause API server timeouts, slow admission webhooks, and general cluster instability.

  Investigate the following:

  • Node disk I/O: Etcd is extremely sensitive to disk performance. Check disk utilization and I/O wait on the control plane node.
  • Network-attached storage: If the etcd data directory resides on NFS, Ceph, or another network storage, check connectivity and latency to the storage backend. Etcd requires a consistently low-latency local disk (SSD recommended).
  • Control plane summary: Review the Kubernetes Cluster → Control Plane Status Grafana dashboard for an overall picture of API server and etcd health, request rates, and error rates.
  • Node base performance: Check node CPU and RAM utilization and load average. If the node is CPU-bound (load average significantly exceeds the number of CPU cores), consider scaling up the control plane node.
• D8InvalidEtcdDefragCronSchedule CE S6
  Invalid etcd defragmentation cron schedule in ModuleConfig.

  The controlPlaneManager.etcd.defrag.cronSchedule value XXX is not a valid cron expression.

  Deckhouse has fallen back to the default schedule (0 1 * * *) to keep defragmentation running.

  To fix this, set a valid 5-field cron expression:

  d8 k patch moduleconfig control-plane-manager --type merge \
    -p '{"spec":{"settings":{"etcd":{"defrag":{"cronSchedule":"0 1 * * *"}}}}}'
  

  Valid examples:

  • 0 1 * * * — daily at 01:00 UTC
  • 0 */6 * * * — every 6 hours
  • 30 2 * * 0 — weekly on Sunday at 02:30 UTC
• D8KubeEtcdDatabaseSizeCloseToTheLimit CE S3
  etcd database size is approaching the limit.

  The etcd database size on NODE_NAME is nearing its size limit. This may be caused by a high number of events, such as pod evictions or the recent creation of numerous resources in the cluster.

  Possible solutions:

  • Defragment the etcd database by running the following command:

    d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \
      --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt \
      --key /etc/kubernetes/pki/etcd/ca.key \
      --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s
    

  • Increase node memory. Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

    Example:

    Node memory	quota-backend-bytes
    16 GB	2147483648 (2 GB)
    24 GB	3221225472 (3 GB)
    32 GB	4294967296 (4 GB)
    40 GB	5368709120 (5 GB)
    48 GB	6442450944 (6 GB)
    56 GB	7516192768 (7 GB)
    64 GB	8589934592 (8 GB)
    72 GB	8589934592 (8 GB)
    …	…

• D8KubernetesStaleTokensDetected CE S8
  Stale service account tokens detected for XXX/XXX.

  Service account XXX/XXX is using a stale token. This issue may occur if an application reads the token only at startup and does not reload it periodically. As a result, an outdated token might be used, leading to security breach and authentication failures.

  Recommended actions:

  • Ensure your application is configured to periodically reload the token from the file system.
  • Verify that you are using an up-to-date client library that supports automatic token rotation.

  Note that currently these tokens are not blocked because the --service-account-extend-token-expiration flag is enabled by default (Default: true). With this flag enabled, admission-injected tokens are extended up to 1 year during token generation to facilitate a safe transition from legacy tokens to the bound service account token feature, ignoring the value of service-account-max-token-expiration.

  For further investigation: You can find the problematic pods using the following command:

  d8 k get pods -A -o json | jq -r '.items[] | select(.spec.serviceAccountName=="XXX" and .metadata.namespace=="XXX") | .metadata.name'
  

• D8KubernetesVersionIsDeprecated CE S7
  Kubernetes version VERSION_NUMBER is deprecated.

  The current Kubernetes version VERSION_NUMBER has been deprecated, and support for it will be removed in upcoming releases.

  Please migrate to the next kubernetes version (at least 1.34)

  Read documentation about how to update the Kubernetes version in the cluster.

• D8NeedDecreaseEtcdQuotaBackendBytes CE S6
  Deckhouse suggests reducing quota-backend-bytes.

  When the control plane node memory is reduced, Deckhouse may suggest reducing quota-backend-bytes. While Deckhouse is capable of automatically increasing this value, reducing it must be done manually.

  To modify quota-backend-bytes, set the controlPlaneManager.etcd.maxDbSize parameter. Before setting a new value, check the current database usage on every control plane node by running:

  for pod in $(d8 k get pod -n kube-system -l component=etcd,tier=control-plane -o name); do
    d8 k -n kube-system exec -ti "$pod" -- /usr/bin/etcdctl \
      --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt \
      --key /etc/kubernetes/pki/etcd/ca.key \
      endpoint status -w json | jq --arg a "$pod" -r \
      '.[0].Status.dbSize / 1024 / 1024 | tostring | $a + ": " + . + " MB"';
  done
  

  Things to note:

  • The maximum value for controlPlaneManager.etcd.maxDbSize is 8 GB.
  • If control plane nodes have less than 24 GB, set controlPlaneManager.etcd.maxDbSize to 2 GB.

  • Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

    Example:

    Node memory	quota-backend-bytes
    16 GB	2147483648 (2 GB)
    24 GB	3221225472 (3 GB)
    32 GB	4294967296 (4 GB)
    40 GB	5368709120 (5 GB)
    48 GB	6442450944 (6 GB)
    56 GB	7516192768 (7 GB)
    64 GB	8589934592 (8 GB)
    72 GB	8589934592 (8 GB)
    …	…

• D8ObsoleteClusterDomainInClusterConfiguration CE S6
  The obsolete clusterDomain field is still present in ClusterConfiguration.

  ClusterConfiguration still carries the obsolete clusterDomain field. The place to manage the cluster domain is ModuleConfig control-plane-manager.

  This fires simply because the field is present, even if ModuleConfig control-plane-manager already overrides it - the goal is to remove the deprecated field, not just override it.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Migrate the value straight from the Secret into ModuleConfig:

     DOMAIN=$(d8 k -n kube-system get secret d8-cluster-configuration \
       -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d \
       | sed -n 's/^clusterDomain:[[:space:]]*//p' | tr -d "\"'")
     
     d8 k patch moduleconfig control-plane-manager --type merge \
       -p "{\"spec\":{\"version\":3,\"settings\":{\"network\":{\"clusterDomain\":\"$DOMAIN\"}}}}"
     

     The value must match the one the cluster currently runs with; admission rejects a mismatch.

  3. Remove the field from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete clusterDomain, then save.

  Once the field is removed from ClusterConfiguration, this alert is resolved.

• D8ObsoleteEncryptionAlgorithmInClusterConfiguration CE S6
  Obsolete encryptionAlgorithm in ClusterConfiguration needs to be migrated to control-plane-manager ModuleConfig.

  The encryptionAlgorithm field is set to a non-default value in ClusterConfiguration but has not been migrated to ModuleConfig control-plane-manager yet.

  Until the migration is complete, the value from ClusterConfiguration is used to avoid unintended certificate rotation.

  To resolve this alert, migrate the setting:

  1. If ModuleConfig control-plane-manager does not exist, create it first:

     d8 s module enable control-plane-manager
     

  2. Migrate the value from ClusterConfiguration to ModuleConfig in one command:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF
     {"spec": {"version": 3, "settings": {"encryptionAlgorithm": "$(d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d | grep encryptionAlgorithm | awk '{print $2}')"}}}
     EOF
     )"
     

  Once ModuleConfig is updated, this alert will be resolved automatically.

  1. Remove encryptionAlgorithm from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete the encryptionAlgorithm field and save.

• D8ObsoleteGlobalControlPlaneResourcesRequests CE S6
  Obsolete global control-plane resource requests need to be migrated.

  Control-plane resource requests are being migrated from global settings to moduleconfig control-plane-manager.

  The obsolete settings are still used as a fallback, but they should be moved to the control-plane-manager module settings. The migration does not change the meaning of the values: cpu and memory remain the total resource requests for control-plane components on each master node.

  To complete the migration, follow the steps below:

  1. Review the current obsolete settings in moduleconfig global:

     d8 k get mc global -o jsonpath='{.spec.settings.modules.resourcesRequests.controlPlane}'; echo
     

     Example of obsolete settings:

     spec:
       settings:
         modules:
           resourcesRequests:
             controlPlane:
               cpu: 1000m
               memory: 500Mi
     

  2. Make sure the moduleconfig control-plane-manager resource exists and uses settings version 3.

     If the resource does not exist, create it:

     d8 s module enable control-plane-manager
     

     Set the settings version:

     d8 k patch mc control-plane-manager --type merge \
       -p '{"spec":{"version":3}}'
     

  3. Copy the obsolete values from moduleconfig global to moduleconfig control-plane-manager.

     The following command preserves the original JSON value types, including numeric CPU values:

     d8 k patch mc control-plane-manager --type merge -p "$(
       d8 k get mc global -o json | jq -c \
         '{"spec":{"version":3,"settings":{"resourcesRequests": .spec.settings.modules.resourcesRequests.controlPlane}}}'
     )"
     

     After migration, the moduleconfig control-plane-manager resource should contain:

     spec:
       version: 3
       settings:
         resourcesRequests:
           cpu: 1000m
           memory: 500Mi
     

  4. Check that the values were applied:

     d8 k get mc control-plane-manager -o jsonpath='{.spec.settings.resourcesRequests}'; echo
     

  5. Remove the obsolete settings from moduleconfig global:

     d8 k patch mc global --type merge \
       -p '{"spec":{"settings":{"modules":{"resourcesRequests":null}}}}'
     

• D8ObsoleteKubernetesVersionFieldInClusterConfiguration CE S6
  The obsolete kubernetesVersion field is still present in ClusterConfiguration.

  ClusterConfiguration still carries the obsolete kubernetesVersion field. The place to manage the cluster Kubernetes version is settings.kubernetesVersion in ModuleConfig control-plane-manager.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Read what the obsolete field currently holds:

     d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d | grep '^kubernetesVersion:'
     

  3. If ModuleConfig has no kubernetesVersion yet, set that same value there, so the cluster keeps the version it runs today (replace 1.34 with what you saw; if you saw Automatic, use Default):

     d8 k patch moduleconfig control-plane-manager --type merge -p '{"spec":{"version":3,"settings":{"kubernetesVersion":"1.34"}}}'
     

     Set Default instead if you want Deckhouse to choose the version. Note that this starts an upgrade whenever the release default is higher than the pinned version:

     d8 k patch moduleconfig control-plane-manager --type merge -p '{"spec":{"version":3,"settings":{"kubernetesVersion":"Default"}}}'
     

  4. Remove kubernetesVersion from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete the kubernetesVersion field and save.

  Once the kubernetesVersion field is removed from ClusterConfiguration, this alert is resolved.

• D8ObsoleteNetworkFieldsInClusterConfiguration CE S6
  Obsolete network fields are still present in ClusterConfiguration.

  ClusterConfiguration still carries one or more of the obsolete podSubnetCIDR, serviceSubnetCIDR, podSubnetNodeCIDRPrefix fields. The place to manage the cluster’s network is the network section of ModuleConfig control-plane-manager.

  This fires simply because a field is present, even if ModuleConfig control-plane-manager already overrides it - the goal is to remove the deprecated field, not just override it.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Migrate the values straight from the Secret into ModuleConfig using the following command:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(
       d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d \
         | yq -o=json '{"podSubnetCIDR": .podSubnetCIDR, "serviceSubnetCIDR": .serviceSubnetCIDR, "podSubnetNodeCIDRPrefix": .podSubnetNodeCIDRPrefix}' \
         | jq -c '{spec: {version: 3, settings: {network: with_entries(select(.value != null))}}}'
     )"
     

     Once a field is set in ModuleConfig this way, it cannot be migrated again: the network group is immutable once set.

  3. Remove all three fields from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete podSubnetCIDR, serviceSubnetCIDR and podSubnetNodeCIDRPrefix, then save.

  Once all three fields are removed from ClusterConfiguration, this alert is resolved.

• D8ObsoletePublishAPIinUserAuthn CE S6
  Obsolete Publish API settings in user-authn need to be migrated.

  Publish API ingress settings are being migrated from mc user-authn to mc control-plane-manager.

  To complete the migration, follow the steps below:

  1. Review current settings for publishAPI in the moduleconfig user-authn resource.

     Note that legacy user-authn spec.version: 1 uses publishAPI.enable instead of publishAPI.enabled. No manual rewrite is required: the command in step 4 uses the normalized payload from the temporary ConfigMap.

  2. If the moduleconfig control-plane-manager resource exists, update its spec to spec.version: 3. If settings.apiserver.loadBalancer is set, migrate it to settings.apiserver.publishAPI.loadBalancer. If the resource does not exist, proceed to step 3.

  3. If moduleconfig control-plane-manager does not exist, create it via d8:

     d8 s module enable control-plane-manager
     

     And set the settings version:

     d8 k patch moduleconfig control-plane-manager --type merge \
     -p '{"spec":{"version":3}}'
     

  4. Apply the migrated settings into moduleconfig control-plane-manager from temporary configMap by executing the following script:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF
     {"spec": {"settings": {"apiserver": {"publishAPI": {"ingress": \
     $(d8 k -n kube-system get cm d8-publishapi-config-migration -ojsonpath='{.data.config}')
        }}}}}
     EOF
     )"
     

  5. Remove obsolete spec.settings.publishAPI section from moduleconfig user-authn.

• D8ProblematicFeatureGateInUse CE S9
  Feature gate XXX is STATUS_REFERENCE in current Kubernetes version XXX and should be removed from the configuration. Is not using in the cluster control-plane.

  The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it is STATUS_REFERENCE in the current Kubernetes version XXX.

  XXX This feature gate does not exist in any component (apiserver, kube-controller-manager, kube-scheduler, kubelet) and is not being applied to the cluster. XXX

  Recommended actions:

  1. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:

  d8 k edit mc control-plane-manager
  

  For more information about Kubernetes feature gates, refer to the official documentation.

• D8UsingFeatureGateWillBeDeprecated CE S9
  Feature gate XXX will be deprecated in Kubernetes version XXX.

  The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it will be deprecated in Kubernetes version XXX.

  Current Kubernetes version: XXX

  Recommended actions:

  1. Review the usage of this feature gate in your cluster.
  2. Plan migration away from this feature gate before upgrading to version XXX.
  3. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:

  d8 k edit mc control-plane-manager
  

  For more information about Kubernetes feature gates, refer to the official documentation.

• HelmReleasesHasResourcesWithDeprecatedVersions CE S5
  At least one Helm release contains resources with deprecated apiVersion, which will be removed in Kubernetes version VERSION_NUMBER.

  To list all affected resources, run the following Prometheus query:

  max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 1
  

  For more details on upgrading deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

  Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.

• HelmReleasesHasResourcesWithUnsupportedVersions CE S4
  At least one Helm release contains resources with unsupported apiVersion for Kubernetes version VERSION_NUMBER.

  To list all affected resources, run the following Prometheus query:

  max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 2
  

  For more details on migrating deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

  Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.

• K8SApiserverDown CE S3
  API servers can't be reached.

  No API servers are reachable, or they have all disappeared from service discovery.

• K8sCertificateExpiration CE S5
  Kubernetes has API clients with soon-to-expire certificates.

  Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than a day on node COMPONENT_NAME.

  To check control plane certificates, use kubeadm:

  1. Install kubeadm using the following command:

     apt install kubeadm=1.24.*
     

  2. Check certificates:

     kubeadm alpha certs check-expiration
     

  To check kubelet certificates, do the following on each node:

  1. Check kubelet configuration:

     ps aux \
       | grep "/usr/bin/kubelet" \
       | grep -o -e "--kubeconfig=\S*" \
       | cut -f2 -d"=" \
       | xargs cat
     

  2. Locate the client-certificate or client-certificate-data field.
  3. Check certificate expiration using OpenSSL.

  Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.

• K8sCertificateExpiration CE S6
  Kubernetes has API clients with soon-to-expire certificates.

  Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than 7 days on node NODE_NAME.

  To check control plane certificates, use kubeadm:

  1. Install kubeadm using the following command:

     apt install kubeadm=1.24.*
     

  2. Check certificates:

     kubeadm alpha certs check-expiration
     

  To check kubelet certificates, do the following on each node:

  1. Check kubelet configuration:

     ps aux \
       | grep "/usr/bin/kubelet" \
       | grep -o -e "--kubeconfig=\S*" \
       | cut -f2 -d"=" \
       | xargs cat
     

  2. Locate the client-certificate or client-certificate-data field.
  3. Check certificate expiration using OpenSSL.

  Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.

• K8SControllerManagerTargetDown CE S3
  Controller manager is down.

  There is no running kube-controller-manager. As a result, deployments and replication controllers are not progressing.

• K8SSchedulerTargetDown CE S3
  Scheduler is down.

  The Kubernetes scheduler is not running. As a result, new pods are not being assigned to nodes.

• KubeEtcdHighFsyncDurations CE S7
  Syncing (fsync) WAL files to disk is slow.

  In the last 15 minutes, the 99th percentile of the fsync duration for WAL files exceeded 0.5 seconds: VALUE.

  Possible causes:

  • High disk latency where etcd data is stored.
  • High CPU usage on the node.
• KubeEtcdHighNumberOfLeaderChanges CE S5
  The etcd cluster is re-electing the leader too frequently.

  There have been VALUE leader re-elections for the etcd cluster member running on node NODE_NAME in the last 10 minutes.

  Possible causes:

  • High disk latency where etcd data is stored.
  • High CPU usage on the node.
  • Degradation of network connectivity between cluster members in the multi-master mode.
• KubeEtcdInsufficientMembers CE S4
  Insufficient members in the etcd cluster.

  The etcd cluster has too few members, increasing the risk of failure if another member becomes unavailable. To resolve this issue, check the status of etcd Pods: bash d8 k -n kube-system get pod -l component=etcd

• KubeEtcdNoLeader CE S4
  The etcd cluster member running on node NODE_NAME has lost the leader.

  To resolve this issue, check the status of the etcd Pods: bash d8 k -n kube-system get pod -l component=etcd | grep NODE_NAME

• KubeEtcdTargetAbsent CE S5
  There is no etcd target in Prometheus.

  Steps to troubleshoot:

  1. Check the status of the etcd Pods:

     d8 k -n kube-system get pod -l component=etcd
     

  2. Review Prometheus logs:

     d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus
     

• KubeEtcdTargetDown CE S5
  Prometheus is unable to scrape etcd metrics.

  Steps to troubleshoot:

  1. Check the status of the etcd Pods:

     d8 k -n kube-system get pod -l component=etcd
     

  2. Review Prometheus logs:

     d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus
     

• KubernetesServiceExternalIPsDeprecated CE S6
  The cluster has Services that use the deprecated spec.externalIPs field.

  There are VALUE Service(s) in user namespaces that use the spec.externalIPs field. Services in the d8-* and kube-system namespaces are not checked.

  The spec.externalIPs field is officially deprecated in Kubernetes because of the CVE-2020-8554 security vulnerability. Starting with Kubernetes 1.36, the API server returns a warning for every request that sets this field. Support for the field will be removed in future Kubernetes versions.

  Plan the migration of the affected Services to a supported way of exposing them, for example:

  • A Service of the LoadBalancer type. In bare-metal clusters, use the metallb module to assign addresses.
  • A Service of the NodePort type.
  • An Ingress or Gateway API resource for HTTP(S) traffic.

  For details, refer to the Kubernetes announcement.

  To list the affected Services with their namespaces and external IP addresses, run:

  d8 k get services -A -o json | jq -r '["NAMESPACE", "NAME", "TYPE", "EXTERNAL-IPS"], (.items[] | select(.spec.externalIPs) | select(.metadata.namespace | test("^(d8-.*|kube-system)$") | not) | [.metadata.namespace, .metadata.name, .spec.type, (.spec.externalIPs | join(","))]) | @tsv' | column -t
  

  Alternatively, run the following Prometheus query:

  max by (namespace, service, external_ip) (kube_service_spec_external_ip{namespace!~"d8-.*|kube-system"})
  

Модуль deckhouse

• ApplicationIsInMaintenanceMode CE S6
  Application NAME in namespace NAMESPACE is running in maintenance mode.

  Application NAME in namespace NAMESPACE is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

  To switch the application back to normal mode, edit the corresponding Application resource with the following command:

  d8 k patch application NAME -n NAMESPACE --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
  

• D8CNIEnabledMoreThanOne CE S2
  More than one CNI is enabled in the cluster.

  Deckhouse has detected that multiple CNIs are enabled in the cluster. For the cluster to work correctly, exactly one CNI must be enabled.

  To resolve this issue, follow these steps:

  1. List the CNI modules and their enabled state:

     d8 k get moduleconfig -o custom-columns='NAME:.metadata.name,ENABLED:.spec.enabled' | grep -E 'NAME|cni-'
     

  2. Decide which CNI the cluster must keep. Check which one actually serves the nodes before disabling anything:

     d8 k -n d8-cni-cilium get pods 2>/dev/null; d8 k -n d8-cni-flannel get pods 2>/dev/null; d8 k -n d8-cni-simple-bridge get pods 2>/dev/null
     

  3. Disable every other CNI module, one at a time:

     d8 k patch moduleconfig CNI_MODULE_NAME --type=merge -p '{"spec":{"enabled":false}}'
     

  Changing the CNI of a running cluster disrupts pod networking. If you are unsure which CNI to keep, contact support before disabling anything.

• D8CNIMisconfigured CE S5
  The parameters specified in ModuleConfig of CNI_NAME do not match the ones that are actually being used in the cluster.

  This happened because there were several sources for configuring CNI parameters in the cluster, and ModuleConfig did not have the highest priority previously.

  To resolve this issue, the following steps should be taken:

  1. Find the ConfigMap d8-system/desired-cni-moduleconfig in the cluster, which contains the actual ModuleConfig settings.

     d8 k -n d8-system get configmap desired-cni-moduleconfig -o yaml
     

  2. Apply this prepared ModuleConfig to the cluster. This will not cause any actual reconfigurations in the cluster and is completely safe.

  3. Once the parameters in “ModuleConfig” match those used in the cluster:
     • this alert will be resolved,
     • and the priority of the sources for configuring CNI will change, and ModuleConfig will become the main source of truth.
  4. After that, you can add the desired parameters to the ModuleConfig CNI and they will be applied immediately.
• D8DeckhouseConfigInvalid CE S5
  Deckhouse rejected the cluster configuration: one of the ModuleConfig resources is invalid.

  Deckhouse has detected errors in the cluster configuration. The usual causes are spec.settings that do not match the module’s settings schema, and a failed conversion of spec.settings to the latest schema version.

  This alert does not name the module, because the underlying metric has no labels. Deckhouse puts the exact error into the status.message field of the ModuleConfig that caused it, prefixed with Error:.

  To resolve this issue, follow these steps:

  1. Find the affected configuration. The MESSAGE column holds the error and the VERSION column holds the settings schema version currently in use:

     d8 k get moduleconfig
     

  2. Read the full error for the ModuleConfig that reports one (global is a valid value for MODULE_NAME):

     d8 k get moduleconfig MODULE_NAME -o jsonpath='{.status.message}{"\n"}'
     

  3. Fix spec.settings, or spec.version if the error mentions a settings version or a conversion:

     d8 k edit moduleconfig MODULE_NAME
     

  4. If no ModuleConfig reports an error, check the Deckhouse logs:

     d8 k -n d8-system logs -f -l app=deckhouse
     

• D8DeckhouseCouldNotDeleteModule CE S4
  Deckhouse is unable to delete the MODULE_NAME module.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotDiscoverModules CE S4
  Deckhouse is unable to discover modules.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunGlobalHook CE S5
  Deckhouse is unable to run the HOOK_NAME global hook.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunModule CE S4
  Deckhouse is unable to start the MODULE_NAME module.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunModuleHook CE S7
  Deckhouse is unable to run the MODULE_NAME/HOOK_NAME module hook.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCustomTargetDown CE S4
  Prometheus is unable to scrape custom metrics generated by Deckhouse hooks.

  Prometheus cannot reach the endpoint that serves metrics exported by Deckhouse hooks. Deckhouse core metrics may still be scraped, so this usually points at the hook metrics endpoint rather than at Deckhouse being down.

  To investigate the issue, follow these steps:

  1. Check that the pod is up:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Look for hook failures in the logs, since a hook that never completes exports no metrics:

     d8 k -n d8-system logs -l app=deckhouse --tail=100 | grep -i error
     

  3. Check that the scrape configuration for hook metrics is still present:

     d8 k -n d8-monitoring get podmonitor deckhouse -o jsonpath='{range .spec.podMetricsEndpoints[*]}{.path}{"\n"}{end}'
     

• D8DeckhouseDeprecatedConfigmapManagedByArgoCD CE S4
  Deprecated Deckhouse ConfigMap managed by Argo CD.

  The Deckhouse ConfigMap is no longer used.

  To resolve this issue, remove the d8-system/deckhouse ConfigMap from Argo CD.

• D8DeckhouseGlobalHookFailsTooOften CE S9
  The HOOK_NAME Deckhouse global hook is crashing too frequently.

  The HOOK_NAME hook has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseHasNoAccessToRegistry CE S7
  Deckhouse is unable to connect to the registry.

  Deckhouse can’t connect to the registry to check for new releases. With a release channel set, it checks the channel every 3 minutes; without one, it checks the digest of its own image every 15 seconds. Without access to the registry, automatic updates are unavailable.

  Module images come from the same registry, so pods that need a new image fail to start as well, sometimes with errors that do not mention the registry. The usual causes are expired or changed credentials, a registry certificate that has expired or was replaced with one signed by an unknown authority, and a network or proxy change.

  To investigate the issue, follow these steps:

  1. Read the registry error in the logs. UNAUTHORIZED points at the credentials, x509 at the certificate, and a timeout or a refused connection at the network:

     d8 k -n d8-system logs -l app=deckhouse --tail=500 | grep -iE 'check deckhouse release|registry'
     

  2. Check which registry address the cluster is configured to use:

     d8 k -n d8-system get secret deckhouse-registry -o jsonpath='{.data.address}' | base64 -d
     

  3. If the address is correct, the credentials are the next suspect. Check when the secret was last updated:

     d8 k -n d8-system get secret deckhouse-registry
     

  4. If the error mentions x509, check who issued the registry certificate and when it expires. Run the command on a master node and substitute the host from the registry address, and its port if it is not 443:

     openssl s_client -connect REGISTRY_HOST:443 -servername REGISTRY_HOST </dev/null 2>/dev/null | openssl x509 -noout -issuer -enddate
     

  5. Check that the node running Deckhouse can reach the registry at all, for example through a proxy or a firewall rule that changed.

• D8DeckhouseIsHung CE S4
  Deckhouse is down.

  Deckhouse is probably down, since the deckhouse_live_ticks metric in Prometheus has stopped increasing. This metric is expected to increment every 10 seconds.

  To investigate the issue, follow these steps:

  1. Check the pod state and its restart count:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the logs. A deadlock or a hook that never returns stops the ticks while the pod still looks healthy:

     d8 k -n d8-system logs -l app=deckhouse --tail=200
     

  3. Check whether a queue stopped moving at the same time:

     d8 system queue list
     

  4. If the pod is alive but stuck, restart it:

     d8 k -n d8-system rollout restart deploy/deckhouse
     

• D8DeckhouseIsNotOnReleaseChannel CE S9
  Deckhouse isn't subscribed to any regular release channels.

  Deckhouse in this cluster isn’t subscribed to any of the regular release channels: Alpha, Beta, EarlyAccess, Stable, or RockSolid.

  To resolve this issue, follow these steps:

  1. Check the current release channel used in the cluster:

     d8 k -n d8-system  get deploy deckhouse -o json | jq '.spec.template.spec.containers[0].image' -r
     

  2. Subscribe to one of the regular release channels by adjusting the deckhouse module configuration.

• D8DeckhouseModuleHookFailsTooOften CE S9
  The MODULE_NAME/HOOK_NAME Deckhouse hook is crashing too frequently.

  The HOOK_NAME hook of the MODULE_NAME module has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseModuleUpdatePolicyNotFound CE S5
  Module update policy not found for MODULE_RELEASE.

  The module update policy for MODULE_RELEASE is missing.

  To resolve this issue, remove the label from the module release using the following command:

  d8 k label mr MODULE_RELEASE modules.deckhouse.io/update-policy-
  

  A new suitable policy will be detected automatically.

• D8DeckhouseModuleValidationError CE S5
  Module configuration failed for module MODULE_NAME.

  Initial config for module MODULE_NAME is not valid.

  You can get more details via

  d8 k get mr -l module=MODULE_NAME
  

  Provided error: XXX

• D8DeckhouseObsoleteMonitoringDeckhouseModuleConfig CE S9
  Obsolete monitoring-deckhouse ModuleConfig should be removed.

  The monitoring-deckhouse module was merged into the deckhouse module, but its ModuleConfig still exists in the cluster and is no longer used.

  Remove it manually:

  d8 k delete moduleconfig monitoring-deckhouse
  

  If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.

• D8DeckhousePodIsNotReady CE S4
  The Deckhouse Pod is NOT Ready.

  The Deckhouse pod is running but has not become Ready. While this lasts, Deckhouse is not converging the cluster: module updates and configuration changes are not applied.

  To investigate the issue, follow these steps:

  1. Check the pod state and its restart count:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the events and probe failures at the bottom of the pod description:

     d8 k -n d8-system describe pod -l app=deckhouse
     

  3. Check the logs. A readiness probe that never succeeds is most often a startup hook that fails or hangs:

     d8 k -n d8-system logs -l app=deckhouse --tail=200
     

  4. Check whether a queue is stuck, which keeps Deckhouse from reporting Ready:

     d8 system queue list
     

• D8DeckhousePodIsNotRunning CE S4
  The Deckhouse Pod is NOT Running.

  There is no running Deckhouse pod in the d8-system namespace. Nothing is reconciling the cluster: no module updates, no configuration changes, and no Deckhouse releases are applied.

  To investigate the issue, follow these steps:

  1. Check the pod state. Pending points at scheduling, ImagePullBackOff at the registry, and CrashLoopBackOff at Deckhouse failing to start:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the events to find out why the pod is not running:

     d8 k -n d8-system describe pod -l app=deckhouse
     

  3. If the pod is restarting, read the logs of the previous run:

     d8 k -n d8-system logs -l app=deckhouse --previous --tail=200
     

  4. Check that the Deployment itself is present and has the replicas it expects:

     d8 k -n d8-system get deploy deckhouse
     

• D8DeckhousePodIsRestartingTooOften CE S9
  Excessive Deckhouse restarts detected.

  Number of restarts in the last hour: VALUE.

  Excessive Deckhouse restarts indicate a potential issue. Normally, Deckhouse should be up and running continuously.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHung CE S6
  Module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.

  The module queue head has remained unchanged for more than 10 minutes.

  Affected queue:

  • Module: MODULE_NAME
  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHungCritical CE S4
  Critical module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.

  The critical module queue head has remained unchanged for more than 10 minutes.

  Affected queue:

  • Module: MODULE_NAME
  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHungGlobal CE S4
  Global/Parallel task XXX (hook HOOK_NAME) is hung.

  The global/parallel task head has remained unchanged for more than 10 minutes.

  Affected task:

  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseSelfTargetAbsent CE S4
  There is no Deckhouse target in Prometheus.

  Prometheus has no deckhouse target at all. Unlike a target that is down, this means the target was never discovered, so the problem is usually with the Service, its endpoints, or the monitoring configuration rather than with Deckhouse itself.

  To investigate the issue, follow these steps:

  1. Check that the Deckhouse pod exists and is running:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Check that the Service has endpoints. An empty ENDPOINTS column means nothing matches the selector:

     d8 k -n d8-system get endpoints deckhouse
     

  3. Check that the scrape configuration for Deckhouse exists:

     d8 k -n d8-system get servicemonitor,podmonitor -l app=deckhouse
     

  4. Check that the monitoring stack itself is healthy:

     d8 k -n d8-monitoring get pods
     

• D8DeckhouseSelfTargetDown CE S4
  Prometheus is unable to scrape Deckhouse metrics.

  Prometheus cannot reach the deckhouse metrics endpoint. Deckhouse itself may still be working, but its own metrics are missing, so most other Deckhouse alerts are blind while this lasts.

  To investigate the issue, follow these steps:

  1. Check that the pod is up and how long it has been running:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Look for scrape errors, such as a connection refused or a TLS failure:

     d8 k -n d8-system logs -l app=deckhouse --tail=100
     

  3. Check that the pod still exposes the self metrics port:

     d8 k -n d8-system get pod -l app=deckhouse -o jsonpath='{range .items[*].spec.containers[*].ports[*]}{.name}{"="}{.containerPort}{"\n"}{end}'
     

• D8DeckhouseWatchErrorOccurred CE S5
  Possible API server connection error in the client-go informer.

  Deckhouse has detected an error in the client-go informer, possibly due to connection issues with the API server.

  Steps to investigate:

  1. Check Deckhouse logs for more information by running:

     d8 k -n d8-system logs deploy/deckhouse | grep error | grep -i watch
     

  2. This alert attempts to detect a correlation between the faulty snapshot invalidation and API server connection errors, specifically for the handle-node-template hook in the node-manager module.

     To compare the snapshot with the actual node objects for this hook, run the following command:

     diff -u <(d8 k get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'|sort) <(d8 k -n d8-system exec svc/deckhouse-leader -c deckhouse -- deckhouse-controller module snapshots node-manager -o json | jq '."040-node-manager/hooks/handle_node_templates.go"' | jq '.nodes.snapshot[] | .filterResult.Name' -r | sort)
     

• D8HasModuleConfigAllowedToDisable CE S4
  The ModuleConfig annotation for allowing module disabling is set.

  The ModuleConfig is pending module disabling.

  It is recommended that you keep your module configurations clean by removing unnecessary approval annotations.

  If you ignore this alert and do not clear the annotation, the module may be accidentally removed from the cluster, potentially leading to irreversible consequences.

  To resolve this issue and stop the alert, run the following command:

  d8 k annotate moduleconfig MODULE_NAME modules.deckhouse.io/allow-disabling-
  

  With Module v2 enabled, remove the packages.deckhouse.io/allow-disabling annotation instead:

  d8 k annotate moduleconfig MODULE_NAME packages.deckhouse.io/allow-disabling-
  

• D8ModuleOutdatedByMajorVersion CE S9
  Module MODULE_NAME is running XXX major versions behind.

  Module MODULE_NAME is running XXX major versions behind on major version releases. Major version updates may contain breaking changes and require immediate attention and careful planning.

  To check the module releases status, run the following command:

  d8 k get mr -l module=MODULE_NAME
  ```XXXXXX
  
  This major release has from-to version constraints that must be met before applying the update.
  To check the update instructions (from-to versions), run the following command:
  
  ```bash
  d8 k get mr NAME -o jsonpath='{.spec.update.versions[*]}'
  

  This will show you the update paths including the from-to versions defined in the release.XXX

  To approve the module major release update, run the following command:

  bash d8 k annotate mr NAME modules.deckhouse.io/approved="true" XXX

• D8NodeHasDeprecatedOSVersion CE S4
  Nodes with deprecated OS versions detected.

  Deckhouse has detected nodes running deprecated OS versions.

  Steps to troubleshoot:

  1. Get a list of affected nodes by running the following Prometheus query:

     kube_node_info{os_image=~"Ubuntu 18.04.*|Debian GNU/Linux 10.*|CentOS Linux 7.*"}
     

  2. Update the affected nodes to a supported OS version.

• DeckhouseEditionNotFound CE S6
  Incorrect Deckhouse Platform edition.

  Specify a Deckhouse Platform edition in MODULE_NAME ModuleConfig.

  d8 k patch moduleconfig MODULE_NAME --type merge --patch '{"spec": {"settings": {"license": {"edition": "DKP_EDITION"}}}}'
  

  Where DKP_EDITION is the edition you want to use: CE, BE, EE, SE, or SE-plus.

• DeckhouseHighMemoryUsage CE S3
  Deckhouse memory usage is too high.

  Deckhouse pod POD_NAME on node NODE_NAME has high memory usage.

  Please, if possible, get the dumps for debugging purposes (it will take around 30 seconds):

  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/heap" -o /tmp/heap.pprof
  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/goroutine" -o /tmp/goroutine.pprof
  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/profile?seconds=30" -o /tmp/cpu.pprof
  

  and send these files (/tmp/heap.pprof, /tmp/cpu.pprof and /tmp/goroutine.pprof) to the support team.

• DeckhouseMigratedModuleNotDownloaded CE S6
  Migrated module MODULE_NAME has not been downloaded yet.

  Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and is available in a ModuleSource registry, but it has not been downloaded yet (no deployed ModuleRelease exists). DeckhouseRelease will be locked until the module is downloaded.

  This alert is usually temporary while the module source controller downloads the module from the registry. If the alert persists, this may indicate:

  • The module’s update policy or ModuleSource is misconfigured, preventing the module from being downloaded.
  • A network issue is preventing the module from being downloaded.

  To investigate the issue, do the following:

  • Check the ModuleRelease objects for the module:

    d8 k get mr -l module=MODULE_NAME
    

  • Check the configured ModuleSources:

    d8 k get modulesources
    

  • Check the deckhouse logs for more details:

    d8 k -n d8-system logs -f -l app=deckhouse
    

• DeckhouseMigratedModuleNotFound CE S6
  Migrated module MODULE_NAME was not found in any ModuleSource registry.

  Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and was not found in any available ModuleSource registry. DeckhouseRelease will be locked until the module becomes available in one of the configured ModuleSource registries.

  This may indicate:

  • A ModuleSource registry is misconfigured.
  • The module has not been published to the registry yet.
  • A network issue is preventing access to the registry.

  To investigate the issue, do the following:

  • Check the configured ModuleSources:

    d8 k get modulesources
    

  • List the available module sources:

    d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get sources
    

  • List the modules available in a specific source:

    d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get modules source_name
    

  • Check the deckhouse logs for more details:

    d8 k -n d8-system logs -f -l app=deckhouse
    

• DeckhouseModuleUpdatingFailedBrokenSequence CE S4
  Module update has failed. Updating sequence is broken.

  The ‘MODULE_NAME’ module update has failed.

  Current version: XXX. Desired version: VERSION_NUMBER.

  Attempt to download interim releases failed. Deckhouse updates modules one minor version at a time, so every intermediate version must exist in the registry.

  Possible reasons:

  • Some of the necessary versions of the module image are not available in the registry.
  • The registry is unreachable.

  To resolve this issue, follow these steps:

  1. Check which module releases exist and where the sequence stopped:

     d8 k get modulereleases -l module=MODULE_NAME
     

  2. Read the reason recorded on the release that is not deployed:

     d8 k get modulereleases -l module=MODULE_NAME -o custom-columns='NAME:.metadata.name,VERSION:.spec.version,PHASE:.status.phase,MESSAGE:.status.message'
     

  3. Find the module source that serves this repo. The registry label above is a repo URL, not a source name:

     d8 k get modulesource -o custom-columns='NAME:.metadata.name,REPO:.spec.registry.repo'
     

  4. Ensure that every minor version of the ‘MODULE_NAME’ module image between XXX and VERSION_NUMBER is available in the registry ‘XXX’. If the cluster uses a mirror, the missing versions have to be pulled into it first.

• DeckhouseModuleUpdatingFailedModuleIsNotValid CE S4
  Module update has failed. Module is not valid.

  The ‘MODULE_NAME’ module update has failed.

  Desired version: VERSION_NUMBER.

  Possible reasons:

  • The module image is corrupted.

  To resolve this issue, ensure that the VERSION_NUMBER image in the registry ‘XXX’ is valid.

  Check which versions of the ‘MODULE_NAME’ module are available in the registry:

  d8 cr ls XXX/MODULE_NAME/release
  

  For a private registry, pass the credentials from the corresponding ModuleSource:

  d8 cr ls -u USERNAME -p PASSWORD XXX/MODULE_NAME/release
  

  To get the exact error, check Deckhouse logs: d8 k -n d8-system logs deploy/deckhouse | grep "failed to download module"

• DeckhouseReleaseDisruptionApprovalRequired CE S4
  Deckhouse release disruption approval required.

  The new Deckhouse release includes a disruptive update that requires manual approval.

  To check the details, run the following command:

  d8 k describe DeckhouseRelease NAME
  

  To approve the disruptive update, run the following command:

  d8 k annotate DeckhouseRelease NAME release.deckhouse.io/disruption-approved=true
  

• DeckhouseReleaseIsBlocked CE S5
  Deckhouse release requirements haven't been met.

  The requirements for the Deckhouse release haven’t been met.

  To check the details, run the following command:

  d8 k describe DeckhouseRelease NAME
  

• DeckhouseReleaseIsWaitingManualApproval CE S3
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseIsWaitingManualApproval CE S6
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseIsWaitingManualApproval CE S9
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseNotificationNotSent CE S4
  Deckhouse release notification webhook hasn't been sent.

  The Deckhouse release notification webhook failed to send.

  To check the notification webhook address, run the following command:

  d8 k get mc deckhouse -o yaml
  

• DeckhouseUpdating CE S4

  Deckhouse is being updated to XXX.

• DeckhouseUpdatingFailed CE S4
  Deckhouse update has failed.

  The Deckhouse update to version VERSION_NUMBER has failed.

  Possible reasons:

  • The next minor or patch version of the Deckhouse image is not available in the registry.
  • The Deckhouse image is corrupted.
  • Deckhouse cannot reach the registry.

  To resolve this issue, follow these steps:

  1. Check the state of the releases. The one stuck in Pending is the one that failed:

     d8 k get deckhousereleases
     

  2. Read the reason recorded on that release:

     d8 k get deckhousereleases -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'
     

  3. Read the corresponding error in the logs:

     d8 k -n d8-system logs -l app=deckhouse --tail=200 | grep -i release
     

  4. Ensure that version VERSION_NUMBER of the Deckhouse image is present in the registry. If the cluster uses a mirror, the missing version has to be pulled into it first.

• ModuleAtConflict CE S4
  Conflict detected for module MODULE_NAME.

  Deckhouse has detected conflicting sources for the MODULE_NAME module: it is available from more than one ModuleSource and no source is selected, so the module is not deployed.

  To resolve this issue, follow these steps:

  1. List the sources that provide the module:

     d8 k get module MODULE_NAME -o jsonpath='{.properties.availableSources}{"\n"}'
     

  2. Compare what each source offers, so you pick the one you actually want:

     d8 k get modulerelease -l module=MODULE_NAME
     

  3. Select the source in the module configuration:

     d8 k patch moduleconfig MODULE_NAME --type=merge -p '{"spec":{"source":"SOURCE_NAME"}}'
     

  4. Verify that the module has left the Conflict phase. The SOURCE and PHASE columns show the result:

     d8 k get module MODULE_NAME
     

• ModuleConfigObsoleteVersion CE S4
  ModuleConfig NAME uses obsolete settings version VERSION_NUMBER, the latest one is XXX.

  Deckhouse has detected that ModuleConfig NAME has spec.version: VERSION_NUMBER, while the latest settings schema version of the NAME module is XXX.

  Deckhouse converts spec.settings from version VERSION_NUMBER to version XXX on the fly, so the module keeps working, but the stored configuration remains outdated. Support for version VERSION_NUMBER may be dropped in a future release.

  To resolve this issue, follow these steps:

  1. Review the migration notes for the NAME module settings, one line per conversion step:

     d8 k get modulesettingsdefinition NAME -o jsonpath='{range .spec.versions[*].conversions[*]}{.descriptions.en}{"\n"}{end}'
     

  2. Migrate spec.settings accordingly and set spec.version to XXX:

     d8 k edit moduleconfig NAME
     

  3. Verify that the version has been updated and no warnings are left:

     d8 k get moduleconfig NAME -o jsonpath='{.spec.version}{"\t"}{.status.message}{"\n"}'
     

• ModuleHasDeprecatedUpdatePolicy CE S4
  Module MODULE_NAME is matched by the deprecated update policy UPDATE_POLICY_NAME.

  Deckhouse has detected that the module MODULE_NAME is using the deprecated update policy UPDATE_POLICY_NAME. The v1alpha1 policy has a selector that no longer functions.

  To specify the update policy in the module configuration, run the following command:

  d8 k patch moduleconfig MODULE_NAME --type='json' -p='[{"op": "add", "path": "/spec/updatePolicy", "value": "UPDATE_POLICY_NAME"}]'
  

  After resolving all alerts related to the update policy UPDATE_POLICY_NAME, clear the selector by running the following command:

  d8 k patch moduleupdatepolicies.v1alpha1.deckhouse.io UPDATE_POLICY_NAME --type='json' -p='[{"op": "replace", "path": "/spec/moduleReleaseSelector/labelSelector/matchLabels", "value": {"": ""}}]'
  

• ModuleIsDeprecated CE S9
  Module MODULE_NAME is deprecated.

  The module MODULE_NAME is deprecated and will not receive new updates. The module may be removed in a future release, so disable it in a controlled manner rather than waiting for it to disappear.

  To resolve this issue, follow these steps:

  1. Check the module state before changing anything. The STAGE column reads Deprecated, and the conditions show whether anything still runs:

     d8 k get module MODULE_NAME
     

  2. Find out whether other modules declare a dependency on it:

     d8 k get modules -o json | jq -r '.items[] | select(.properties.requirements.modules | has("MODULE_NAME")) | .metadata.name'
     

  3. Once nothing depends on it and its workloads are no longer needed, disable the module:

     d8 k patch moduleconfig MODULE_NAME --type=merge -p '{"spec":{"enabled":false}}'
     

• ModuleIsInMaintenanceMode CE S6
  Module MODULE_NAME is running in maintenance mode.

  Module MODULE_NAME is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

  To switch the module back to normal mode, edit the corresponding ModuleConfig resource with the following command:

  d8 k patch moduleconfig MODULE_NAME --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
  

• ModuleReleaseIsBlockedByRequirements CE S6
  A new release for module MODULE_NAME has been blocked due to unmet requirements.

  A new release for module MODULE_NAME has been blocked because it doesn’t meet the required conditions.

  To check the requirements, run the following command:

  d8 k  get mr NAME -o json | jq .spec.requirements
  

• ModuleReleaseIsOutdated CE S5
  Module MODULE_NAME is XXX or more minor versions behind the latest available release.

  Module MODULE_NAME is running XXX or more minor versions behind the latest available release. This module is significantly outdated and may have compatibility issues.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsOutdated CE S7
  Module MODULE_NAME is 2 minor versions behind the latest available release.

  Module MODULE_NAME is running 2 minor versions behind the latest available release. Updating is recommended to maintain compatibility and access to latest features.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsOutdated CE S9
  Module MODULE_NAME is 1 minor version behind the latest available release.

  Module MODULE_NAME is running 1 minor version behind the latest available release. Consider updating to maintain compatibility and security.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsWaitingManualApproval CE S6
  A new release for module MODULE_NAME is awaiting manual approval.

  A new release for module MODULE_NAME is available but requires manual approval before it can be applied.

  To approve the module release, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

Модуль descheduler

Модуль documentation

• ModuleConfigDeprecated CE S9
  Deprecated deckhouse-web ModuleConfig detected.

  The deckhouse-web module has been renamed to documentation, and a new documentation ModuleConfig is generated automatically.

  Steps to troubleshoot:

  1. Remove the deprecated deckhouse-web ModuleConfig from the CI deployment process.

  2. Delete it using the following command:

     d8 k delete mc deckhouse-web
     

• ModuleDocumentationRenderError CE S6
  Documentation rendering failed for module MODULE_NAME.

  An error occurred while rendering documentation for the MODULE_NAME module. The broken module was removed from the documentation site during Hugo build.

  Check the ModuleDocumentation resource status for details:

  d8 k get moduledocumentations MODULE_NAME -o yaml
  

  Check docs-builder logs for more information:

  d8 k -n d8-system logs -l app=docs-builder | grep -i "removed broken module"
  

Модуль istio

• D8IstioActiveWaypointsWithAmbientDisabled EE S6
  Active Istio waypoint exists while ambient mesh is disabled.

  Deckhouse has detected an active waypoint Deployment DEPLOYMENT_NAME in namespace NAMESPACE, created for WaypointInstance XXX, while ambient mesh is disabled.

  This usually happens if ambient mesh was enabled, a WaypointInstance was created, and then ambient mesh was disabled. In this state the waypoint-controller is not running, so it cannot reconcile or clean up WaypointInstance resources.

  To inspect affected resources, run:

  d8 k -n NAMESPACE get waypointinstance XXX -o yaml
  d8 k -n NAMESPACE get deploy,svc,sa,gateway,pdb,hpa,vpa -l istio.deckhouse.io/waypoint-instance=XXX
  

  Recommended remediation:

  1. Temporarily enable ambient mesh.
  2. Wait until waypoint-controller starts.
  3. Delete the affected WaypointInstance or change configuration as needed.
  4. Wait until managed waypoint resources are reconciled as needed or removed.
  5. Disable ambient mesh again.

  If ambient mesh cannot be re-enabled, manually remove the managed waypoint resources and then remove the WaypointInstance finalizer only after verifying that no managed resources remain.

• D8IstioActualDataPlaneVersionNotEqualDesired EE S8
  There are pods with Istio data plane version VERSION_NUMBER, but desired version is XXX.

  There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, while the desired version is XXX. As a result, the Istio version will be changed after the pod is restarted.

  To resolve the issue, use the following cheat sheet:

  ### Namespace-wide configuration
  # `istio.io/rev=vXYZ`: Use a specific revision.
  # `istio-injection=enabled`: Use the global revision.
  d8 k get ns NAMESPACE --show-labels
  
  ### Pod-wide configuration
  d8 k -n NAMESPACE get pods -l istio.io/rev=DESIRED_VISION
  

• D8IstioActualVersionIsNotInstalled EE S4
  The control plane version for pods with injected sidecars isn't installed.

  There are pods in the NAMESPACE namespace with injected sidecars of version VERSION_NUMBER (revision REVISION_NUMBER), but the corresponding control plane version is not installed. As a result, these pods have lost synchronization with the state in Kubernetes.

  To resolve this issue, install the required control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

  To identify orphaned pods, run the following command:

  d8 k -n NAMESPACE get pods -l 'service.istio.io/canonical-name' -o json | jq --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
  

• D8IstioAdditionalControlplaneDoesntWork CE S4
  Additional control plane isn't working.

  Deckhouse has detected that the additional Istio control plane ISTIO_REVISION_LABEL isn’t working.

  As a result, sidecar injection for pods with ISTIO_REVISION_LABEL isn’t working as well.

  To check the status of the control plane pods, run the following command:

  d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL
  

• D8IstioAllianceAmbientGatewayAddressUnusable EE S6
  The ambient east-west gateway has no address peers can dial.

  The ambient east-west gateway is deployed in this cluster, but publishes no address peer clusters can dial: its Service is missing, it has no external address, no node is running a gateway pod, or every address it has is neither an IP address nor a usable DNS name. The metadata-exporter pod logs in d8-istio say which.

  Peer clusters keep using the addresses they fetched last, if any. Those that have none cannot reach ambient-mode workloads here.

  To fix, set alliance.ambientGateway.advertise to the IP address or DNS name peers should dial. Or make the load balancer assign an address (alliance.ambientGateway.loadBalancerClass, .serviceAnnotations). Or switch alliance.ambientGateway.inlet to NodePort and check that gateway pods are scheduled (.nodeSelector, .tolerations). Or, if ambient multicluster is not needed, switch all multicluster members back to sidecar mode.

• D8IstioAllianceDataPlaneNotConnected EE S4
  Data-plane connectivity check failed for XXX NAME.

  The data-plane health check of XXX NAME with uuid XXX has been failing for more than 5 minutes.

  This means that actual service mesh traffic between clusters is not flowing properly, even though the control plane metadata exchange may be working.

  Possible causes:

  • The remote cluster’s alliance-healthcheck pod is not running.
  • Istio sidecar injection is not working properly on the healthcheck pods.
  • The ingress gateway between clusters is misconfigured or unreachable.
  • mTLS or trust domain configuration issues between clusters.

  To check the status:

  d8 k get XXX NAME -o json | jq '.status.conditions[] | select(.type=="DataplaneConnectionReady")'
  

• D8IstioAllianceDiscoveryMetadataEndpointRequestedViaDeprecatedSubdomain EE S5
  A peer cluster requests private metadata using a deprecated subdomain.

  Remote XXX NAME, (UUID: XXX) requests private metadata from hostname XXX, that hostname uses the deprecated istio subdomain.

  Update the remote cluster XXX resource: set spec.metadataEndpoint to (publicDomainTemplate + istio-metadata)/metadata/.

  Migration steps on the remote cluster:

  1. Ensure DNS resolves the new target (publicDomainTemplate + istio-metadata).
  2. Update spec.metadataEndpoint to use the new domain name. Keep the path /metadata/ unchanged.
  3. Verify remote metadata fetch and alliance health after the change.
• D8IstioAllianceIngressGatewayAddressUnusable EE S6
  The ingress gateway for federation and multicluster has no address peers can dial.

  The ingress gateway that peer clusters reach this cluster through is deployed, but publishes no address they can dial: its Service is missing, it has no external address, or no node is running a gateway pod. The metadata-exporter pod logs in d8-istio say which.

  Peer clusters keep using the addresses they fetched last, if any. Those that have none cannot reach federated services or, over multicluster, sidecar-mode workloads here.

  To fix, set alliance.ingressGateway.advertise to the addresses peers should dial. Or make the load balancer assign an address (alliance.ingressGateway.loadBalancerClass, .serviceAnnotations). Or switch alliance.ingressGateway.inlet to NodePort and check that gateway pods are scheduled (.nodeSelector, .tolerations).

• D8IstioConfigAnalysisConflictingGateways EE S5
  Conflicting Istio Gateways detected (IST0145).

  The Istio configuration analyzer found conflicting Gateways in revision REVISION_NUMBER.

  Code: XXX (ConflictingGateways) Namespace: NAMESPACE Resource: XXX

  Two or more Gateways share the same selector, port, and host match. Ingress traffic may be routed unpredictably or return 404.

  Steps to resolve:

  1. List Gateways and compare spec.selector, server ports, and hosts for overlaps involving XXX.
  2. Deduplicate or narrow hosts/ports so each combination is unique.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0145/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ConflictingGateways",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisError EE S5
  Istio configuration analysis detected an error-level issue.

  The Istio configuration analyzer found an error in revision REVISION_NUMBER.

  Type: ERROR_TYPE (XXX) Namespace: NAMESPACE Resource: XXX

  Inspect the reported resource in namespace NAMESPACE.

  Documentation for analysis codes: https://istio.io/latest/docs/reference/config/analysis/

  List analysis issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{revision="REVISION_NUMBER",severity="Error"}
  

  Check analyzer pods and logs:

  d8 k -n d8-istio get pods -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER
  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisImageAutoWithoutInjection EE S5
  Pod uses image auto without Istio injection (IST0147).

  The Istio configuration analyzer found pods with image: auto that are not targeted for sidecar injection in revision REVISION_NUMBER.

  Code: XXX (ImageAutoWithoutInjectionError) Namespace: NAMESPACE Resource: XXX

  image: auto is rewritten only when the Istio injector mutates the pod. Without injection the container image stays literally auto and the pod will not run correctly.

  Steps to resolve:

  1. Enable sidecar injection for the namespace or workload (istio-injection=enabled / istio.io/rev=REVISION_NUMBER, or a matching RevisionTag).
  2. Restart / recreate the affected pods after injection is enabled.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0147/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ImageAutoWithoutInjectionError",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisInvalidGatewayCredential EE S5
  Gateway TLS credential is invalid (IST0161).

  The Istio configuration analyzer found an invalid Gateway credential in revision REVISION_NUMBER.

  Code: XXX (InvalidGatewayCredential) Namespace: NAMESPACE Resource: XXX

  The Secret referenced by the Gateway for TLS is missing or does not contain a usable certificate/key. HTTPS listeners on that Gateway will fail.

  Steps to resolve:

  1. Open the Gateway XXX in namespace NAMESPACE and note credentialName / TLS secret references.
  2. Verify the Secret exists in the same namespace as the ingress Gateway workload and has valid tls.crt / tls.key (or the expected keys).
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0161/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="InvalidGatewayCredential",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisReferencedResourceNotFound EE S5
  Istio resource references a missing dependency (IST0101).

  The Istio configuration analyzer found a missing reference in revision REVISION_NUMBER.

  Code: XXX (ReferencedResourceNotFound) Namespace: NAMESPACE Resource: XXX

  An Istio object points to a Gateway, Service, ServiceEntry, or another dependency that does not exist (typo, wrong namespace, or resource not created yet). Traffic to that route may fail.

  Steps to resolve:

  1. Inspect the reported resource in namespace NAMESPACE and fix the broken reference (use FQDN for cross-namespace hosts, e.g. reviews.backend.svc.cluster.local).
  2. Confirm the target object exists.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0101/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ReferencedResourceNotFound",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisSchemaValidationError EE S5
  Istio resource failed schema validation (IST0106).

  The Istio configuration analyzer found a schema validation error in revision REVISION_NUMBER.

  Code: XXX (SchemaValidationError) Namespace: NAMESPACE Resource: XXX

  The resource does not match the Istio API schema and may be ignored or rejected by the control plane.

  Steps to resolve:

  1. Inspect XXX in namespace NAMESPACE and compare fields with the Istio API for your control plane version.
  2. Fix invalid or unknown fields, then re-apply the manifest.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0106/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="SchemaValidationError",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisWarning EE S4
  Istio configuration analysis detected a warning-level issue.

  The Istio configuration analyzer found a warning in revision REVISION_NUMBER.

  Type: ERROR_TYPE (XXX) Namespace: NAMESPACE Resource: XXX

  Inspect the reported resource in namespace NAMESPACE.

  Documentation for analysis codes: https://istio.io/latest/docs/reference/config/analysis/

  List analysis warnings in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{revision="REVISION_NUMBER",severity="Warning"}
  

  Check analyzer pods and logs:

  d8 k -n d8-istio get pods -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER
  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioCustomCAMaterialInvalid CE S4
  The Istio custom CA material is invalid but is being served to istiod.

  The Istio module is publishing CA material (XXX) that failed validation to istiod.

  The module never hard-blocks on module-owned or last-good material (doing so would rotate the mesh root and break mTLS cluster-wide), so it keeps serving this material with only a warning. However, istiod may reject it, which can prevent it from issuing or renewing workload certificates.

  Depending on the source:

  • cacerts — the d8-istio/cacerts Secret (or the CA reused from it) is not a valid CA. Inspect the Secret and restore valid ca-cert.pem/ca-key.pem (and root-cert.pem/cert-chain.pem if used).
  • secretRef:namespace/name — the last-good CA that was previously resolved from this ca.secretRef has become invalid (for example, its chain expired). Fix the referenced Secret so the next re-resolution (every 5 minutes) succeeds.

  Refer to the custom CA documentation.

• D8IstioCustomCASecretRefUnresolved CE S5
  The Istio custom CA ca.secretRef cannot be resolved.

  The Istio module could not re-resolve the configured ca.secretRef (XXX) and is keeping the last successfully-resolved CA.

  The mesh is not regressed — istiod keeps running on the exact CA material it already had — but any change made to the source Secret is not being picked up. Common causes:

  • the referenced Secret was deleted or renamed;
  • the referenced Secret was made malformed (not valid PEM, not a CA certificate, or a mismatched key pair);
  • a transient Kubernetes API problem.

  The module re-resolves the Secret every 5 minutes; this alert clears automatically once resolution succeeds. Inspect the referenced Secret and restore valid CA material.

  Refer to the custom CA documentation.

• D8IstioDataPlaneVersionMismatch EE S8
  There are pods with data plane version different from the control plane version.

  There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, which is different from the control plane version DESIRED_VERSION.

  Steps to resolve the issue:

  1. Restart affected pods and use the following PromQL query to get a full list:

     max by (namespace, dataplane_pod) (d8_istio_dataplane_metadata{full_version="VERSION_NUMBER"})
     

  2. Use the automatic Istio data plane upgrade.

• D8IstioDataPlaneWithoutIstioInjectionConfigured EE S4
  Detected pods with Istio sidecars but istio-injection isn't configured.

  There are pods in the NAMESPACE namespace with Istio sidecars, but istio-injection isn’t configured. As a result, these pods will lose their Istio sidecars after being recreated.

  To identify the affected pods, run the following command:

  d8 k -n NAMESPACE get pods -o json | jq -r --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
  

• D8IstioDeprecatedIstioVersionInstalled CE
  The installed Istio version has been deprecated.

  Deckhouse has detected that a deprecated Istio version VERSION_NUMBER is installed.

  Support for this version will be removed in upcoming Deckhouse releases. The higher the alert severity, the greater the probability of support being discontinued.

  To learn how to upgrade Istio, refer to the upgrade guide.

• D8IstioDesiredVersionIsNotInstalled EE S6
  Desired control plane version isn't installed.

  There is a desired Istio control plane version XXX (revision REVISION_NUMBER) configured for pods in the NAMESPACE namespace, but that version isn’t installed. As a result, pods can’t be recreated in the NAMESPACE namespace.

  To resolve this issue, install the desired control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

  Use the following cheat sheet:

  ### Namespace-wide configuration
  # `istio.io/rev=vXYZ`: Use a specific revision.
  # `istio-injection=enabled`: Use the global revision.
  d8 k get ns NAMESPACE --show-labels
  
  ### Pod-wide configuration
  d8 k -n NAMESPACE get pods -l istio.io/rev=REVISION_NUMBER
  

• D8IstioFederationMetadataEndpointDoesntWork EE S6
  IstioFederation FEDERATION_NAME gets no usable metadata from the remote cluster.

  The Deckhouse hook could not get usable metadata from the endpoint ENDPOINT_NAME for IstioFederation FEDERATION_NAME.

  The reason and the details are in the PublicMetadataExchangeReady and PrivateMetadataExchangeReady conditions of the IstioFederation. The endpoint may respond normally while the metadata is still unusable, for example, when the ingress gateway of the remote cluster has no address yet.

  To view the conditions, run the following command:

  d8 k get istiofederation FEDERATION_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  To reproduce the request to the public endpoint, run the following command:

  curl ENDPOINT_NAME
  

  To reproduce the request to private endpoints (run from the Deckhouse pod), run the following:

  KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
  LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
  REMOTE_CLUSTER_UUID="$(d8 k get istiofederation FEDERATION_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
  TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-federation --ttl 1h)"
  curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME
  

• D8IstioFederationRemoteClusterUUIDMismatch EE S4
  IstioFederation peer cluster UUID changed.

  Public metadata from ENDPOINT_NAME for IstioFederation FEDERATION_NAME reports cluster UUID XXX, but the previously pinned UUID is XXX.

  Deckhouse stops private metadata exchange and does not send a federation JWT until the peer identity is confirmed again.

  Check whether spec.metadataEndpoint was pointed to another cluster or the remote endpoint was replaced.

  To inspect the resource:

  d8 k get istiofederation FEDERATION_NAME -o yaml
  curl ENDPOINT_NAME
  

  If the peer change is intentional, recreate the IstioFederation resource so a new UUID can be pinned.

• D8IstioGlobalControlplaneDoesntWork CE S4
  Global control plane isn't working.

  Deckhouse has detected that the global Istio control plane ISTIO_REVISION_LABEL isn’t working.

  As a result, sidecar injection for pods with global revision isn’t working as well, and the validating webhook for Istio resources is absent.

  To check the status of the control plane pods, run the following command:

  d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL
  

• D8IstioMulticlusterMetadataEndpointDoesntWork EE S6
  IstioMulticluster MULTICLUSTER_NAME gets no usable metadata from the remote cluster.

  The Deckhouse hook could not get usable metadata from the endpoint ENDPOINT_NAME for IstioMulticluster MULTICLUSTER_NAME.

  The reason and the details are in the PublicMetadataExchangeReady and PrivateMetadataExchangeReady conditions of the IstioMulticluster. The endpoint may respond normally while the metadata is still unusable, for example, when the remote cluster does not deploy the gateway this cluster reaches it through.

  To view the conditions, run the following command:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  To reproduce the request to the public endpoint, run the following command:

  curl ENDPOINT_NAME
  

  To reproduce the request to private endpoints (run from the d8-system/deckhouse pod), run the following:

  KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
  LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
  REMOTE_CLUSTER_UUID="$(d8 k get istiomulticluster MULTICLUSTER_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
  TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-multicluster --ttl 1h)"
  curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME
  

• D8IstioMulticlusterRemoteAPIHostDoesntWork EE S6
  Multicluster remote API host health check failed.

  The remote API host API_HOST for IstioMulticluster MULTICLUSTER_NAME has failed the health check performed by the Deckhouse monitoring hook.

  To reproduce the request (run from the d8-system/deckhouse pod), run the following:

  TOKEN="$(deckhouse-controller module values istio -o json | jq -r --arg ah API_HOST '.internal.multiclusters[]| select(.apiHost == $ah)| .apiJWT ')"
  curl -H "Authorization: Bearer $TOKEN" https://API_HOST/version
  

• D8IstioMulticlusterRemoteClusterUUIDMismatch EE S4
  IstioMulticluster peer cluster UUID changed.

  Public metadata from ENDPOINT_NAME for IstioMulticluster MULTICLUSTER_NAME reports cluster UUID XXX, but the previously pinned UUID is XXX.

  Deckhouse stops private metadata exchange and does not send a multicluster JWT until the peer identity is confirmed again.

  Check whether spec.metadataEndpoint was pointed to another cluster or the remote endpoint was replaced.

  To inspect the resource:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o yaml
  curl ENDPOINT_NAME
  

  If the peer change is intentional, recreate the IstioMulticluster resource so a new UUID can be pinned.

• D8IstioOperatorReconcileError CE S5
  The istio-operator is unable to reconcile Istio control plane setup.

  Deckhouse has detected an error in the istio-operator reconciliation loop.

  To investigate the issue, check the operator logs:

  d8 k -n d8-istio logs -l app=operator,revision=REVISION_NUMBER
  

• D8IstioPodsWithoutIstioSidecar EE S4
  Detected pods without Istio sidecars but with istio-injection configured.

  There is a pod POD_NAME in the NAMESPACE namespace without Istio sidecars, but with istio-injection configured.

  To identify the affected pods, run the following command:

  d8 k -n NAMESPACE get pods -l '!service.istio.io/canonical-name' -o json | jq -r '.items[] | select(.metadata.annotations."sidecar.istio.io/inject" != "false") | .metadata.name'
  

• D8IstioRemoteClusterNotSynced EE S4
  XXXIstioMulticluster MULTICLUSTER_NAME is not synchronized.XXXIstio remote cluster CLUSTER_ID is not synchronized.XXX

  XXXThe Istio control plane instance INSTANCE_NAME cannot synchronize with the remote cluster described by IstioMulticluster MULTICLUSTER_NAME (cluster ID CLUSTER_ID).XXXThe Istio control plane instance INSTANCE_NAME cannot synchronize with the remote cluster CLUSTER_ID.XXX

  Possible causes:

  • The remote cluster is offline.
  • The remote API endpoint is not reachable.
  • The remote ServiceAccount token is invalid or expired.
  • There is a TLS or certificate issue between the clusters.
  • Remote and local Istio versions mismatch.

  XXXCheck the conditions of the IstioMulticluster:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  All conditions must be True. If a condition is not True, its reason and message say what is wrong. Access to the remote API is reported by the RemoteAPIServerReady condition.

  If there is no such resource, this cluster was registered by the secret XXX alone — check the API address and the token in the kubeconfig it stores. XXXNo IstioMulticluster resource registered this cluster, so check the API address and the token in the kubeconfig stored in the secret XXX. XXX

• D8IstioReservedUIDConflict CE S4
  Application container is running as UID 1337 (reserved by Istio).

  Container CONTAINER_NAME in pod POD_NAME in namespace NAMESPACE is running as UID 1337.

  UID 1337 is reserved by Istio for the istio-proxy sidecar container. When application containers run with this UID, their traffic bypasses Istio completely — no routing rules, mTLS, or telemetry will be applied.

  To fix the issue, change the runAsUser in the container’s or pod’s securityContext to a different UID.

• D8IstioVersionIsIncompatibleWithK8sVersion CE S3
  The installed Istio version is incompatible with the Kubernetes version.

  The installed Istio version VERSION_NUMBER may not work properly with the current Kubernetes version VERSION_NUMBER because it’s not supported officially.

  To resolve the issue, upgrade Istio following the guide.

• IstioIrrelevantExternalServiceFound CE S5
  External service found with irrelevant ports specifications.

  A service NAME in the NAMESPACE namespace has an irrelevant port specification.

  The .spec.ports[] field isn’t applicable for services of the ExternalName type. However, Istio renders port listeners for external services as 0.0.0.0:port, which captures all traffic to the specified port. This can cause problems for services that aren’t registered in the Istio registry.

  To resolve the issue, remove the .spec.ports section from the service configuration. It is safe.

Модуль kube-dns

• KubernetesCoreDNSHasCriticalErrors CE S5
  Critical errors found in CoreDNS.

  Deckhouse has detected at least one critical error in the CoreDNS pod POD_NAME.

  To resolve the issue, review the container logs:

  d8 k -n kube-system logs POD_NAME
  

Модуль metallb

• D8MetallbBGPSessionDown EE S4
  MetalLB BGP session is down.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME has BGP session PEER down.

  Check the logs for details:

  d8 k -n d8-metallb logs daemonset/speaker -c speaker
  

• D8MetallbConfigNotLoaded EE S4
  The MetalLB configuration hasn't been loaded.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME hasn’t been loaded.

  To find the cause of the issue, review the controller logs:

  d8 k -n d8-metallb logs deploy/controller -c controller
  

• D8MetallbConfigStale EE S4
  MetalLB is running on a stale configuration.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME is running on a stale configuration because the latest configuration failed to load.

  To find the cause of the issue, review the controller logs:

  d8 k -n d8-metallb logs deploy/controller -c controller
  

• D8MetallbNotSupportedServiceAnnotationsDetected SE S4
  The annotation 'ANNOTATION_NAME' has been deprecated for the service 'NAME' in the 'NAMESPACE' namespace.

  The annotation ‘ANNOTATION_NAME’ has been deprecated for the service ‘NAME’ in the ‘NAMESPACE’ namespace.

  The following service annotations are no longer effective:

  • metallb.universe.tf/ip-allocated-from-pool: Remove this annotation.
  • metallb.universe.tf/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
  • metallb.universe.tf/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
  • metallb.universe.tf/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.
  • metallb.io/ip-allocated-from-pool: Remove this annotation.
  • metallb.io/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
  • metallb.io/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
  • metallb.io/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.

  Please note. Existing LoadBalancer services of Deckhouse have been migrated automatically, but the new ones will not be.

• D8MetallbObsoleteLayer2PoolsAreUsed SE S7
  The metallb module has obsolete layer2 pools configured.

  In ModuleConfig version 2, addressPool ‘NAME’ of type “layer2” are ignored. They should be removed from the configuration.

• D8MetallbUpdateMCVersionRequired SE S5
  The metallb ModuleConfig settings are outdated.

  D8 MetalLB settings are outdated.

  To resolve this issue, increase version in the ModuleConfig metallb.

• L2LoadBalancerOrphanServiceFound SE S4
  Orphaned service with an irrelevant L2LoadBalancer name has been found.

  The cluster contains an orphaned service NAME in the NAMESPACE namespace with an irrelevant L2LoadBalancer name.

  To resolve this issue, verify the L2LoadBalancer name specified in the annotation network.deckhouse.io/l2-load-balancer-name.

Модуль multitenancy-manager

• ClusterResourceGrantPolicyViolation CE S6
  Objects in namespace XXX use cluster-wide resources that are not available to its project.

  Objects in namespace XXX reference cluster-wide resources (for example, a StorageClass or a ClusterRole) that the current ClusterResourceGrantPolicy objects do not make available to the project of this namespace. The project label of the metric holds the namespace, not the project name. Such an object was created before the policy changed. It keeps working, and when it is updated, only new values are checked.

  Steps to troubleshoot:

  1. Find the objects, their fields and the resources on the “Security” → “Cluster Resource Grant Violations” Grafana dashboard, or in the d8_cluster_objects_grant_violated{project="XXX"} metric (labels violating_object_name, violating_field, violating_resource and grant).

  2. Check which resources are available to the project:

     d8 k get available RESOURCE_NAME -n XXX -o yaml
     

  3. Make the resource available to the project in a ClusterResourceGrantPolicy, or change the field of the object to an available value.
• MultitenancyManagerHelmProjectTemplate CE S9
  ProjectTemplate XXX still renders its projects from a Helm text in resourcesTemplate.

  ProjectTemplate XXX describes the objects of its projects with a Helm text in resourcesTemplate, and its projects are rendered from that text. They keep working as before. This form of a template is deprecated, and resourcesTemplate goes away in a future release, so move the template to structured fields and manifests before then. Moving a template from a Helm text describes the move step by step.

  The built-in templates default, secure and secure-with-dedicated-nodes are structured and accept the same parameters as their Helm texts did, except administrators and resourceQuota, so they show how a text maps to the fields. The full mapping is in Moving a template from a Helm text. Objects of the text that have no structured field go to manifests, which the controller applies with the d8:project:admin role of the project. An AuthorizationRule of the Admin access level goes to .spec.administrators of each Project instead, because manifests does not create one. Cluster-scoped objects have no place in a template.

  Once the template has structured fields in place of resourcesTemplate, the AuthorizationRule and ResourceQuota objects that .spec.administrators and .spec.quota replace are deleted after these fields work, and an AuthorizationRule that names nobody from .spec.administrators any more is deleted at once. Every other object of the text that the structured fields do not render under the same kind and name is left in place, and the MultitenancyManagerProjectKeepsHelmTemplateObjects alert names the project until manifests takes the object over or it is deleted. The administrators then have the d8:project:admin role in place of the Admin access level, and the two roles do not grant the same permissions. See Switching a project from a Helm text.

  Steps:

  1. Read the template and compare it with the built-in template it was copied from, if any:

     d8 k get projecttemplate XXX -o yaml
     d8 k get projecttemplate default -o yaml
     

  2. List the projects of the template. A template that no project uses can be deleted, and the alert goes with it:

     d8 k get projects -l projects.deckhouse.io/project-template=XXX
     

  3. Find the projects of the template that still set administrators or resourceQuota in .spec.parameters:

     d8 k get projects -l projects.deckhouse.io/project-template=XXX -o json | jq -r '.items[] | select((.spec.parameters // {}) | has("administrators") or has("resourceQuota")) | .metadata.name'
     

     These values get no standard fields and are not handed over, so move them to .spec.administrators and .spec.quota first, and only as far as they mean the same there. .spec.administrators gives users and groups the d8:project:admin role, so an administrator with another access level or another kind of subject needs a ProjectRoleBinding of its own. .spec.quota enforces every limit it holds, 0 included, so leave out a limit that the text did not render, such as a 0 that it skipped.

  4. If the structured fields and manifests cover what the projects need, which they do unless the projects need cluster-scoped objects of the text or objects that the d8:project:admin role does not allow, write the template as deckhouse.io/v1alpha2 with these fields and manifests and without resourcesTemplate, in one change, where the template is applied from, for example in Git or in CI. A template that projects use cannot lose resourcesTemplate without structured fields or manifests in its place, and once it has fields that deckhouse.io/v1alpha1 cannot describe, a write of it through deckhouse.io/v1alpha1 is refused. If the template is an unchanged copy of a built-in template, you can instead move its projects to that template in .spec.projectTemplateName and then delete the copy.

  5. Check each project after the switch. Its HandoverPending condition is removed once the replaced objects are deleted, its ManifestsApplied condition says whether the objects of manifests are applied, and .status.keptObjects lists the objects left in place:

     d8 k get project PROJECT_NAME -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
     

• MultitenancyManagerPreviousRevisionStillRunning CE S6
  Pods of several revisions of the multitenancy-manager controller have run side by side for more than 15 minutes.

  Pods of VALUE revisions of the multitenancy-manager Deployment in the d8-multitenancy-manager namespace have been pending or running side by side for more than 15 minutes.

  The rollout of the controller is stuck: the new pod is pending or not ready, or an old pod is stuck terminating on a lost node. The new pod waits for the pods of older revisions for 10 minutes at most and then starts its controllers anyway. If an old pod of a revision that runs without leader election is still running, both pods write the Helm releases of the projects, and objects in the project namespaces may be deleted and created again.

  Steps to troubleshoot:

  1. List the controller pods with their revisions and nodes:

     d8 k -n d8-multitenancy-manager get pods -L pod-template-hash -o wide
     

  2. Find out whether the new pod has started its controllers:

     d8 k -n d8-multitenancy-manager logs NEW_POD_NAME -c multitenancy-manager | grep 'other revisions'
     

     • hold the controllers until the pods of other revisions finish means that the new pod holds the lease and waits.
     • no pods of other revisions to wait for means that it has started its controllers at once, because the old pods had finished or were deleted more than 10 minutes ago.
     • the pods of other revisions finished means that it has started its controllers after waiting, because the old pods finished or were deleted more than 10 minutes ago.
     • the pods of other revisions did not finish in time, starting the controllers anyway means that it has started its controllers after 10 minutes of waiting.

     Without any of these lines, the new pod waits for the lease that the old pod holds. When the wait ends after 10 minutes, the new pod also writes a Warning event with the reason PreviousRevisionStillRunning, and the message of the event names the old pods. An event is kept for an hour by default:

     d8 k -n d8-multitenancy-manager get events --field-selector reason=PreviousRevisionStillRunning
     

  3. Check the node of the old pod, shown in the NODE column:

     d8 k get node NODE_NAME
     

  4. Act on what you see:

     • The old pod is Running, not terminating, and the new pod is not ready. The Deployment removes the old pods only after the new ones are ready, and deleting an old pod does not help, since its ReplicaSet creates it again. Find out why the new pod is not ready:

       d8 k -n d8-multitenancy-manager describe pod NEW_POD_NAME
       

     • The old pod is Terminating on a node that is NotReady. Such a pod stays until its node comes back or the Node object is deleted. If the node is lost for good, delete the Node object, and its pods are deleted with it. If you know the machine of the node is powered off, you can force-delete the old pod instead:

       d8 k -n d8-multitenancy-manager delete pod OLD_POD_NAME --force --grace-period=0
       

       Do not force-delete a pod whose containers may still run. The pod disappears from the API, while its process keeps writing the Helm releases.

• MultitenancyManagerProjectHandoverStuck CE S6
  Project XXX still keeps the AuthorizationRule and ResourceQuota objects its standard fields replace.

  An upgrade of the release of project XXX dropped AuthorizationRule or ResourceQuota objects that the Helm text of its template had rendered for the administrators and the quota of the project. These are the AuthorizationRule objects with the Admin access level of the administrators and a quota such as all-pods whose every limit .spec.quota sets. That happens when a template switches from resourcesTemplate to structured fields, when a project moves to such a template, or when the text no longer renders these objects. The controller kept them through the upgrade, so that the administrators never lost access and the namespace never lost its quota, and deletes them once the standard fields of the project replace them. The handover began more than 30 minutes ago and has not finished.

  Until it does, the kept AuthorizationRule objects keep granting the Admin access level, the user-authz:admin role, next to the d8:project:admin role of the d8-administrators ProjectRoleBinding. The kept quotas keep applying next to the d8-project-quota ResourceQuota, and the tighter of them wins.

  The kept objects are deleted by a reconcile of the project that succeeds once all of the following hold, and not earlier than 2 minutes after d8-administrators became Ready and after the handover began:

  • the d8-administrators ProjectRoleBinding names the administrators from .spec.administrators and is Ready for its current generation, when a kept AuthorizationRule names one of them;
  • the d8:project:admin ClusterRole has its aggregated rules, when a kept AuthorizationRule names one of the administrators;
  • the d8-project-quota ResourceQuota has the limits from .spec.quota and is calculated, that is, its status.hard matches its spec.hard, when a kept quota has limits.

  A kept AuthorizationRule that names nobody from .spec.administrators any more is deleted at once, and a subject removed from .spec.administrators is removed at once from a kept AuthorizationRule that names other administrators too. A reconcile that cannot find the template, validate the project against it, upgrade the release or apply the standard fields stops before the deletion, and the kept objects stay. A template that grants a role forbidden in projects puts the project in the Error state too, but it does not stop the deletion.

  The HandoverPending condition of the project says what the controller waits for.

  Steps to troubleshoot:

  1. Read what the controller waits for, and the state and the other conditions of the project:

     d8 k get project XXX -o jsonpath='{.status.state}{"\n"}{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
     

  2. Check the administrators binding. Its status.observedGeneration must match metadata.generation, and its Ready condition must be True:

     d8 k -n XXX get projectrolebinding d8-administrators -o yaml
     

     If the binding lags behind, the controller could not create its RoleBinding in some namespace of the project. The logs of the controller name the namespace and the error:

     d8 k -n d8-multitenancy-manager logs -l app=multitenancy-manager -c multitenancy-manager --tail=-1 --prefix | grep d8-administrators
     

  3. Check that the d8:project:admin ClusterRole has rules. The aggregation controller of kube-controller-manager fills them:

     d8 k get clusterrole d8:project:admin -o jsonpath='{.rules}'
     

  4. Check the project quota and the events in the namespace of the project:

     d8 k -n XXX get resourcequota d8-project-quota -o yaml
     d8 k -n XXX get events --sort-by=.lastTimestamp
     

     A quota whose status.hard stays empty is not processed by the resource quota controller of kube-controller-manager.

  5. Fix what the checks show, for example a RoleBinding that an admission policy refuses in one of the namespaces of the project, a kube-controller-manager that does not process the quota, or the condition that stops the reconcile of the project. The controller checks again whenever d8-administrators or d8-project-quota changes, and at least every minute, and deletes the kept objects once the conditions hold.

• MultitenancyManagerProjectKeepsHelmTemplateObjects CE S7
  Project XXX has VALUE objects left from the Helm text of its template.

  The release of project XXX no longer holds VALUE objects that the Helm text of a resourcesTemplate had rendered, because the project switched to structured fields, or because .spec.quota stopped covering a quota kept for a handover. Examples are AuthorizationRule or RoleBinding objects, quotas, network policies, workloads or their volumes. Instead of deleting them, the controller left them in place:

  • They keep working as before. The rules and the bindings keep granting access, the quotas keep applying, and the workloads keep running.
  • The project no longer manages them. A change of the template or of the project does not change them. When the project is deleted, the ones in its namespaces go with the namespaces, and the controller deletes the cluster-scoped ones that still carry the label.
  • They carry the projects.deckhouse.io/kept-from-helm-template: "true" label and the helm.sh/resource-policy: keep annotation, and they have lost the heritage: multitenancy-manager label. While they carry the label, only a requester who may make the same change to objects of their kind across the cluster may change or delete them, so the users of the project may not.

  The project lists them in .status.keptObjects until each of them is deleted, loses the label, is rendered by the template again, or is taken over by the manifests of the template. Decide for each object:

  • If it is still needed, describe it in the template or in the project, for example with a ProjectRoleBinding in place of an AuthorizationRule or a RoleBinding, or with a limit in .spec.quota, and then delete the object. Or remove the label to hand it over to the project, whose users may then change and delete it.
  • If it is in a namespace of the project, manifests of the template can take it over. Once manifests renders an object of the same kind, namespace and name, the controller applies it as the system:multitenancy-manager:project:XXX user, removes the label and takes the object off the list. The object then has what manifests renders, and a field that the Helm text set and manifests leaves out is removed.
  • If it is not needed, delete it. An access it grants ends only when it is deleted.
  • If the template renders an object of the same kind and name again, the project release takes it over. The controller then removes the label, puts the heritage: multitenancy-manager label back and removes the helm.sh/resource-policy: keep annotation, unless the template sets it itself.

  Steps:

  1. List the objects:

     d8 k get project XXX -o jsonpath='{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
     

  2. Keep an object as your own:

     d8 k -n NAMESPACE label KIND NAME projects.deckhouse.io/kept-from-helm-template-
     

  3. Delete an object:

     d8 k -n NAMESPACE delete KIND NAME
     

  The controller looks at the list every 10 minutes, and at a reconcile of the project at most once a minute.

• MultitenancyManagerProjectNamedLikeAdditionalNamespace CE S8
  Project XXX is named like an additional namespace of project XXX.

  The name of project XXX is the name of project XXX, a dash and a suffix. Additional namespaces of a project are named this way (project-spec.name of a ProjectNamespace), but XXX is not one of them. It is a separate project with a namespace of its own. The administrators, quota, bindings and template of project XXX do not apply to it.

  Such a pair appears when both namespaces existed before they became projects, or when a project is created under such a name. Nothing is broken, but the names read as one project to anyone who looks at them.

  If the two projects are meant to be separate, nothing has to change in the cluster. The alert keeps firing for as long as both projects exist, so silence it for this pair in Alertmanager.

  If XXX is meant to be an additional namespace of project XXX, recreate it as one. A namespace cannot move between projects in place. Deleting the project deletes its namespace with everything in it.

  1. Make sure that everything running in the XXX namespace can be deployed again, or move it elsewhere first.

  2. Find the ClusterProjectRoleBinding objects that name the ServiceAccounts of the XXX namespace. While a cluster-wide binding names them, the ProjectNamespace in step 4 is refused, so remove these subjects from the bindings first and add them back once the namespace is there again:

     d8 k get clusterprojectrolebindings -o json | jq -r --arg ns XXX '.items[] | select(any(.spec.subjects[]?; (.kind == "ServiceAccount" and .namespace == $ns) or (.kind == "User" and (.name | startswith("system:serviceaccount:" + $ns + ":"))) or (.kind == "Group" and .name == "system:serviceaccounts:" + $ns))) | .metadata.name'
     

  3. Delete the project. The command returns once the namespace is gone:

     d8 k delete project XXX
     

  4. Create a ProjectNamespace in the XXX namespace, with spec.name set to the part of the name after XXX-. The controller creates the XXX namespace again, as an additional namespace of project XXX.
  5. Deploy the workloads into the namespace.

Модуль node-local-dns

• D8NodeLocalDNSCacheDenialEvictionsHigh BE S6
  node-local-dns denial cache is being frequently evicted.

  The denial (NXDOMAIN) cache in node-local-dns is evicting entries frequently.

  This might indicate that there are Services that are being recreated too often in cluster. So, denial cache size is not enough (denial 65536 is currently configured).

  Affected node: NODE_NAME

• D8NodeLocalDNSCacheSuccessEvictionsHigh BE S6
  node-local-dns success cache is being frequently evicted.

  The success cache in node-local-dns is evicting entries frequently (>100 in 5 minutes).

  This could mean there are too many different DNS records application have to resolve (to many Services?). So, node-local-dns cache capacity is too small (success 120000 is currently configured), or traffic patterns changed.

  Affected node: NODE_NAME

• D8NodeLocalDNSForwardMaxConcurrentRejects BE S5
  The forward plugin is rejecting DNS queries due to max concurrency limit.

  More than 20 queries were rejected within the last 5 minutes on node NODE_NAME because the max concurrent query limit was reached.

• D8NodeLocalDNSKubeforwardRequestLatencyP95High BE S4
  High DNS request latency in kubeforward (p95 > 300ms under load).

  The 95th percentile DNS request latency for A-records through kubeforward exceeded 300 milliseconds over the last 5 minutes while handling more than 0.1 QPS.

  This may indicate slow upstreams, network issues, or DNS overload on this node.

  Affected node: NODE_NAME

  For more detailed diagnostics, inspect Top 5 upstreams by slow kubeforward requests in Grafana.

  If the alert fires only on one node, run the checks below from node NODE_NAME (for example, from a debug pod on this node). If the alert fires on many nodes, treat it as a potential systemic issue first.

  Get kubeforward upstream IPs from a master host:

  d8 k -n kube-system get po -l k8s-app=kube-dns -o jsonpath='{.items[*].status.podIP}'
  

  Then run latency checks on the affected node NODE_NAME (or from a debug pod on that node):

  UPSTREAMS="paste kube-dns pod IPs here"
  for ip in $UPSTREAMS; do
    echo "== upstream ${ip} =="
    for i in $(seq 1 20); do
      dig @"$ip" -p 5353 kubernetes.default.svc.cluster.local A +tries=1 +time=1 +stats | grep "Query time"
    done
  done
  

  To identify slow requests, enable module logs (enableLogs) and inspect slow query messages:

  pod=$(kubectl -n kube-system get po -l k8s-app=node-local-dns --field-selector spec.nodeName=NODE_NAME -o jsonpath='{.items[0].metadata.name}')
  d8 k -n kube-system logs "$pod" | grep '\[kubeforward\] slow query'
  

  Example log line: [kubeforward] slow query A kubernetes.default.svc.cluster.local. took 425ms (rcode=NOERROR, upstream=10.96.0.10:53)

  Interpretation:

  • If most upstreams are slow and many nodes are affected, suspect systemic network/CoreDNS overload.
  • If one upstream is slow or only one node is affected, suspect a localized issue.
• D8NodeLocalDNSUpstreamHealthcheckFailed BE S5
  Too many upstream health check failures in forward plugin.

  The health check system of the forward plugin failed more than 10 times within 5 minutes on node NODE_NAME. As a result, all upstreams were considered unhealthy.

Модуль node-manager

• CapsInstanceUnavailable CE S8
  There are unavailable instances in the MACHINE_DEPLOYMENT_NAME MachineDeployment.

  The MachineDeployment MACHINE_DEPLOYMENT_NAME has VALUE unavailable instances.

  Check the status of the instances in the cluster with the following command:

  d8 k get instance -l node.deckhouse.io/group=MACHINE_DEPLOYMENT_NAME
  

• ClusterHasOrphanedDisks CE S6
  Cloud data discoverer found orphaned disks in the cloud.

  The cloud data discoverer has found disks in the cloud that do not have a corresponding PersistentVolume in the cluster.

  You can safely delete these disks manually from your cloud provider:

  ID: ID, Name: NAME

• D8BashibleApiserverLocked CE S6
  Bashible-apiserver has been locked for too long.

  Bashible-apiserver has been locked for an extended period.

  To resolve the issue, check if the bashible-apiserver Pods are up-to-date and running:

  d8 k -n d8-cloud-instance-manager get pods -l app=bashible-apiserver
  

• D8CloudDataDiscovererCloudRequestError CE S6
  Cloud data discoverer cannot get data from the cloud.

  Refer to the cloud data discoverer’s logs for details:

  d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
  

• D8CloudDataDiscovererSaveError CE S6
  Cloud data discoverer cannot save data to a Kubernetes resource.

  Refer to the cloud data discoverer’s logs for details:

  d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
  

• D8ClusterAutoscalerManagerPodIsNotReady CE S8
  The POD_NAME Pod is NOT Ready.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE logs POD_NAME
  

• D8ClusterAutoscalerPodIsNotRunning CE S8
  The cluster-autoscaler Pod is NOT Running.

  The POD_NAME Pod is STATUS.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
  

• D8ClusterAutoscalerPodIsRestartingTooOften CE S9
  Too many cluster-autoscaler restarts detected.

  The cluster-autoscaler has restarted VALUE times in the past hour.

  Frequent restarts may indicate a problem. The cluster-autoscaler is expected to run continuously without interruption.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
  

• D8ClusterAutoscalerTargetAbsent CE S8
  Cluster-autoscaler target is missing in Prometheus.

  The cluster-autoscaler automatically scales nodes in the cluster. If it’s unavailable, it will be impossible to add new nodes when there’s not enough resources for scheduling Pods. It may also lead to unnecessary cloud costs due to unused but still provisioned cloud instances.

  To resolve the issue, follow these steps:

  1. Check availability and status of cluster-autoscaler Pods:

     d8 k -n d8-cloud-instance-manager get pods -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
     

  2. Verify that the cluster-autoscaler Deployment exists:

     d8 k -n d8-cloud-instance-manager get deploy -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
     

  3. Check the Deployment’s status:

     d8 k -n d8-cloud-instance-manager describe deploy deployment-name
     

• D8ClusterAutoscalerTargetDown CE S8

  Prometheus is unable to scrape cluster-autoscaler’s metrics.

• D8ClusterAutoscalerTooManyErrors CE S8
  Cluster-autoscaler is issuing too many errors.

  The cluster-autoscaler encountered multiple errors from the cloud provider when attempting to scale the cluster.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
  

• D8KubeletCheckpointStateUnpinnedContainers CE S4
  Pod NAMESPACE/POD_NAME may run in a NUMA zone other than the one recorded by kubelet.

  Kubelet on Node NODE_NAME reset the memory manager checkpoint and recorded new NUMA zones for Pod NAMESPACE/POD_NAME. Zones are applied only when a container is created, so the running containers stay in their previous zones, which may differ from the recorded ones.

  As a result, the zone the Pod really uses may also be assigned to another Guaranteed Pod, causing both Pods to share its memory resources. The Pod’s CPUs and memory may also be assigned to different NUMA zones, which slows down its memory access.

  The Pod is not restarted automatically. The restart time depends on the workload. To restore resource isolation, do the following:

  • To recreate the Pod’s containers in newly assigned NUMA zones, delete the Pod:

    d8 k -n NAMESPACE delete pod POD_NAME
    

  • For a virtual machine, perform a live migration to another node instead, without stopping it.
  • If the node runs many such Pods, you can drain it instead of deleting them one by one.

  The alert resolves within a few minutes after the Pod is recreated. No kubelet restart is needed.

• D8MachineControllerManagerPodIsNotReady CE S8

  The POD_NAME Pod is NOT Ready.

• D8MachineControllerManagerPodIsNotRunning CE S8
  The machine-controller-manager Pod is NOT Running.

  The POD_NAME Pod is STATUS.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
  

• D8MachineControllerManagerPodIsRestartingTooOften CE S9
  Too many machine-controller-manager restarts detected.

  The machine-controller-manager has restarted VALUE times in the past hour.

  Frequent restarts may indicate a problem. The machine-controller-manager is expected to run continuously without interruption.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=machine-controller-manager -c controller
  

• D8MachineControllerManagerTargetAbsent CE S8
  Machine-controller-manager target is missing in Prometheus.

  Machine-controller-manager controls ephemeral nodes in the cluster. If it becomes unavailable, it will be impossible to create or delete nodes in the cluster.

  To resolve the issue, follow these steps:

  1. Check availability and status of machine-controller-manager Pods:

     d8 k -n d8-cloud-instance-manager get pods -l app=machine-controller-manager
     

  2. Verify availability of the machine-controller-manager Deployment:

     d8 k -n d8-cloud-instance-manager get deployment machine-controller-manager
     

  3. Check the Deployment’s status:

     d8 k -n d8-cloud-instance-manager describe deployment machine-controller-manager
     

• D8MachineControllerManagerTargetDown CE S8

  Prometheus is unable to scrape the machine-controller-manager’s metrics.

• D8NeedMigrateStateToOpenTofu CE S4
  Terraform state must be migrated to OpenTofu.

  This likely means the automatic migration to OpenTofu was canceled due to destructive changes detected in the cluster state.

  Verify the following:

  • Current Terraform version:

    d8 k -n d8-system exec -it deployments/terraform-state-exporter -c exporter -- terraform version
    

  • Version in the Terraform state:

    d8 k -n d8-system get secret d8-cluster-terraform-state -o json | jq -r '.data["cluster-tf-state.json"] | @base64d | fromjson | .terraform_version'
    

  • Check for destructive changes:

    d8 k exec -it deploy/terraform-state-exporter -n d8-system -- dhctl terraform check
    

  To resolve the issue and migrate to OpenTofu manually, follow these steps:

  1. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run:

     dhctl converge
     

     This should resolve all destructive changes in the cluster.

  2. After that, the terraform-auto-converger Pod should complete the migration automatically.

  If the automatic migration doesn’t happen, use the install container with the current Deckhouse release and run the following command to enforce migration of the Terraform state to OpenTofu:

  dhctl converge-migration
  

  If this alert appears in a cluster created by Deckhouse Commander, it means the cluster has been updated to a new Deckhouse version without approving state changes (for example, destructive changes).

  To resolve this, follow these steps:

  1. In Deckhouse Commander, click Delete manually to temporarily remove the cluster from control.

  2. In the cluster, remove the commander-agent module using the following command:

     d8 k delete mc commander-agent
     

  3. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run the following command to resolve all destructive changes in the cluster:

     dhctl converge
     

  4. In Deckhouse Commander, click Attach to reattach the cluster.
• D8NodeCgroupV2NotSupported CE S8
  Node NODE_NAME is using deprecated cgroup v1 – migration to v2 required.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support cgroup v2. Starting from Kubernetes 1.35, cgroup v1 is deprecated. To migrate to cgroup v2, please upgrade the kernel to version 5.8 or newer. For more information, please refer to the documentation.

• D8NodeContainerdV2NotSupported CE S8
  Node NODE_NAME is using deprecated containerd v1 – migration to v2 required.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support containerd v2. Starting from Kubernetes 1.38, containerd v1.y will no longer be supported. Please check the requirements for containerd v2 and schedule your migration plan.

  The exact reason is listed in the node.deckhouse.io/containerd-v2-err annotation on the node.

  If the reason is erofs and the node runs Ubuntu in AWS, the module is shipped in a separate linux-modules-extra package. DKP installs it automatically, but for some kernels, mostly outdated ones, the package is no longer published in the APT repository. In that case, the installation is skipped and a warning is written to the node logs. The problem has been confirmed for the 5.15.0-1028-aws, 6.8.0-1024-aws, and 6.8.0-1029-aws kernels. Switch the node group to a newer AMI or kernel version that includes erofs.

• D8NodeControllerReconcileErrorsHigh CE S8
  Node-controller reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeControllerReconcileP99High CE S8
  Node-controller reconcile p99 is high for CONTROLLER_NAME.

  The p99 reconcile duration for controller CONTROLLER_NAME has been above 5 seconds for more than 10 minutes.

  This may indicate slow reconciliation, Kubernetes API latency, or degraded controller performance.

  To inspect node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeControllerWorkqueueDepthHigh CE S8
  Node-controller workqueue NAME has a high backlog.

  The node-controller workqueue NAME has more than 50 queued objects for more than 15 minutes.

  This may indicate that the controller cannot process reconciliation events fast enough.

  To inspect node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeGroupIsNotUpdating CE S8
  NodeGroup NODE_GROUP_NAME is not handling the update correctly.

  There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. Although Nodes have detected the update, none of them have received approval to start the update process.

  Most likely, there is a problem with the update_approval hook of the node-manager module.

• D8NodeIsNotUpdating CE S7
  Node NODE_NAME cannot complete the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received approval, started the update, and encountered a step that could cause downtime. The update manager (the update_approval hook of the node-group module) granted downtime approval, but no success message was received, which indicates that the update has not completed.

  To investigate the details, view Bashible logs on the Node:

  journalctl -fu bashible
  

• D8NodeIsNotUpdating CE S8
  Node NODE_NAME cannot complete the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, but failed to complete the update.

  To investigate the details, view Bashible logs on the Node:

  journalctl -fu bashible
  

• D8NodeIsNotUpdating CE S9
  Node NODE_NAME is not updating.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has a pending update but it’s neither receiving it nor attempting to.

  Most likely, Bashible is not handling the update correctly. It should annotate the Node with update.node.deckhouse.io/waiting-for-approval before the update can proceed.

  Options to investigate the details:

  • Check the expected configuration checksum for the NodeGroup:

    d8 k -n d8-cloud-instance-manager get secret configuration-checksums -o jsonpath={.data.NODE_GROUP_NAME} | base64 -d
    

  • Check the current configuration checksum on the Node:

    d8 k get node NODE_NAME -o jsonpath='{.metadata.annotations.node\.deckhouse\.io/configuration-checksum}'
    

  • View Bashible logs on the Node:

    journalctl -fu bashible
    

• D8NodeIsUnmanaged CE S9
  The NODE_NAME Node is not managed by the node-manager module.

  The NODE_NAME Node is not managed by the node-manager module.

  To resolve this issue, follow the instructions how to clean up the node and add it to the cluster.

• D8NodeUpdateStuckWaitingForDisruptionApproval CE S8
  Node NODE_NAME cannot obtain disruption approval.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received initial approval, and started the update. However, it reached a stage that could cause downtime and was unable to obtain disruption approval. A disruption approval is normally issued automatically by the update_approval hook of the node-manager module.

  To resolve this issue, investigate why the approval could not be granted to proceed with the update.

• D8ProblematicNodeGroupConfiguration CE S8
  Node NODE_NAME cannot begin the update.

  There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. However, Node NODE_NAME cannot begin the update.

  The Node is missing the node.deckhouse.io/configuration-checksum annotation, which may indicate that its bootstrap process did not complete correctly.

  Troubleshooting options:

  • Check the cloud-init log (/var/log/cloud-init-output.log) on the node.
  • Check the NodeGroupConfiguration resource associated with the NODE_GROUP_NAME NodeGroup for potential issues.
• NodeGroupHasStaticInternalNetworkCIDRsField CE S9
  NodeGroup NAME uses deprecated field spec.static.internalNetworkCIDRs.

  The spec.static.internalNetworkCIDRs field is deprecated and has been moved to the static cluster configuration.

  To resolve this issue, delete the field from NodeGroup NAME. This is safe, as the setting has already been migrated automatically.

• NodeGroupMasterTaintIsAbsent CE S4
  The master NodeGroup is missing the required control-plane taint.

  The master NodeGroup doesn’t have the node-role.kubernetes.io/control-plane: NoSchedule taint. This may indicate a misconfiguration where control-plane nodes can run non-control-plane Pods.

  To resolve the issue, add the following to the master NodeGroup spec:

    nodeTemplate:
      taints:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
  

  Note that the taint key: node-role.kubernetes.io/master is deprecated and has no effect starting from Kubernetes 1.24.

• NodeGroupProviderTypeIsInvalid CE S4
  The spec.providerType field is set incorrectly in NodeGroup NAME.

  The spec.providerType field of NodeGroup NAME is set to XXX, which is not the cloud provider the nodes of the group run in.

  To resolve this issue, set the field to a correct value:

  • If the nodes of the group run in a cloud, set the field to the type of that cloud provider, as published in its registration secret:

    kubectl -n kube-system get secrets -l cloud-provider.deckhouse.io/registration -o json \
    | jq -r '.items[] | .data.type | @base64d'
    

  • If the cluster is static, or the NodeGroup has nodeType: Static, set the field to None or leave it empty:

    kubectl patch nodegroup NAME --type=merge -p '{"spec":{"providerType":"None"}}'
    

• NodeGroupReplicasUnavailable CE S7
  NodeGroup NODE_GROUP_NAME has no available instances.

  This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupReplicasUnavailable CE S8
  Too many unavailable instances in the NODE_GROUP_NAME NodeGroup.

  The number of simultaneously unavailable instances in the NODE_GROUP_NAME NodeGroup exceeds the allowed threshold. This may indicate that the autoscaler has provisioned too many nodes at once. Check the state of the Machine in the cluster. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupReplicasUnavailable CE S8
  NodeGroup NODE_GROUP_NAME has unavailable instances.

  There are VALUE unavailable instances in the NodeGroup NODE_GROUP_NAME. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupUsesDeprecatedInCoreGPU CE S9
  NodeGroup NAME uses the deprecated built-in GPU support.

  NodeGroup NAME has spec.gpu configured while the gpu module is disabled. As a result, GPU nodes in the NodeGroup use the built-in GPU support, which is deprecated and will be removed in a future DKP release.

  To use the supported GPU implementation, enable the gpu module.

• NodeRequiresDisruptionApprovalForUpdate CE S8
  Node NODE_NAME requires disruption approval to proceed with the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, received initial approval, and started the update. However, it encountered a stage that may cause downtime and requires manual disruption approval, because the NodeGroup is configured with disruptions.approvalMode: Manual.

  To resolve the issue, ensure the Node is ready for unsafe updates (drained) and grant disruption approval by annotating the Node with update.node.deckhouse.io/disruption-approved=.

  Caution:

  • Nodes in manual mode aren’t drained automatically.
  • Do not drain master nodes.

  1. To drain the Node and grant the update approval, run the following command:

     d8 k drain NODE_NAME --delete-emptydir-data --ignore-daemonsets --force=true &&
       d8 k annotate node NODE_NAME update.node.deckhouse.io/disruption-approved=
     

  2. Uncordon the Node once the update is complete and the annotation update.node.deckhouse.io/approved is removed:

     while d8 k get node NODE_NAME -o json | jq -e '.metadata.annotations | has("update.node.deckhouse.io/approved")' > /dev/null; do sleep 1; done
     d8 k uncordon NODE_NAME
     

  If the NodeGroup has multiple Nodes, repeat this process for each one, since only one Node is updated at a time. Consider temporarily switching to automatic disruption approval (disruptions.approvalMode: Automatic).

• NodeStuckInDraining CE S6
  Node NODE_NAME keeps failing to drain.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME is stuck in the draining process.

  The Node has the error from the last attempt in the update.node.deckhouse.io/drain-failed annotation, truncated to 1 KB. To read it, run:

  d8 k get node NODE_NAME -o json | jq -r '.metadata.annotations["update.node.deckhouse.io/drain-failed"]'
  

  The full error text is in the DrainFailed events on the Node:

  d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=DrainFailed --sort-by='.metadata.creationTimestamp'
  

  The drain is retried until it succeeds, so the alert clears on its own once the cause is gone. Until then the Node holds the update slot of its NodeGroup, and no other Node in that group is updated.

  The usual causes are a PodDisruptionBudget that never allows an eviction and a pod that never terminates. Fix the cause named in the error above.

• NodeStuckInDrainingForDisruptionDuringUpdate CE S6
  Node NODE_NAME is stuck in draining.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, started the update, and reached a step that could cause downtime. It is currently stuck in the draining process while waiting for automatic disruption approval.

  To get more details, run the following:

  d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=ScaleDown --sort-by='.metadata.creationTimestamp'
  

• StandbyHolderDeploymentReplicasUnavailable CE S5
  No available replicas in Deployment standby-holder-NODE_GROUP_NAME.

  Deckhouse has detected that there are no available replicas in the standby-holder-NODE_GROUP_NAME Deployment in the d8-cloud-instance-manager namespace.

  Check the Deployment and associated Pods for issues.

• UnmetCloudConditions CE S6
  Deckhouse update is unavailable due to unmet cloud provider conditions.

  Deckhouse has detected that some cloud provider–specific conditions have not been met. Until these issues are resolved, updating to the new Deckhouse release is not possible.

  Troubleshooting details:

  • Name: NAME
  • Message: MESSAGE_CONTENTS

Модуль okmeter

• D8OkmeterAgentPodIsNotReady CE S6

  Okmeter agent is not Ready.

Модуль openvpn

• OpenVPNClientCertificateExpired CE S4
  OpenVPN client certificate expired for XXX

  The OpenVPN client certificate for XXX has expired.

  Check the certificate information and make sure it actually expires:

  d8 k -n d8-openvpn get secrets -l name="XXX"  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  Renew or delete the expired certificate. Check the documentation for details.

• OpenVPNClientCertificateExpired CE S5
  OpenVPN client certificate expires for XXX in less than 7 days.

  The OpenVPN client certificate for XXX is set to expire in VALUE days.

  Renewal of certificate required. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=XXX  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNClientCertificateExpired CE S6
  OpenVPN client certificate expires for XXX in less than 30 days.

  The OpenVPN client certificate for XXX is set to expire in VALUE days.

  Renewal of certificate required. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=XXX  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCACertificateExpired CE S4
  OpenVPN CA certificate has expired.

  OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets openvpn-pki-ca  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  The hook should rotate CA and server certificates before expiry.

• OpenVPNServerCACertificateExpiresTomorrow CE S4
  OpenVPN CA certificate will expire tomorrow

  OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets openvpn-pki-ca  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  The hook should rotate CA and server certificates before expiry.

• OpenVPNServerCACertificateExpiringInAWeek CE S5
  OpenVPN CA certificate expires in less than 7 days.

  The OpenVPN CA certificate is set to expire in VALUE days.

  Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCACertificateExpiringSoon CE S5
  OpenVPN CA certificate expires in less than 30 days.

  The OpenVPN CA certificate is set to expire in VALUE days.

  Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCertificateExpired CE S4
  OpenVPN server certificate has expired.

  The OpenVPN server certificate has expired.

  OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  Check the certificate information and make sure it actually expires.

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCertificateExpiresTomorrow CE S4
  OpenVPN server certificate will expire tomorrow.

  The OpenVPN server certificate is set to expire in less than 1 day.

  OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  Check the certificate information and make sure it actually expires:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

Модуль registry

• D8RegistryAirGapTransitionHeld CE S5
  The registry upstream cannot be removed yet.

  The configuration has no upstream, so the cluster is meant to be air-gapped, and the module is still holding the previous upstream: either the cache is not complete enough to stand alone, or workloads in the platform namespaces (kube-system and d8-*) still reference the upstream and would have nowhere to pull from without it. Workloads in other namespaces are not counted and do not hold the transition.

  This is the safe outcome and the cluster is pulling images normally. It is also the one transition in this module that could cut every node off from images, which is why it waits rather than proceeds — and why a wait this long is worth looking at: nothing about it resolves on its own if the cache has stopped filling, or if a rollout onto the in-cluster address is stuck.

  Check the cache and the current count of references to the upstream (upstreamReferences):

  d8 k get registrystorage registry -o jsonpath='{.status}' | jq '{phase,fill,safeToDropUpstream,upstreamReferences,leader}'
  d8 k get registryconfig registry -o jsonpath='{.status.effectiveUpstream}' | jq
  

  The registry-controller log names up to five of the referencing objects each time the count changes. To see the last such line:

  d8 k -n d8-system logs -l app=registry-controller -c registry-controller --prefix --tail=-1 | grep 'still names the upstream' | tail -1
  

• D8RegistryConfigInvalid CE S4
  The registry configuration was rejected.

  The registry module cannot use its configuration, so the cluster keeps whatever arrangement it already had. Nothing has broken yet; nothing new will take effect either.

  What was wrong is in the resource:

  d8 k get registryconfig registry -o jsonpath='{.status.conditions}' | jq
  

• D8RegistryDrainStuck CE S6
  The registry module cannot finish leaving the pull path.

  The registry ModuleConfig asks for Unmanaged, and the module is still running its components and still serving registry.d8-system.svc:5001. That is deliberate: workloads in the cluster still name that address, and taking it away before they move would leave them unable to pull. It has been waiting for half an hour, which is longer than it should take.

  Nothing is degraded while this alert is firing. The cluster pulls, the module serves, and the only thing not happening is the withdrawal.

  What to look at:

  1. What still names the address. The module logs the first few on every pass, and the count is a metric:

     d8 k -n d8-system logs deploy/deckhouse -c deckhouse | grep "still serving the in-cluster registry"
     

  2. Then the workloads themselves, which is where the answer usually is:

     d8 k get pods,deployments,daemonsets,statefulsets -A -o json \
       | jq -r '.items[] | select([.spec.template.spec.containers[]?.image, .spec.containers[]?.image] | any(startswith("registry.d8-system.svc:5001"))) | "\(.kind) \(.metadata.namespace)/\(.metadata.name)"'
     

  A third cause is a cluster with nowhere to go, and it is the one to check first on a cluster that was air-gapped or installed with no upstream. Leaving the pull path means pointing the cluster back at the registry the module was fetching from, and a module serving purely from its own store has no such registry to name. It says so in the log:

  d8 k -n d8-system logs deploy/deckhouse -c deckhouse | grep "no upstream to point the cluster back at"
  

  Give the cluster a registry to pull from, or ask for Managed again. Nothing is degraded while it waits, and the withdrawal will not complete until one of the two happens.

  Otherwise two causes account for it. Either a release is not being re-rendered — check that the operator queue is not stuck, since the references move only as each module’s release is rendered again:

  d8 k -n d8-system exec deploy/deckhouse -c deckhouse -- deckhouse-controller queue main
  

  Or something outside the platform’s own renders names the address: a workload of yours that hardcoded it, or a manually edited manifest. Those will never move on their own, and until they are changed to pull from the registry the cluster was installed with, the module will keep serving them — which is the intended behaviour, not a failure.

  One shape of this is worth naming, because it was hit the first time this ran on a live cluster: a rolling update stuck for reasons of its own. The new pod is already on the upstream registry and never becomes ready, so the superseded ReplicaSet keeps its old pod, and that pod names the in-cluster address. A pod that has already pulled is not waited on, so this alone does not hold the withdrawal — but a pod stuck BEFORE its containers start does, and rightly, since it is exactly what would fail. Fix the rollout, or delete the pod that cannot start:

  d8 k get pods -A --field-selector status.phase=Pending
  

  A drain can also be abandoned by asking for Managed again: the module then keeps the pull path it never finished giving up, and this alert goes away.

• D8RegistryMigrationPending CE S8
  The registry module is still running its previous implementation.

  This cluster still pulls images the way the previous implementation of the registry module arranged it, and the handover to the current one has not happened.

  Reason: XXX

  Read the reason first, because it separates two very different situations:

  • The reason says the previous implementation has not recorded a mode yet, or that it is transitioning. Nothing is wrong: the handover is waiting for a state to settle and will happen on its own.

  • The reason names a mode other than Unmanaged — Direct, Proxy or Local. Then this cluster’s pull path is already degraded, and this alert is how it shows: this release renders none of the previous implementation’s objects, so the in-cluster proxy, the registry Service and the per-node secrets it served are gone, while the nodes still point at them. Images already on a node keep working; anything that has to be pulled will not. Do not wait for this to resolve itself — it will not. Go to D8RegistryMigrationPreflightBlocked, which says what to do per case, and for Local note that the documented procedure does not apply at all.

  The handover waits because both implementations configure the same thing on every node: which registry the container runtime asks, and with which credentials. Running both would not merge those two answers, it would race them. So the current implementation takes over only once the previous one has let go of the pull path, which is its Unmanaged state.

  What to do:

  1. Bring the registry configuration in the deckhouse ModuleConfig to Unmanaged and wait for that transition to finish:

     d8 k get mc deckhouse -o jsonpath='{.spec.settings.registry.mode}'
     d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head
     

  2. Once the state reports mode: Unmanaged with no pending target mode, the handover happens on the module’s next reconciliation and this alert clears. The cluster keeps pulling from the same registry throughout.

  3. To have the module manage the pull path afterwards, set mode: Managed in the registry ModuleConfig together with the registry to pull from. The module publishes a ready-made configuration for this cluster in the registry-suggested-config secret in d8-system.

• D8RegistryMigrationPreflightBlocked CE S6
  The registry migration must not be started on this cluster yet.

  A precondition of the registry module’s migration is not met, and the migration must not be started until it is: XXX — XXX.

  Whether anything is broken right now depends on which check it is, and the list below says so per case: upstream_reachable and node_containerd_registry_config are work to do before starting, while mode and not_local mean the cluster is already past the point where the documented procedure could be started cleanly.

  What each blocking check means:

  • mode — the cluster arrived on this release without having been brought to Unmanaged first, or the previous implementation is mid-transition. This release renders none of the previous implementation’s objects, so whatever they served — the in-cluster proxy in Proxy, the registry Service — is already gone, and the nodes still point at it. The discriminator has NOT flipped while this is failing, so the way back is the image: return the previous release, bring the cluster to Unmanaged by the documented procedure, then upgrade again.

  • not_local — the cluster’s registry is the cluster (Local). The ordinary procedure brings the cluster through Unmanaged, where every node pulls straight from an upstream, and this cluster has no upstream to pull from: the previous implementation does not even write node configuration in Unmanaged without one. This cluster has a procedure of its own instead — an OCI registry stood up in a namespace of your own, loaded with the image set and declared as the upstream for the length of the migration, then removed once the in-cluster storage holds the set itself. The blobs already on the control-plane nodes are adopted rather than downloaded again; what it costs is a second copy of the set on disk while the temporary registry exists. See “How do I migrate an air-gapped Local cluster?” in the module’s FAQ, and do not start the ordinary procedure here.

  • node_containerd_registry_config — one or more nodes carry a containerd conf.d file that sets registry fields of its own. Under the current implementation that file is not merged: the node’s registry configuration comes from /etc/containerd/registry.d, and the bashible step refuses to merge a file that would contradict it — by failing, which stops that node’s whole convergence, including kubelet configuration and version upgrades. The detail names the nodes. Move those settings into /etc/containerd/registry.d before the handover.

  • upstream_reachable — the registry this cluster pulls from does not answer. The procedure points every node straight at it, so starting now turns a documented step into an outage. Fix the reachability first:

    d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head
    d8 k -n d8-system logs deploy/deckhouse -c deckhouse | grep -i 'registry checker'
    

  The non-blocking checks are reported by the same metric with blocking="false":

  d8_registry_migration_preflight
  

• D8RegistryNodeForeignRegistryConfig CE S5
  Node NODE_NAME carries container runtime registry configuration the registry module did not write.

  Node NODE_NAME has a file under /etc/containerd/conf.d/ (containerd v1) or /etc/containerd/conf2.d/ (containerd v2) that configures a registry, and the registry module manages this cluster’s registry configuration. Two writers of that configuration is not a state the platform supports, so bashible refuses to merge such a file.

  Right now nothing is broken, and that is the danger. The refusal happens inside the bashible step that builds the container runtime configuration, and that step only runs when something makes it run. Until then the file has no effect at all: bashible compares the node’s configuration checksum, finds it unchanged and reports “Configuration is in sync, nothing to do”.

  The step does run on its own, though: a node re-applies its whole configuration periodically (roughly every four hours), and also on any registry change, platform upgrade or reboot. At that point the step fails with

  Failed to merge /etc/containerd/conf.d/file: it contains custom registry fields,
  configure them in /etc/containerd/registry.d instead
  

  and retries, so the node stops applying configuration entirely — hours after the change that caused it, and with nothing else to connect the two.

  What to do, before that happens:

  1. Find the file on the node:

     ls /etc/containerd/conf.d/ /etc/containerd/conf2.d/ 2>/dev/null
     grep -l registry /etc/containerd/conf.d/*.toml /etc/containerd/conf2.d/*.toml 2>/dev/null
     

  2. Move the registry part of it to a host directory, which is where this platform configures registries and where the module’s node agent leaves other people’s entries alone:

     /etc/containerd/registry.d/registry host/hosts.toml
     

  3. Remove the registry section from the conf.d file (anything non-registry in it can stay), and remove the NodeGroupConfiguration that writes it if that is where it comes from.

  Doing it across a fleet with a NodeGroupConfiguration has one catch worth knowing in advance: give it a weight below 32. The step that refuses the file is 032_configure_containerd, and a node stops at the step that fails — so a repair placed at a later weight never runs on the nodes that need it most. Measured: a cleanup configuration at weight 10 reached both nodes and cleared the label in 90 seconds.

  A node already stuck in the retry loop recovers on its own once the file is gone: measured, the configuration checksum returned in 75 seconds.

• D8RegistryNodeNotConverged CE S5
  Some nodes have not applied their registry configuration.

  The agent on one or more nodes has not applied the layout it was given. Those nodes keep pulling images the way they were configured before, so this is not an outage — but a change made to the registry configuration has not reached them, and the next one will not either.

  Which nodes, and what their agents say:

  d8 k get registrynodes -o custom-columns=\
  NODE:.metadata.name,GEN:.metadata.generation,APPLIED:.status.observedGeneration,OK:.status.reconciled
  d8 k -n kube-system logs -l component=registry-agent --tail=50
  

  Twenty minutes, because the agent re-reads its layout on a timer and a node that is merely slow or briefly unreachable resolves this on its own.

• D8RegistryNodeRunningFromDisk CE S6
  Some nodes are routing pulls from their on-disk copy of the configuration.

  The agent on one or more nodes cannot reach the API server and is routing pulls from the copy it keeps on disk. That is the copy working as designed — those nodes pull images normally, which is exactly why this is worth saying out loud: from every other angle it looks like success, and their configuration can drift arbitrarily far behind the cluster’s.

  An hour, because a node that briefly loses the API server is not news; a node that has not seen it since is.

  d8 k get registrynodes -o json | jq -r '
    .items[] | select(.status.conditions[]? | select(.type=="Reconciled" and .reason=="AppliedFromCache"))
    | .metadata.name'
  

• D8RegistryStaleCacheData CE S8
  A node still holds registry cache data that nothing uses.

  Node NODE_NAME holds data of the in-cluster registry cache while no cache is configured. Nothing reads it, and nothing will unless the cache is turned back on.

  It is kept on purpose. Turning the cache back on refills from what is already there rather than from scratch, and over a slow link that difference is hours — so the module does not delete it, because that would make the decision irreversible in the one direction that hurts. What it will not do is keep it quietly, which is what this alert is.

  Nothing is wrong and nothing needs doing urgently. To reclaim the space, remove the directory the store kept its blobs in: /var/lib/deckhouse/registry on a node of a NodeGroup with systemType: Immutable, /opt/deckhouse/registry on other nodes. With the cache off there is no RegistryStorage left to ask, so the command below looks at both.

  Open a root shell on the node over SSH. A node of a NodeGroup with systemType: Immutable has no SSH: start a debug pod on it and switch to the node’s root filesystem:

  d8 k debug node/NODE_NAME -it --image=ubuntu
  chroot /host
  

  Then remove the data:

  for dir in /opt/deckhouse/registry /var/lib/deckhouse/registry; do
    [ -d "$dir" ] && du -sh "$dir" && rm -rf "$dir"
  done
  

  Twenty-four hours, so that turning the cache off and on again in one working session never raises it.

• D8RegistryStorageIncomplete CE S6
  Some registry cache replicas do not hold the whole image set.

  One or more replicas of the in-cluster cache are still filling, or have stopped short. While an upstream is configured this costs nothing at pull time — a node falls back to it — but it does mean the cluster could not survive losing that upstream, and it is what holds an air-gap transition back.

  Two hours, because filling a cache over a slow link legitimately takes that long.

  d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq
  

• D8RegistryStorageNotReclaimed CE S6
  The registry cache has not reclaimed its disk for a week.

  No replica of the in-cluster cache has finished a garbage collection in the last week. Nothing is broken yet: the cache serves images normally, and it will keep doing so until the disk fills.

  It is worth attention because the collection is the only thing that ever removes anything. Every release adds a slice of the repository, so a store nobody reclaims grows in one direction — and a full store cannot be written to, which in an air-gapped cluster means it cannot be updated.

  Where to look:

  d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq \
    'map({node, collectedAt, collectionError})'
  d8 k get registrystorage registry -o jsonpath='{.spec.garbageCollection}' | jq
  

  The likely reasons, in the order they are worth checking:

  • the collection is turned off in the registry ModuleConfig under storage.garbageCollection.enabled;
  • the schedule has not come round yet, or names an hour that never arrives — a collection only runs on the schedule, and an unreadable expression is refused rather than guessed at;
  • no release is deployed, which is the one input that justifies deleting anything: every deletion is “older than what the cluster runs”, and without that there is nothing to compare against;
  • collectionError on a replica, which says what its own last attempt hit.

  One reading tells the first two apart from the rest. Each replica logs the time of its next collection, so its absence means none was ever scheduled — a different problem from one that is scheduled and failing:

  d8 k -n d8-system logs registry-storage-0 -c syncer | grep -i "garbage collection"
  

  Collecting puts one replica read-only while it runs. That is not an outage — it keeps serving every image it holds — but it does mean the schedule belongs at a quiet hour, which is where it defaults to.

• D8RegistryStoreNearlyFull CE S6
  The registry cache on node NODE_NAME holds over 85% of storage.size.

  The in-cluster registry cache on node NODE_NAME holds more than 85% of its budget, storage.size in the registry ModuleConfig. Nothing is refused yet. The next release may not fit: once the budget is reached, the cache stops accepting writes and D8RegistryStoreWritesRefused follows.

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store}'
  

  Raise storage.size if the disk under the store has room — filesystemFreeBytes above reserveBytes is what is left — or check that the garbage collection runs: it is the only thing that reclaims space, and D8RegistryStorageNotReclaimed says when it has not.

• D8RegistryStoreUnbounded CE S7
  The registry cache has no storage.size, so only the node's disk limits it.

  The in-cluster registry cache is on, and storage.size is not set in the registry ModuleConfig. The node is still protected: the store never takes the space kept free for the kubelet. But that is the only limit, so the store may grow until it leaves the node exactly that and no more — and on a master that shares its disk with the store, everything else on it then works with the minimum.

  Set storage.size to what the store may take on each master: two releases of the image set is the usual floor, and it has to fit on the disk under the store, above the reserve:

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store}'
  

  An hour, so that turning the cache on and setting the size in two steps does not raise it.

• D8RegistryStoreWritesRefused CE S4
  The registry cache on node NODE_NAME refuses writes (XXX).

  The in-cluster registry cache on node NODE_NAME has stopped accepting writes, because XXX. The node is safe — this is the limit doing its job — and the cache keeps serving every image it holds. What stops is everything that ADDS to it: filling the next release, replicating from the leader, d8 mirror push. Images the store cannot keep are streamed from the upstream while there is one; in an air-gapped cluster a release that is not in the cache cannot be installed.

  What the replica measured:

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store}'
  

  BudgetExhausted — the store holds as much as storage.size allows. Either raise storage.size in the registry ModuleConfig, if the disk has room for it, or let the garbage collection reclaim what no deployed release needs any more: it runs on storage.garbageCollection.schedule and keeps the deployed release, the previous one and everything newer.

  ReserveExhausted — the filesystem under the store is down to the space kept free for the node: the kubelet’s eviction threshold plus a margin. storage.size is not what stops it, the disk is. Free space on the node, or — which is what fixes it for good — give the store a disk of its own, sized for storage.size.

• D8RegistryUpstreamProbeFailing CE S4
  The configured registry upstream does not pass its preflight check.

  A change to the primary upstream was refused, and the cluster is still using the last one that worked. Nothing is degraded — that is what holding the previous upstream is for — but the change has not taken effect and will not until the check passes.

  The outcome label says which of three things is wrong, and they are not the same problem:

  • unreachable — the registry did not answer at all: network, firewall, or the registry itself.
  • auth — it answered and rejected the credentials. Usually an expired or wrong license key.
  • sentinel — it answered and accepted the credentials, but does not hold the Deckhouse images. Usually the wrong repository path.

  Current outcome: XXX

  d8 k get registryconfig registry -o jsonpath='{.status.conditions}' | jq
  

• D8RegistryUpstreamRejected CE S6
  An additional registry was refused.

  A RegistryUpstream resource was not accepted, so pulls for the registry it names are not intercepted on any node. Whatever asked for it — a module, or an operator — is not getting it.

  Reason: XXX

  d8 k get registryupstreams -o custom-columns=\
  NAME:.metadata.name,MATCH:.spec.match,ACCEPTED:.status.conditions[0].status,REASON:.status.conditions[0].reason
  

Модуль secret-copier

• D8SecretCopierDeprecatedLabels CE S9
  Obsolete antiopa_secret_copier=yes label detected.

  The secret-copier module has changed its labeling approach for original secrets in the default namespace.

  The label antiopa-secret-copier: "yes" is deprecated and will be removed soon.

  To resolve this issue, replace the label antiopa-secret-copier: "yes" with secret-copier.deckhouse.io/enabled: "" for all secrets used by the secret-copier module in the default namespace.

Модуль terraform-manager

• D8TerraformStateExporterClusterStateChanged CE S8
  Terraform-state-exporter cluster state change detected.

  The current Kubernetes cluster state is STATUS_REFERENCE compared to the Terraform state.

  It is important to reconcile the states.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterClusterStateError CE S8
  Terraform-state-exporter cluster state error.

  The terraform-state-exporter can’t check difference between the Kubernetes cluster state and the Terraform state.

  That was likely caused by terraform-state-exporter failing to run Terraform with the current state and configuration.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterHasErrors CE S8
  Terraform-state-exporter has encountered errors.

  Errors occurred during the operation of the terraform-state-exporter.

  To get more details, check the Pod logs:

  d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
  

• D8TerraformStateExporterNodeStateChanged CE S8
  Terraform-state-exporter node state change detected.

  The current state of node NODE_GROUP_NAME/NAME is STATUS_REFERENCE compared to the Terraform state.

  It is important to reconcile the states.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterNodeStateError CE S8
  Terraform-state-exporter node state error.

  The terraform-state-exporter can’t check the difference between the node NODE_GROUP_NAME/NAME state and the Terraform state.

  Probably, it occurred because terraform-manager had failed to run Terraform with the current state and configuration.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterNodeTemplateChanged CE S8
  Terraform-state-exporter node template change detected.

  The terraform-state-exporter has detected a mismatch between the node template in the cluster provider configuration and the one specified in the NodeGroup NAME`.

  Node template is STATUS_REFERENCE.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Adjust NodeGroup settings to fix the issue or bring the cluster in sync via the following command:

     dhctl converge
     

• D8TerraformStateExporterPodIsNotReady CE S8
  Terraform-state-exporter Pod is not Ready.

  The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

  To resolve the issue, check the following:

  1. Deployment description:

     d8 k -n d8-system describe deployment terraform-state-exporter
     

  2. Pod status:

     d8 k -n d8-system describe pod -l app=terraform-state-exporter
     

• D8TerraformStateExporterPodIsNotRunning CE S8
  Terraform-state-exporter Pod is not Running.

  The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

  To resolve the issue, check the following:

  1. Deployment description:

     d8 k -n d8-system describe deployment terraform-state-exporter
     

  2. Pod status:

     d8 k -n d8-system describe pod -l app=terraform-state-exporter
     

• D8TerraformStateExporterTargetAbsent CE S8
  Terraform-state-exporter target is missing in Prometheus.

  Prometheus cannot find the terraform-state-exporter target.

  To investigate the details:

  • Check the Pod status:

    d8 k -n d8-system get pod -l app=terraform-state-exporter
    

  • Check the container logs:

    d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
    

• D8TerraformStateExporterTargetDown CE S8
  Prometheus can't scrape terraform-state-exporter.

  Prometheus is unable to scrape metrics from the terraform-state-exporter.

  To investigate the details:

  • Check the Pod status:

    d8 k -n d8-system get pod -l app=terraform-state-exporter
    

  • Check the container logs:

    d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
    

Модуль user-authn

• D8DexAllTargetsDown CE S6
  Prometheus is unable to scrape Dex metrics.

  Prometheus cannot scrape any Dex pod in the d8-user-authn namespace. Users may be unable to sign in.

  Check the Dex pods and their logs:

  d8 k -n d8-user-authn get pods -l app=dex
  d8 k -n d8-user-authn logs deploy/dex
  

• D8DexAuthenticatorNamespaceUnusable CE S7
  DexAuthenticator NAMESPACE/NAME is left out of the release because its namespace accepts no objects.

  The NAMESPACE namespace is terminating, so the API server refuses every create in it. The user-authn module leaves the DexAuthenticator NAME out of its release: with it, the release would fail on the objects it cannot create and the Deckhouse main queue would stop behind the retrying module.

  A namespace does not finish terminating while something in it holds a finalizer, and the authenticator is usually that something. Check what keeps the namespace:

  d8 k get namespace NAMESPACE -o jsonpath='{.status.conditions}'
  d8 k -n NAMESPACE get dexauthenticator NAME -o jsonpath='{.metadata.finalizers}'
  

  Let the deletion finish: remove the finalizer that holds the authenticator, or delete the object that owns it. The authenticator is deleted with its namespace; to keep it, recreate it in a live namespace. The alert resolves once the authenticator is gone.

• D8DexAuthRequestsQuotaExhausted CE S3
  Dex AuthRequest objects ResourceQuota is exhausted in namespace "d8-user-authn".

  The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn has been fully exhausted.

  Dex is currently unable to create new AuthRequest custom resources, which will cause:

  • Dex health checks to fail immediately.
  • Authentication through external identity providers to stop working.
  • Users unable to log in via OAuth/OIDC flows.

  Immediate actions required:

  1. Check current AuthRequest objects and clean up stale ones:

     d8 k get authrequests.dex.coreos.com -n d8-user-authn --no-headers | wc -l
     d8 k delete authrequests.dex.coreos.com --all -n d8-user-authn
     

     Deleting AuthRequest objects interrupts sign-ins that are in progress.

  2. Temporarily increase the ResourceQuota limit:

     d8 k patch resourcequota authrequest-quota -n d8-user-authn --type='json' -p='[{"op": "replace", "path": "/spec/hard/count\/authrequests.dex.coreos.com", "value": "20000"}]'
     

     The module restores the quota on its next run.

  3. Investigate why Dex is not cleaning up AuthRequest objects automatically.

  4. AuthRequest objects expire after 10 minutes. A growing count usually means that a client or a bot repeatedly starts sign-ins. Find the source in the Dex logs.
• D8DexAuthRequestsQuotaHigh CE S4
  Dex AuthRequest objects quota usage is higher than 80% in namespace "d8-user-authn".

  The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn is approaching its limit.

  Current usage: XXX%

  When the quota is exhausted, Dex will be unable to create new AuthRequest custom resources, which may cause:

  • Dex health checks to fail.
  • Authentication through external identity providers (GitLab, GitHub, etc.) to stop working.

  Investigation steps:

  1. Check for stale AuthRequest objects that Dex should have cleaned up:

     d8 k get authrequests.dex.coreos.com -n d8-user-authn
     

  2. AuthRequest objects expire after 10 minutes. A growing count usually means that a client or a bot repeatedly starts sign-ins. Find the source in the Dex logs.

  3. Consider temporarily increasing the ResourceQuota limit if needed. The module restores the quota on its next run.
• D8DexProviderLDAPTLSConflict CE S8
  DexProvider NAME has a conflicting LDAP TLS configuration.

  DexProvider NAME combines mutually exclusive LDAP TLS settings: XXX. Dex silently ignores the redundant flag, which masks misconfiguration:

  • insecureNoSSL + startTLS: TLS is disabled (ldap://, port 389), so startTLS is ignored.
  • insecureNoSSL + insecureSkipVerify: TLS is disabled, there is no certificate to verify.
  • insecureNoSSL + rootCAData: TLS is disabled, the CA chain is ignored.

  The CRD schema rejects this combination on create/update; this DexProvider was created before the validation was added and remains active.

  Update the DexProvider to remove the conflict:

  • Plaintext LDAP (port 389): keep spec.ldap.insecureNoSSL: true, remove startTLS, insecureSkipVerify, and rootCAData.
  • STARTTLS on port 389: set spec.ldap.insecureNoSSL: false, spec.ldap.startTLS: true. Use insecureSkipVerify or rootCAData as needed.
  • ldaps:// on port 636: leave insecureNoSSL and startTLS at false. Use insecureSkipVerify or rootCAData as appropriate.

  Audit all LDAP DexProvider objects:

  d8 k get dexproviders.deckhouse.io -o yaml | yq '.items[] | select(.spec.type=="LDAP") | {name: .metadata.name, ldap: .spec.ldap | pick(["insecureNoSSL","startTLS","insecureSkipVerify","rootCAData"])}'
  

• D8UserAuthnControllerReconcileErrorsHigh CE S8
  JOB_NAME reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME of JOB_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check logs:

  d8 k -n d8-user-authn logs deploy/JOB_NAME --all-pods=true
  

• D8UserAuthnControllerTargetDown CE S6
  Prometheus is unable to scrape user-authn-controller.

  Prometheus cannot scrape metrics from user-authn-controller in namespace d8-user-authn.

  The controller is down, not Ready (informer cache has not synced), or the PodMonitor is missing. Static user login, UserAccount, expire, and provider checks will not run.

  Check:

  d8 k -n d8-user-authn get pods -l app=user-authn-controller
  d8 k -n d8-user-authn describe pods -l app=user-authn-controller
  

• D8UserAuthnControllerUnavailable CE S6
  user-authn-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the user-authn-controller deployment in the d8-user-authn namespace.

  While this alert is active, static Users may not get Dex Password objects, UserAccount projections stall, expired Users are not deleted, and DexProviderCheck / UserOperation stay pending.

  Check deployment and pod states:

  d8 k -n d8-user-authn get deploy user-authn-controller
  d8 k -n d8-user-authn get pods -l app=user-authn-controller
  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true
  

• D8UserAuthnDexClientControllerTargetDown CE S6
  Prometheus is unable to scrape dexclient-controller.

  Prometheus cannot scrape metrics from dexclient-controller in namespace d8-user-authn.

  The controller is down, not Ready (informer cache has not synced), or the PodMonitor is missing. The credentials of DexClients are not kept up to date.

  Check:

  d8 k -n d8-user-authn get pods -l app=dexclient-controller
  d8 k -n d8-user-authn describe pods -l app=dexclient-controller
  

• D8UserAuthnDexClientControllerUnavailable CE S6
  dexclient-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the dexclient-controller deployment in the d8-user-authn namespace.

  While this alert is active, a new DexClient gets no credentials Secret and no Dex client, and changes of DexClients are not applied.

  Check deployment and pod states:

  d8 k -n d8-user-authn get deploy dexclient-controller
  d8 k -n d8-user-authn get pods -l app=dexclient-controller
  d8 k -n d8-user-authn logs deploy/dexclient-controller --all-pods=true
  

• D8UserAuthnIdentityMigrationIncomplete CE S6
  Dex user data is not fully copied into UserIdentity objects.

  For more than 30 minutes, user-authn-controller has not finished copying the 2FA (TOTP) secret and the lock of Dex users from OfflineSessions and Password objects into UserIdentity objects.

  Until the copy of a provider and user pair completes, its 2FA secret and lock are stored only in the OfflineSessions and Password objects.

  The controller retries failed pairs by itself. Find the reason in its logs:

  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep -E 'identitymigration|UserIdentity'
  

  Typical reasons are API server errors and missing rights on useridentities.dex.coreos.com. Pairs that cannot be copied at all, for example an OfflineSessions object that cannot be decoded, are logged as Not copying the pair into UserIdentity and do not raise this alert.

• D8UserAuthnIDTokenTTLTooLong CE S6
  user-authn idTokenTTL is XXX, which is too long.

  The ModuleConfig user-authn sets idTokenTTL: XXX.

  Dex rotates the key that signs ID tokens every 6 hours (signer.config.keysRotationPeriod: 6h) and keeps a previous key only while tokens signed with it can still be valid.

  idTokenTTL maps to Dex expiry.idTokens. If that lifetime is greater than or equal to the signing-key rotation period (6h), every rotation must retain an extra previous key until all tokens from that key expire. Over time the set of retained signing keys grows without bound and Dex persists that growing key material.

  That is why idTokenTTL must be strictly less than 6h. Lower it, for example:

  apiVersion: deckhouse.io/v1alpha1
  kind: ModuleConfig
  metadata:
    name: user-authn
  spec:
    version: 3
    settings:
      idTokenTTL: 1h
  

  Settings version 3 rejects idTokenTTL >= 6h. A ModuleConfig of an earlier settings version with such a value is still accepted and its value is converted to 5h59m, so Dex already runs with 5h59m; this alert keeps firing until the ModuleConfig itself is updated.

• D8UserAuthnRevocationFeedLagging CE S6
  The revocation feed of user-authn-controller is not delivering to dex-authenticator.

  For more than 5 minutes, a stream of the revocation feed of user-authn-controller has not written to its dex-authenticator for over 30 seconds. A stream writes at least a heartbeat every 10 seconds, so the stream is stuck.

  While the stream is stuck, a session that is revoked (user lock, password reset, logout) keeps working in that dex-authenticator until its next refresh in Dex: up to 30 seconds while Dex is reachable, and no longer than idTokenTTL otherwise.

  Check the logs of the controller and of the dex-authenticator pods:

  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep revocation-feed
  

  The controller closes a stream that falls behind, and the dex-authenticator reconnects by itself. A lag that does not go away usually means a network problem between the dex-authenticator pods and the controller pods.

• D8UserAuthnRevocationFeedNotConsumed CE S6
  The dex-authenticator of the Dex client XXX does not receive the revocation feed.

  For more than 15 minutes, no replica of user-authn-controller has had a stream of the revocation feed open for the Dex client XXX. The client belongs to a DexAuthenticator (the client ID is dex-authenticator-name@namespace).

  The dex-authenticator of this client works without the feed: a session revoked by a user lock, a password reset, or a logout stops working there only at its next refresh in Dex, within 30 seconds while Dex is reachable, and no later than idTokenTTL.

  What to check:

  • the logs of the dex-authenticator pods: errors connecting to https://revocation-feed.d8-user-authn/revocations and answers 401 or 429;
  • that the Secret of the DexAuthenticator holds the CA of the feed in the revocation-feed-ca.crt key;
  • network policies in the namespace of the DexAuthenticator: they must allow connections to the user-authn-controller pods in d8-user-authn on port 8443.

  d8 k -n d8-user-authn get endpoints revocation-feed
  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep revocation-feed
  

Модуль user-authz

• D8UserAuthzAuthorizationRuleInvalid CE S7
  The bindings of XXX XXXXXX/XXXNAME cannot be applied (XXX).

  user-authz-controller cannot apply the bindings of XXX XXXXXX/XXXNAME; the Ready condition of the object is False with the reason XXX.

  • InvalidSpec: the rule cannot be rendered into bindings (unsupported accessLevel, an accessLevel not allowed for a namespaced rule, an additionalRoles entry that is not a ClusterRole or has an invalid name). Fix the object.
  • ApplyError: the API server rejected a write. The message of the condition names the binding and the error; a binding whose roleRef was changed by hand has to be deleted so that the controller recreates it.

  d8 k XXX-n XXX XXXget XXX NAME -o jsonpath='{.status.conditions}'
  

• D8UserAuthzAuthorizationRulesInvalid CE S7
  VALUE XXX objects cannot be applied (XXX).

  user-authz-controller reports VALUE XXX objects whose bindings cannot be applied, reason XXX. Up to 50 of them are named by the D8UserAuthzAuthorizationRuleInvalid alerts; a large number with the reason ApplyError points at a cluster-wide cause (an admission webhook, API server errors) rather than at the objects.

  d8 k get XXX -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',REASON:'.status.conditions[?(@.type=="Ready")].reason' | grep -v True
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzBindingsDrift CE S7
  user-authz-controller cannot converge the bindings of XXX.

  For more than 15 minutes some bindings of XXX objects differ from the state user-authz-controller wants (missing, superfluous, or with different subjects, roleRef, labels or owner). The controller reports drift only while its writes fail, so users may be missing access or keep access they should have lost.

  Find the objects and the reason:

  d8 k get clusterauthorizationrules -o custom-columns=NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',REASON:'.status.conditions[?(@.type=="Ready")].reason',MESSAGE:'.status.conditions[?(@.type=="Ready")].message'
  d8 k get authorizationrules -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',MESSAGE:'.status.conditions[?(@.type=="Ready")].message'
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzControllerReconcileErrorsHigh CE S8
  user-authz-controller reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME of user-authz-controller has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check logs:

  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzControllerTargetDown CE S6
  Prometheus is unable to scrape user-authz-controller.

  Prometheus cannot scrape metrics from user-authz-controller in the d8-user-authz namespace.

  The controller is down, not Ready (its informer cache has not synced), or the PodMonitor is missing. Access rules are not reconciled while it is down.

  Check:

  d8 k -n d8-user-authz get pods -l app=user-authz-controller
  d8 k -n d8-user-authz describe pods -l app=user-authz-controller
  d8 k -n d8-monitoring get podmonitor user-authz-controller
  

• D8UserAuthzControllerUnavailable CE S6
  user-authz-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the user-authz-controller deployment in the d8-user-authz namespace.

  While this alert is active, changes to ClusterAuthorizationRule and AuthorizationRule objects are not applied to their ClusterRoleBindings and RoleBindings, and the d8:dict bindings and the RoleBindings of system and subsystem roles in module namespaces are not updated. Existing bindings keep working.

  Check deployment and pod states:

  d8 k -n d8-user-authz get deploy user-authz-controller
  d8 k -n d8-user-authz get pods -l app=user-authz-controller
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzCustomRolesNotAggregated CE S6
  The aggregated ClusterRole of the XXX access level has no rules although custom roles for it exist.

  Custom ClusterRoles annotated with user-authz.deckhouse.io/access-level are granted to users through the aggregated ClusterRole user-authz:level:custom, which kube-controller-manager fills from the roles labelled with the level. For more than 10 minutes the aggregated role of the XXX level has no rules while custom roles it must aggregate exist, so users of that level (and every level above it) have none of the custom permissions.

  Check that the aggregated role exists and that the clusterrole-aggregation controller of kube-controller-manager is running:

  d8 k get clusterrole -l user-authz.deckhouse.io/access-level
  d8 k get clusterrole -o name | grep 'user-authz:.*:custom' | xargs -n1 d8 k get -o yaml
  d8 k -n kube-system logs -l component=kube-controller-manager --tail=200 | grep -i aggregat
  

• D8UserAuthzDeprecatedRBACv2CapabilityInUse CE S5
  A binding references a deprecated RBACv2 capability that no longer grants access.

  The XXX XXXXXX in namespace NAMESPACEXXX binds the deprecated RBACv2 capability XXX directly.

  Capabilities (d8:manage:permission:*, d8:use:capability:*) are aggregation building blocks — they are meant to be composed into the d8:{system,subsystem,namespace,project}:* roles, not bound directly. They were renamed and have NO compatibility alias, so this binding already grants nothing after the upgrade.

  No ClusterRole named XXX exists in the cluster. The alert also names a binding to an old role name that a module no longer ships, for example after the module moved to the new role names.

  Recreate the binding against the appropriate new role (or a custom role aggregating the new capability). Because roleRef is immutable, delete and recreate the binding:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  

• D8UserAuthzDeprecatedRBACv2ModuleRoleInUse CE S7
  A binding references the old name of a role a module renamed.

  The XXX XXXXXX in namespace NAMESPACEXXX references XXX, the old name of a role that a module renamed to XXX.

  The module keeps the old name as a compatibility alias, so the binding keeps working. The alias may grant less than XXX, for example when the module moved into a wider subsystem and the alias kept the rights of the old role, so check what the subjects of the binding need before recreating it. The module removes the alias in one of its own releases; the Deckhouse release is not held back by this binding. See the documentation of the module for when.

  Recreate the binding on the new name. Because roleRef is immutable, delete and recreate the binding pointing at XXX:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  

• D8UserAuthzDeprecatedRBACv2RoleInUse CE S7
  A binding still references a deprecated RBACv2 role.

  The XXX XXXXXX in namespace NAMESPACEXXX references the deprecated RBACv2 ClusterRole XXX.

  The RBACv2 role model was renamed: d8:manage:subsystem|all:level became d8:system:* / d8:subsystem:*:*, and d8:use:role:level became d8:namespace:level. This name still works via a compatibility alias kept for one release and will be removed in the next one — after that this binding will grant nothing. The release that removes the aliases is held back by the deprecatedRBACv2BindingsCount release requirement until this binding is recreated on the new name.

  Migrate the binding to the new role name. Because roleRef is immutable, delete and recreate the binding pointing at the new role:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  # Recreate it with roleRef.name set to the new role (d8:system:*, d8:subsystem:*, or d8:namespace:*).
  

  XXXuser-authz-controller renders this binding from spec.additionalRoles of the ClusterAuthorizationRule XXX and creates it again when it is deleted. Replace XXX with the new role name in spec.additionalRoles of that ClusterAuthorizationRule instead.XXXIf a project template renders this binding, replace the role name in the ProjectTemplate instead, because multitenancy-manager renders the binding again from it.XXXXXXXXX

  In the new model the rights for user-authn and user-authzXXX and the right to assign rolesXXX come with d8:subsystem:iam:XXX, not with d8:subsystem:security:XXX. To keep them, replace this binding with two bindings with the same subjects, one to d8:subsystem:iam:XXX and one to d8:subsystem:security:XXX. d8:subsystem:iam:XXX also grants the rights for multitenancy-manager.XXX

  The list of subsystems changed as well. deckhouse, infrastructure and kubernetes were merged into cluster, networking became network, the user-authz and user-authn modules moved from security to iam, and multitenancy-manager moved from deckhouse to iam. The aliases of the old subsystem roles select the capabilities of the modules these roles covered, and the old subsystem labels that modules released on their own may still carry. The capabilities themselves changed in this release, so the rights of the binding are close to the ones it had before the upgrade, not the same. The replacement d8:subsystem:cluster:level covers the whole cluster subsystem and grants more, and so do the roles of the other subsystems that the description of the alias names, so check which roles the subjects of the binding need. The user-authz documentation lists the remaining differences.

  A ClusterRoleBinding to a deprecated d8:manage:* name also keeps the RoleBindings d8:use:use role:binding:name that the previous release created for it in the namespaces of the modules of its old subsystem (d8-cloud-provider-*, d8-monitoring and the rest). The use role is viewer for a viewer alias and user for a manager alias. When the binding is recreated on the new role name, they are replaced by RoleBindings to the namespace role of the new role in the namespaces of the modules of the new subsystem.

  Note: the new d8:namespace:admin no longer grants serviceaccounts/token creation or impersonation — these disaster-grade actions now require a superadmin role by design.

• D8UserAuthzForeignAggregationLabel CE S6
  ClusterRole NAME carries the XXX aggregation label without being a custom capability.

  The ClusterRole NAME carries the rbac.deckhouse.io/aggregate-to-XXX-as label but is not labelled rbac.deckhouse.io/kind: custom-capabilityXXX (its kind label is XXX)XXX. The aggregation controller reads nothing but that label, so it merges the rules of this role into XXXthe platform d8:XXX:* rolesXXXthe deprecated d8:manage:all:* and d8:use:role:* roles and every custom role that selects the labelXXXthe d8:subsystem:XXX:* roles, if a subsystem has this name, and into every custom role that selects the labelXXX. Every holder of these roles in the cluster gets the rules, outside the role model’s contract.

  For the aggregation label of the system scope, of the namespace or project scope, or of a built-in subsystem, the rbacv2-cluster-roles.deckhouse.io webhook refuses such an object, and every UPDATE of this role, including a GitOps re-apply, is denied until it is fixed. Choose one:

  • If the role is meant to extend the platform roles, replace it with a custom capability. The name of an object cannot be changed, so create d8:custom:scope-capability:name with the same rules and aggregation label, the rbac.deckhouse.io/kind: custom-capability and rbac.deckhouse.io/scope: scope labels, and no aggregationRule, and then delete this role. Deckhouse reads rbac.deckhouse.io/namespace only on the ClusterRoles it ships: if this role has that label, neither it nor its replacement gives RoleBinding objects in that namespace, so bind the subjects to a namespace role there with a RoleBinding. A capability of the old scheme with the rbac.deckhouse.io/kind: use label and rbac.deckhouse.io/aggregate-to-kubernetes-as: LEVEL or rbac.deckhouse.io/aggregate-to-all-as: LEVEL extended the namespace roles, so its replacement is d8:custom:namespace-capability:name with rbac.deckhouse.io/scope: namespace and rbac.deckhouse.io/aggregate-to-namespace-as: LEVEL instead. For the commands, refer to the corresponding section in the user-authz module FAQ.
  • Otherwise remove the rbac.deckhouse.io/aggregate-to-*-as labels from it.

  List every such role:

  d8 k get clusterroles -o json | jq -r '.items[] | select((.metadata.labels // {}) | to_entries | any(.key | startswith("rbac.deckhouse.io/aggregate-to-"))) | select((.metadata.labels.heritage // "") != "deckhouse") | select((.metadata.labels["rbac.deckhouse.io/kind"] // "") != "custom-capability") | .metadata.name'
  

• D8UserAuthzLegacyRBACv2CustomRoleFound CE S7
  The custom role NAME has lost access it gave through the labels of an earlier role model.

  The ClusterRole NAME has no heritage: deckhouse label and has an aggregationRule field, and one of the following holds:

  • A selector of its aggregationRule field requires the rbac.deckhouse.io/kind: manage or use label of the experimental role model. The built-in capabilities no longer carry these labels, so such a selector matches none of them. The Kubernetes aggregation controller keeps from it only the rules of the custom capabilities that still carry the old labels, and the subjects of the role have lost the other permissions it gave. Custom roles get no compatibility aliases.
  • The role has the rbac.deckhouse.io/use-role label without rbac.deckhouse.io/scope: system or subsystem. Deckhouse creates the RoleBinding objects for the subjects of a role in the namespaces of the modules only for roles with that scope, so it has deleted the ones it had created for this role, even if its selectors are already rewritten.
  • A selector of its aggregationRule field requires the aggregation label of a lineage that no built-in role collects, such as rbac.deckhouse.io/aggregate-to-all-as of the old scheme, the label of a subsystem the role model no longer has, or the label of a subsystem a module has left. The modules label their capabilities only with the lineages that the built-in roles collect, so such a selector matches none of the built-in capabilities either. A selector by a lineage that a custom capability carries is not counted, because it still gets the rules of that capability. A custom capability here is a ClusterRole without the heritage: deckhouse label and without an aggregationRule field. Neither is a selector by the lineage of the subsystem in the rbac.deckhouse.io/subsystem label of the role, because the custom capabilities of that subsystem carry it. Neither exception applies to the all, deckhouse, infrastructure, kubernetes and networking lineages, which do not count as custom ones. A compatibility alias of an old role, a ClusterRole with the rbac.deckhouse.io/deprecated: "true" label, does not count as a built-in role that collects a lineage, because it collects the lineage of an old subsystem only for the bindings to the old name.

  Repair the access in this order:

  1. Create a role of the new scheme next to this one. Give it the d8:custom: name prefix and the rbac.deckhouse.io/kind: custom-role label, and select the capabilities by the labels of the new scheme, without rbac.deckhouse.io/kind. Select the capabilities that a selector by a lineage no built-in role collects was meant for by their rbac.deckhouse.io/capability labels, or by rbac.deckhouse.io/scope: system and the module label of their module. If this role has the rbac.deckhouse.io/use-role label, keep the label on the new role and add rbac.deckhouse.io/scope: subsystem with rbac.deckhouse.io/subsystem: SUBSYSTEM, or rbac.deckhouse.io/scope: system, so that Deckhouse creates the RoleBinding objects in the namespaces of the modules again.
  2. Create capabilities of the new scheme in place of the custom capabilities of the old scheme that this role aggregates. The D8UserAuthzForeignAggregationLabel alert names them.
  3. For every RoleBinding and ClusterRoleBinding object to this role, create one with the same subjects to the new role. If this role has the rbac.deckhouse.io/aggregate-to-all-as: LEVEL label, what it extended depends on its kind label. With rbac.deckhouse.io/kind: manage, its permissions also reached the subjects of d8:manage:all:LEVEL. A custom role cannot be aggregated into d8:system:LEVEL, so bind the new role to those subjects with a ClusterRoleBinding. With rbac.deckhouse.io/kind: use, they reached the subjects of d8:use:role:LEVEL in every namespace, so move the permissions into a namespace capability with rbac.deckhouse.io/aggregate-to-namespace-as: LEVEL, which extends d8:namespace:LEVEL.
  4. Check the permissions of the subjects with d8 k auth can-i VERB RESOURCE --as=USER_NAME.
  5. Delete the old bindings, this role and the old capabilities.

  For the commands, refer to the corresponding section in the user-authz module FAQ.

  List all such roles:

  d8 k get clusterroles -o json | jq -r '
    def lineages: [((.matchLabels // {}) | keys[]), (.matchExpressions[]? | select(.operator == "In" or .operator == "Exists") | .key)]
      | map(capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage);
    ([.items[] | select(.metadata.labels.heritage == "deckhouse" and .metadata.labels["rbac.deckhouse.io/deprecated"] != "true")
          | .aggregationRule.clusterRoleSelectors[]? | lineages[]] + ["system", "namespace", "project"]) as $collected
    | ([.items[] | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule == null) | (.metadata.labels // {}) | keys[]
          | capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage]) as $carried
    | .items[]
    | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule != null)
    | (.metadata.labels["rbac.deckhouse.io/subsystem"] // "") as $own
    | select(
        (
          ((.metadata.labels["rbac.deckhouse.io/use-role"] // "") != "")
          and (((.metadata.labels["rbac.deckhouse.io/scope"] // "") | IN("system", "subsystem")) | not)
        )
        or (
          [
            .aggregationRule.clusterRoleSelectors[]?
            | (.matchLabels["rbac.deckhouse.io/kind"] // empty),
              (.matchExpressions[]? | select(.key == "rbac.deckhouse.io/kind" and .operator == "In") | .values[])
          ]
          | any(IN("manage", "use"))
        )
        or (
          [
            .aggregationRule.clusterRoleSelectors[]? | lineages[]
            | select((IN($collected[]) | not)
                and (IN("all", "deckhouse", "infrastructure", "kubernetes", "networking") or (. != $own and (IN($carried[]) | not))))
          ]
          | length > 0
        )
      )
    | .metadata.name'
  

• D8UserAuthzLegacyUseCapability CE S7
  ClusterRole NAME carries the use labels of the previous scheme, which the namespace and project roles do not collect.

  The ClusterRole NAMEXXX of the MODULE_NAME moduleXXX carries the rbac.deckhouse.io/kind: use and rbac.deckhouse.io/aggregate-to-XXX-as: XXX labelsXXX of the use roles that a module with a subsystem of its own shipped in the previous schemeXXX of a use capability of the previous schemeXXX. rbac.deckhouse.io/kind: use is the label of a use capability of the previous schemeXXX. XXXThe moduleXXXThe capabilityXXX is still on that scheme.

  XXXThe d8:namespace:* and d8:project:* roles do not collect this label. Only the deprecated d8:use:role:* aliases still collect it, and they will be removed in the next release. A RoleBinding moved from d8:use:role:XXX to d8:namespace:XXX and a project role therefore do not get the rules of this ClusterRole.XXXNo role collects a ClusterRole with rbac.deckhouse.io/kind: use and this label. The d8:system:* and d8:subsystem:* roles and the deprecated d8:manage:* aliases leave out rbac.deckhouse.io/kind: use, and the d8:namespace:*, d8:project:* and d8:use:role:* roles do not select this label. Only a binding to this ClusterRole itself gives its rules.XXX

  Choose one:

  • If the ClusterRole belongs to a module, update the module to a release that labels its namespace capabilities with rbac.deckhouse.io/kind: capability and rbac.deckhouse.io/aggregate-to-namespace-as. Until then, give the subjects that need these rules a RoleBinding to NAME in their namespace, next to their RoleBinding to the d8:namespace:* role of their level. Neither role is a deprecated alias, so these RoleBinding objects do not hold the next release. In a project namespace, the cluster-objects-grants-validator.multitenancy.deckhouse.io webhook admits a RoleBinding to NAME only if the ClusterRole has the rbac.deckhouse.io/delegatable: "true" labelXXX, which the use capabilities of the previous scheme do not haveXXX.XXX Where the webhook refuses it, keep the RoleBinding to the d8:use:role:* alias: it keeps these rules, but the D8UserAuthzDeprecatedRBACv2RoleInUse alert reports it, and the next release, which removes the aliases, stays pending until the module is updated and the RoleBinding is moved to the d8:namespace:* role.XXX Where the webhook refuses it, the subjects get these rules only once the module is updated.XXX
  • If it is a custom capability, replace it with d8:custom:namespace-capability:NAME with the same rules and the rbac.deckhouse.io/kind: custom-capability, rbac.deckhouse.io/scope: namespace and rbac.deckhouse.io/aggregate-to-namespace-as: XXX labels. For the commands, refer to the corresponding section in the user-authz module FAQ.

  List every such ClusterRole:

  d8 k get clusterroles -l rbac.deckhouse.io/kind=use -o json | jq -r '.items[] | select(.metadata.labels | keys | map(capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage) | any(IN("kubernetes", "all", "system", "iam", "security", "cluster", "delivery", "network", "storage", "observability", "managed-services", "deckhouse", "infrastructure", "networking"))) | .metadata.name'
  

• D8UserAuthzPermissionBrowserUnavailable CE S6
  Permission Browser API server has unavailable replicas.

  Deckhouse has detected unavailable replicas in the permission-browser-apiserver deployment in the d8-user-authz namespace. Permission Browser requests can fail while this alert is active.

  Check deployment and pod states:

  d8 k -n d8-user-authz get deploy permission-browser-apiserver
  d8 k -n d8-user-authz get pods -l app=permission-browser-apiserver
  

• D8UserAuthzReleaseSlow CE S6
  The user-authz module release repeatedly takes more than 5 minutes.

  Applying the user-authz Helm release took more than 5 minutes on average over the last hour, and it was applied at least twice in that hour, so this is not the single slow pass of a cluster bootstrap or of the one-time migration to user-authz-controller. A release that does not finish within the module release timeout (20 minutes) fails, changes to ClusterAuthorizationRule and AuthorizationRule objects stop being applied and platform updates stall (the D8DeckhouseCouldNotRunModule alert fires in that case).

  The release itself no longer grows with the number of rules: bindings are managed by user-authz-controller, and custom ClusterRoles are aggregated. A slow release therefore points outside the module: an API server that answers slowly, an object stuck with a finalizer, a Deckhouse Pod short of CPU, or the release engine itself. Check what the release is waiting on:

  d8 s queue list
  d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'
  

  If the cluster has just been updated to the release that moved the bindings to the controller, one slow pass is expected and does not raise this alert by itself; if it fires anyway, the pass is being retried. It is proportional to the number of bindings and can exceed the timeout on clusters with more than 5000 of them; the module FAQ describes what to do in that case.

• D8UserAuthzRenderSlow CE S7
  Rendering the user-authz module templates repeatedly takes more than 30 seconds.

  Rendering the templates of the user-authz module took more than 30 seconds on average over the last 30 minutes, across at least three renders; normally one render takes about a second. Rendering is CPU work inside the Deckhouse Pod and does not depend on the number of ClusterAuthorizationRule or AuthorizationRule objects, so the cause is the node, not the module: most often the Deckhouse Pod is short of CPU (CPU steal on a virtual machine, a node running at its limits, a bootstrap that starts every module at once).

  A single slow render during a cluster bootstrap does not raise this alert by itself. If it fires, compare the module with the others and look at the node:

  topk(10, increase(deckhouse_helm_operation_seconds_sum{operation="template"}[30m]) / increase(deckhouse_helm_operation_seconds_count{operation="template"}[30m]))
  rate(node_cpu_seconds_total{mode="steal"}[5m])
  

  d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'
  

• D8UserAuthzRetiredAggregationLabel CE S7
  ClusterRole NAME aggregates by the label of the retired XXX subsystem.

  The ClusterRole NAMEXXX of the MODULE_NAME moduleXXX carries the rbac.deckhouse.io/aggregate-to-XXX-as label.

  The role model has no XXX subsystem any more. deckhouse, infrastructure and kubernetes were merged into cluster, and networking was renamed to network. No d8:system:* or d8:subsystem:* role collects this label, so the holders of these roles do not get the rules of this ClusterRole. Only the deprecated d8:manage:* aliases still collect it, and they will be removed in the next release.

  Choose one:

  • If the ClusterRole belongs to a module, update the module to a release that labels its capabilities with the lineage of its new subsystem. For most modules of the old subsystem that is rbac.deckhouse.io/aggregate-to-XXX-as. The subsystem of each module is listed in the user-authz documentation.
  • If it is a custom capability, replace the label with rbac.deckhouse.io/aggregate-to-XXX-as or with the label of the subsystem the capability belongs to now.

  Until then, a subject bound to a new role gets the rules of such a capability of a module only through a custom role. Create a d8:custom:system:NAME ClusterRole with the rbac.deckhouse.io/kind: custom-role and rbac.deckhouse.io/scope: system labels and bind it to the subjects with a ClusterRoleBinding. A capability of the previous scheme has no rbac.deckhouse.io/capability label, so the role selects the capabilities of the module by the module: XXXMODULE_NAMEXXXMODULEXXX and rbac.deckhouse.io/kind: manage labels. Such a selector takes the capabilities of every level, so add rbac.deckhouse.io/aggregate-to-XXX-as: viewer to it for a role that only views. The D8UserAuthzLegacyRBACv2CustomRoleFound alert names this role, because it selects by a label of the previous scheme. Once the module carries the new labels, select its capabilities by their rbac.deckhouse.io/capability label instead. A capability that already carries rbac.deckhouse.io/kind: capability is selected by its rbac.deckhouse.io/capability label right away.

  List every such ClusterRole:

  d8 k get clusterroles -o json | jq -r '.items[] | select((.metadata.labels["rbac.deckhouse.io/kind"] // "") != "use") | select((.metadata.labels // {}) | keys | any(IN("rbac.deckhouse.io/aggregate-to-deckhouse-as", "rbac.deckhouse.io/aggregate-to-infrastructure-as", "rbac.deckhouse.io/aggregate-to-kubernetes-as", "rbac.deckhouse.io/aggregate-to-networking-as"))) | .metadata.name'
  

• D8UserAuthzRevocationCheckNotSynced CE S6
  The revocation check of the user-authz webhook has not loaded the revocations.

  For more than 10 minutes the user-authz-webhook instance POD_NAME has not loaded the Revocation objects of the user-authn module (the objects in d8-user-authn that record revoked user credentials): it cannot list them, or cannot find out whether the API server serves them.

  Until it loads them, the instance has no opinion about revoked credentials: on the master node of this instance, kubectl with the token of a locked or logged-out user keeps working until the token expires. Other authorization decisions are not affected.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs POD_NAME -c webhook --tail=100 | grep -i revocation
  d8 k auth can-i list revocations.dex.coreos.com -n d8-user-authn --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzRevocationCheckStale CE S6
  The revocation check of the user-authz webhook no longer sees new revocations.

  For more than 15 minutes the user-authz-webhook instance POD_NAME has been unable to watch the Revocation objects of the user-authn module (the objects in d8-user-authn that record revoked user credentials).

  The instance keeps refusing the Dex tokens it already knows to be revoked, but a revocation made since its watch broke does not reach it: on the master node of this instance, kubectl with the token of a locked or logged-out user keeps working until the token expires.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs POD_NAME -c webhook --tail=100 | grep -i revocation
  d8 k auth can-i watch revocations.dex.coreos.com -n d8-user-authn --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzRuleNeedsMultiTenancy CE S6
  A ClusterAuthorizationRule asks for namespace limits that nothing enforces.

  The ClusterAuthorizationRule NAME sets XXX, but the enableMultiTenancy setting of the user-authz module is off, so the authorization webhook that enforces those options is not deployed.

  The rule is not partially applied. It is applied without its limits: the subjects named in it hold its access level in every namespace of the cluster, the system ones included.

  Either enable multi-tenancy, or remove the options from the rule:

  d8 k get clusterauthorizationrule NAME -o yaml
  d8 k get moduleconfig user-authz -o jsonpath='{.spec.settings.enableMultiTenancy}'
  

• D8UserAuthzRulePropagationLag CE S6
  One user-authz webhook instance stopped picking up rule changes.

  One instance of user-authz-webhook has not updated its rules for more than an hour while another instance has. The rules are changing in the cluster and this instance does not see the changes.

  Find the instance and check its log:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100
  

  An instance whose watch keeps failing becomes not ready, so check the readiness of the Pod as well.

• D8UserAuthzRulesNeedMultiTenancy CE S6
  More ClusterAuthorizationRules ask for namespace limits than the per-rule alert names.

  VALUE ClusterAuthorizationRules set limitNamespaces, namespaceSelector or allowAccessToSystemNamespaces while the enableMultiTenancy setting of the user-authz module is off. D8UserAuthzRuleNeedsMultiTenancy names the first 50 of them; this alert accounts for the rest.

  None of those rules is partially applied: the subjects named in each of them hold its access level in every namespace of the cluster, the system ones included.

  Check whether multi-tenancy was meant to be enabled, and list the affected rules:

  d8 k get moduleconfig user-authz -o jsonpath='{.spec.settings.enableMultiTenancy}'
  d8 k get clusterauthorizationrule -o json | jq -r '.items[] | select((.spec.limitNamespaces // [] | length > 0) or (.spec.namespaceSelector != null) or (.spec.allowAccessToSystemNamespaces == true)) | .metadata.name'
  

• D8UserAuthzWebhookDirectoryDiverged CE S6
  The masters disagree about the multi-tenancy rules.

  Instances of user-authz-webhook have been using different sets of ClusterAuthorizationRules for 10 minutes.

  Each master node authorizes the requests that reach it, so the same request is answered differently depending on the node: a user can be inside their namespace limits on one node and outside them on another.

  Find the instance that is behind and look at why:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100 | grep -i 'list/watch'
  

  A growing user_authz_webhook_rules_watch_errors_total on one instance points at its watch; a user_authz_webhook_rules_directory_updated_timestamp_seconds that has stopped moving points at rules that are no longer updated. Deleting the lagging Pod is the last resort: the DaemonSet recreates it, and on a cluster with a single master node every request that the authorization webhook has to answer is denied while the Pod restarts. Control plane components, kubelets, and the ServiceAccounts of the kube-system and d8-* namespaces keep working, because the webhook does not check them.

• D8UserAuthzWebhookRulesQuarantined CE S7
  A ClusterAuthorizationRule cannot be compiled by the webhook.

  For more than 10 minutes the user-authz-webhook has quarantined at least one ClusterAuthorizationRule: its limitNamespaces pattern or namespaceSelector does not compile, or the rule cannot be read at all.

  A rule with a broken filter is applied without that filter, so its subjects get a narrower scope than written and are denied where the filter was meant to allow. A rule that cannot be read is not applied at all, and its subjects are denied every namespace.

  Find the rule:

  d8 k get clusterauthorizationrules -o custom-columns=NAME:.metadata.name,LIMIT:.spec.limitNamespaces,SELECTOR:.spec.namespaceSelector
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100 | grep -i quarantin
  

• D8UserAuthzWebhookRulesWatchErrors CE S7
  The user-authz webhook cannot watch the ClusterAuthorizationRules.

  For more than 10 minutes the user-authz-webhook rules informer has been failing to watch the ClusterAuthorizationRules (a non-zero rate of watch errors). New or changed rules may not be reaching the webhook, so multi-tenancy decisions can lag behind what users configure.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100
  d8 k auth can-i watch clusterauthorizationrules.deckhouse.io --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzWebhookTargetDown CE S6
  Prometheus is unable to scrape the user-authz webhook.

  Prometheus cannot collect metrics from at least one instance of user-authz-webhook in the d8-user-authz namespace. One instance without metrics is enough for the alert to fire.

  If the instance is running and only its metrics are unreachable, it keeps answering authorization requests, but the other alerts of the module cannot see its state. If it is not running or has not read the rules yet, it answers 503 to every authorization request, and the API server denies the request, because the webhook is configured with failurePolicy: Deny.

  Check the DaemonSet, its metrics sidecar and the scrape target:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c kube-rbac-proxy --tail=50
  

Модуль vertical-pod-autoscaler

• VerticalPodAutoscalerDeprecatedUpdateModeAuto CE S9
  VerticalPodAutoscaler objects use the deprecated Auto update mode.

  There are VALUE VerticalPodAutoscaler object(s) in the cluster configured with the deprecated update mode (spec.updatePolicy.updateMode: Auto).

  The Auto update mode is deprecated. It currently behaves like Recreate (the VPA updater evicts pods to apply new resource requests) and its semantics are reserved to change in future versions, which makes cluster behavior ambiguous and potentially disruptive.

  Set the affected VPA object(s) to one of the supported update modes (spec.updatePolicy.updateMode):

  • InPlaceOrRecreate: Apply recommendations in place when possible, fall back to recreation (recommended).
  • Recreate: Evict and recreate pods to apply recommendations.
  • Initial: Apply only when pods are (re)created.
  • Off: Recommendations only, no changes to running pods.

  To list the affected objects, open the VPA Deprecated Update Mode Grafana dashboard, or run:

  kube_verticalpodautoscaler_spec_updatepolicy_updatemode{update_mode="Auto"} == 1