Getting logs from all cluster Pods and sending them to Loki | Чтение логов из всех подов кластера и направление их в Loki |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: all-logs spec: type: KubernetesPods destinationRefs:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: all-logs spec: type: KubernetesPods destinationRefs:
|
Reading Pod logs from a specified namespace with a specified label and redirecting to Loki and Elasticsearch | Чтение логов подов из указанного namespace с указанным label и перенаправление одновременно в Loki и Elasticsearch |
Reading logs from | Чтение логов подов из namespace |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: whispers-booking-logs spec: type: KubernetesPods kubernetesPods: namespaceSelector: matchNames:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: whispers-booking-logs spec: type: KubernetesPods kubernetesPods: namespaceSelector: matchNames:
|
Creating a source in namespace and reading logs of all Pods in that NS with forwarding them to Loki | Создание source в namespace и чтение логов всех подов в этом NS с направлением их в Loki |
Namespaced pipeline - reading logs from | Следующий pipeline создает source в namespace |
yaml apiVersion: deckhouse.io/v1alpha1 kind: PodLoggingConfig metadata: name: whispers-logs namespace: tests-whispers spec: clusterDestinationRefs:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: PodLoggingConfig metadata: name: whispers-logs namespace: tests-whispers spec: clusterDestinationRefs:
|
Reading only Pods in the specified namespace and having a certain label | Чтение только подов в указанном namespace и с определенным label |
Read logs from Pods with label | Пример чтения только подов, имеющих label |
yaml apiVersion: deckhouse.io/v1alpha1 kind: PodLoggingConfig metadata: name: whispers-logs namespace: tests-whispers spec: labelSelector: matchLabels: app: booking clusterDestinationRefs:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: PodLoggingConfig metadata: name: whispers-logs namespace: tests-whispers spec: labelSelector: matchLabels: app: booking clusterDestinationRefs:
|
Migration from Promtail to Log-Shipper | Переход с Promtail на Log-Shipper |
Path | В ранее используемом URL Loki требуется убрать путь |
Vector will add this PATH automatically during working with Loki destination. | Vector сам добавит этот путь при работе с Loki. |
Working with Grafana Cloud | Работа с Grafana Cloud |
This documentation expects that you have created API key. | Данная документация подразумевает, что у вас уже создан ключ API. |
Для начала вам потребуется закодировать в base64 ваш токен доступа к Grafana Cloud. | |
Firstly you should encode your token with base64. | |
bash
echo -n “ | bash
echo -n “ |
Then you can create ClusterLogDestination | Затем нужно создать ClusterLogDestination |
yaml
apiVersion: deckhouse.io/v1alpha1
kind: ClusterLogDestination
metadata:
name: loki-storage
spec:
loki:
auth:
password: PFlPVVItR1JBRkFOQUNMT1VELVRPS0VOPg==
strategy: Basic
user: “ | yaml
apiVersion: deckhouse.io/v1alpha1
kind: ClusterLogDestination
metadata:
name: loki-storage
spec:
loki:
auth:
password: PFlPVVItR1JBRkFOQUNMT1VELVRPS0VOPg==
strategy: Basic
user: “ |
Now you can create PodLogginConfig or ClusterPodLoggingConfig and send logs to Grafana Cloud. | Теперь можно создать PodLogginConfig или ClusterPodLoggingConfig и отправлять логи в Grafana Cloud. |
Adding Loki source to Deckhouse Grafana | Добавление Loki в Deckhouse Grafana |
You can work with Loki from embedded to deckhouse Grafana. Just add GrafanaAdditionalDatasource | Вы можете работать с Loki из встроенной в Deckhouse Grafana. Достаточно добавить GrafanaAdditionalDatasource. |
yaml apiVersion: deckhouse.io/v1 kind: GrafanaAdditionalDatasource metadata: name: loki spec: access: Proxy basicAuth: false jsonData: maxLines: 5000 timeInterval: 30s type: loki url: http://loki.loki:3100 | yaml apiVersion: deckhouse.io/v1 kind: GrafanaAdditionalDatasource metadata: name: loki spec: access: Proxy basicAuth: false jsonData: maxLines: 5000 timeInterval: 30s type: loki url: http://loki.loki:3100 |
Elasticsearch < 6.X usage | Поддержка Elasticsearch < 6.X |
For Elasticsearch < 6.0 doc_type indexing should be set. Config should look like this: | Для Elasticsearch < 6.0 нужно включить поддержку doc_type индексов. Сделать это можно следующим образом: |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 docType: “myDocType” # Set any string here. It should not start with ‘_’ auth: strategy: Basic user: elastic password: c2VjcmV0IC1uCg== | yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 docType: “myDocType” # Укажите значение здесь. Оно не должно начинаться с ‘_’. auth: strategy: Basic user: elastic password: c2VjcmV0IC1uCg== |
Index template for Elasticsearch | Шаблон индекса для Elasticsearch |
It is possible to route logs to particular indexes based on metadata using index templating: | Существует возможность отправлять сообщения в определенные индексы на основе метаданных с помощью шаблонов индексов: |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 index: “k8s-{{ namespace }}-%F” | yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 index: “k8s-{{ namespace }}-%F” |
For the above example for each Kubernetes namespace a dedicated index in Elasticsearch will be created. | В приведенном выше примере для каждого пространства имен Kubernetes будет создан свой индекс в Elasticsearch. |
This feature works well combining with | Эта функция также хорошо работает в комбинации с |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 index: “k8s-{{ service }}-{{ namespace }}-%F” extraLabels: service: “{{ service_name }}” | yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: es-storage spec: type: Elasticsearch elasticsearch: endpoint: http://192.168.1.1:9200 index: “k8s-{{ service }}-{{ namespace }}-%F” extraLabels: service: “{{ service_name }}” |
|
|
Splunk integration | Пример интеграции со Splunk |
It is possible to send logs from Deckhouse to Splunk. | Существует возможность отсылать события из Deckhouse в Splunk. |
|
|
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: splunk spec: type: Splunk splunk: endpoint: https://prd-p-xxxxxx.splunkcloud.com:8088 token: xxxx-xxxx-xxxx index: logs tls: verifyCertificate: false verifyHostname: false | yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: splunk spec: type: Splunk splunk: endpoint: https://prd-p-xxxxxx.splunkcloud.com:8088 token: xxxx-xxxx-xxxx index: logs tls: verifyCertificate: false verifyHostname: false |
Splunk destination doesn’t support pod labels for indexes. Consider exporting necessary labels with the |
|
yaml extraLabels: pod_label_app: ‘{{ pod_labels.app }}’ | yaml extraLabels: pod_label_app: ‘{{ pod_labels.app }}’ |
Simple Logstash example | Простой пример Logstash |
To send logs to Logstash, the | Чтобы отправлять логи в Logstash, на стороне Logstash должен быть настроен входящий поток |
An example of the minimal Logstash configuration: | Пример минимальной конфигурации Logstash: |
hcl input { tcp { port => 12345 codec => json } } output { stdout { codec => json } } | hcl input { tcp { port => 12345 codec => json } } output { stdout { codec => json } } |
An example of the | Пример манифеста |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: logstash spec: type: Logstash logstash: endpoint: logstash.default:12345 | yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLogDestination metadata: name: logstash spec: type: Logstash logstash: endpoint: logstash.default:12345 |
Collect Kubernetes Events | Сбор событий Kubernetes |
Kubernetes Events can be collected by log-shipper if | События Kubernetes могут быть собраны log-shipper’ом, если |
Enable | Включите events-exporter, изменив параметры модуля |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: extended-monitoring spec: version: 1 settings: events: exporterEnabled: true | yaml apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: extended-monitoring spec: version: 1 settings: events: exporterEnabled: true |
Apply the following | Выложите в кластер следующий |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: kubernetes-events spec: type: KubernetesPods kubernetesPods: labelSelector: matchLabels: app: events-exporter namespaceSelector: matchNames:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: kubernetes-events spec: type: KubernetesPods kubernetesPods: labelSelector: matchLabels: app: events-exporter namespaceSelector: matchNames:
|
Log filters | Фильтрация логов |
Users can filter logs by applying two filters:
| Пользователи могут фильтровать логи, используя следующие фильтры:
|
Collect only logs of the
| Сборка логов только для контейнера
|
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: nginx-logs spec: type: KubernetesPods labelFilter:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: nginx-logs spec: type: KubernetesPods labelFilter:
|
Collect logs without strings
| Сборка логов без строки, содержащей
|
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: all-logs spec: type: KubernetesPods destinationRefs:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: all-logs spec: type: KubernetesPods destinationRefs:
|
Audit of kubelet actions | Аудит событий kubelet’а |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: kubelet-audit-logs spec: type: File file: include:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: kubelet-audit-logs spec: type: File file: include:
|
Deckhouse system logs | Системные логи Deckhouse |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: system-logs spec: type: File file: include:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: system-logs spec: type: File file: include:
|
If you need logs from only one or from a small group of a Pods, try to use the kubernetesPods settings to reduce the number of reading filed. Do not use highly grained filters to read logs from a single pod. | Если вам нужны только логи одного пода или малой группы подов, постарайтесь использовать настройки |
Collect logs from production namespaces using the namespace label selector option | Настройка сборки логов с продуктовых namespace’ов, используя опцию namespace label selector |
yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: production-logs spec: type: KubernetesPods kubernetesPods: namespaceSelector: labelSelector: matchLabels: environment: production destinationRefs:
| yaml apiVersion: deckhouse.io/v1alpha1 kind: ClusterLoggingConfig metadata: name: production-logs spec: type: KubernetesPods kubernetesPods: namespaceSelector: labelSelector: matchNames: environment: production destinationRefs:
|
Exclude Pods or namespaces with a label | Исключить поды и namespace’ы, используя label |
There is a preconfigured label to exclude particular namespaces or Pods: | Существует преднастроенный label для исключения определенных подов и namespace’ов: |
yamlapiVersion: v1 kind: Namespace metadata: name: test-namespace labels: log-shipper.deckhouse.io/exclude: “true” — apiVersion: apps/v1 kind: Deployment metadata: name: test-deployment spec: … template: metadata: labels: log-shipper.deckhouse.io/exclude: “true” | yamlapiVersion: v1 kind: Namespace metadata: name: test-namespace labels: log-shipper.deckhouse.io/exclude: “true” — apiVersion: apps/v1 kind: Deployment metadata: name: test-deployment spec: … template: metadata: labels: log-shipper.deckhouse.io/exclude: “true” |