I am trying to add labels in Grafana for my Kubernetes logs. I am using OpenTelemetry Collector Agent to send logs to Loki, and I want to include labels like service, container_name, and pod_name.

Currently, in Grafana, I can query logs, but I don't see these labels available in the label selector.

Here is my Helm values.yaml for the OpenTelemetry Collector Agent:

mode: daemonset

image:

  repository: "otel/opentelemetry-collector-contrib"
  tag: 0.119.0
  pullPolicy: IfNotPresent

imagePullSecrets: 
  - name: "github-registry-credentials"

extraVolumes:
  - name: varlog
    hostPath:
      path: /var/log

extraVolumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true

config:
  receivers:
    filelog:
      poll_interval: 10s
      include:
        - /var/log/pods/*/*/*.log
      start_at: beginning
      include_file_path: true
      include_file_name: true
      operators:
        - type: regex_parser
          id: extract_metadata
          regex: "/var/log/pods/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_[^/]+/(?P<container_name>.+).log"
          parse_from: attributes["log.file.path"]
          on_error: drop  

        - type: add
          id: add_service_name
          field: resource["service.name"]
          value: 'EXPR(attributes["namespace"])'

        - type: move
          from: attributes["pod_name"]
          to: resource["pod.name"]
        
        - type: move
          id: add_container_name
          to: resource["container.name"]
          from: attributes["container_name"]

  processors:
    resource:
      attributes:
        - key: pod_name
          from_attribute: pod_name
          action: upsert

    attributes/add_labels:
      actions:
          
        - action: insert
          key: cluster
          value: cluster-name
    filter/service_name:
      logs:
        exclude:
          match_type: strict
          resource_attributes:
            - key: service.name
              value: "unknown_service"

    transform/example:
      log_statements:
        - context: log
          statements:
            - set(resource.attributes["pod.name"], attributes["pod_name"])

    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    batch:
      send_batch_size: 1000
      timeout: 120s
    k8sattributes:
        auth_type: serviceAccount

  exporters:
    otlphttp/loki:
      logs_endpoint: 
      tls:
        insecure: true
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 60s
        max_elapsed_time: 5m
      sending_queue:
        enabled: true
        num_consumers: 10
        queue_size: 5000

  service:
    extensions:
      - health_check
    pipelines:
      logs:
        receivers: [filelog]
        processors: [attributes/add_labels, filter/service_name,k8sattributes,transform/example, resource, batch]
        exporters: [otlphttp/loki]

resources:
  limits:
    cpu: 1
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

And i see only this

I am trying to add labels in Grafana for my Kubernetes logs. I am using OpenTelemetry Collector Agent to send logs to Loki, and I want to include labels like service, container_name, and pod_name.

Currently, in Grafana, I can query logs, but I don't see these labels available in the label selector.

Here is my Helm values.yaml for the OpenTelemetry Collector Agent:

mode: daemonset

image:

  repository: "otel/opentelemetry-collector-contrib"
  tag: 0.119.0
  pullPolicy: IfNotPresent

imagePullSecrets: 
  - name: "github-registry-credentials"

extraVolumes:
  - name: varlog
    hostPath:
      path: /var/log

extraVolumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true

config:
  receivers:
    filelog:
      poll_interval: 10s
      include:
        - /var/log/pods/*/*/*.log
      start_at: beginning
      include_file_path: true
      include_file_name: true
      operators:
        - type: regex_parser
          id: extract_metadata
          regex: "/var/log/pods/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_[^/]+/(?P<container_name>.+).log"
          parse_from: attributes["log.file.path"]
          on_error: drop  

        - type: add
          id: add_service_name
          field: resource["service.name"]
          value: 'EXPR(attributes["namespace"])'

        - type: move
          from: attributes["pod_name"]
          to: resource["pod.name"]
        
        - type: move
          id: add_container_name
          to: resource["container.name"]
          from: attributes["container_name"]

  processors:
    resource:
      attributes:
        - key: pod_name
          from_attribute: pod_name
          action: upsert

    attributes/add_labels:
      actions:
          
        - action: insert
          key: cluster
          value: cluster-name
    filter/service_name:
      logs:
        exclude:
          match_type: strict
          resource_attributes:
            - key: service.name
              value: "unknown_service"

    transform/example:
      log_statements:
        - context: log
          statements:
            - set(resource.attributes["pod.name"], attributes["pod_name"])

    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    batch:
      send_batch_size: 1000
      timeout: 120s
    k8sattributes:
        auth_type: serviceAccount

  exporters:
    otlphttp/loki:
      logs_endpoint: https://loki-gateway.domain/otlp/v1/logs
      tls:
        insecure: true
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 60s
        max_elapsed_time: 5m
      sending_queue:
        enabled: true
        num_consumers: 10
        queue_size: 5000

  service:
    extensions:
      - health_check
    pipelines:
      logs:
        receivers: [filelog]
        processors: [attributes/add_labels, filter/service_name,k8sattributes,transform/example, resource, batch]
        exporters: [otlphttp/loki]

resources:
  limits:
    cpu: 1
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

And i see only this

• grafana
• grafana-loki
• otel

1 Answer 1

Loki 3+ has predefined list of OTEL resource attributes, which are stored as a labels:

otlp_config:
  # List of default otlp resource attributes to be picked as index labels
  # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
  [default_resource_attributes_as_index_labels: <list of strings> | default = [service.name service.namespace service.instance.id deployment.environment cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.pod.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]]

If resource attribute is not in this list, then it is stored as a metadata only.

Yours options:

1. name that resource attribute pod.name as resource (not log) attribute k8s.pod.name

   or

2. modify that Loki default_resource_attributes_as_index_labels config and add also your's pod.name

I would recommend 1.) because that follows OTEL K8S semantic conventions.

For the users, who just copy&paste config snippets and don't read documentation: copy this copy&paste snippet into your Loki config file:

distributor:
  otlp_config:
    default_resource_attributes_as_index_labels:
      - "pod.name"