airbyte/docs/operator-guides/collecting-metrics.md

Collecting Metrics

Airbyte supports two ways to collect metrics - using datadog or open telemetry. Fill in METRIC_CLIENT field in .env file to get started!

Prerequisite: In order to get metrics from airbyte we need to deploy a container / pod called metrics-reporter like below

airbyte-metrics: 
  image: airbyte/metrics-reporter:${VERSION} 
  container_name: airbyte-metrics 
  environment: 
    - METRIC_CLIENT=${METRIC_CLIENT} 
    - OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}

Open Telemetry

1. In .env change METRIC_CLIENT to otel.
2. Similarly, configure OTEL_COLLECTOR_ENDPOINT to tell Airbyte where to send metrics RPC to.

Example

Run Opentelemetry and Airbyte locally

In this example we will run Airbyte locally along with an Open Telemetry Collector. The Open telemetry collector will expose port 4317 to the localhost as the receiving endpoint.

Steps:

1. Setting up Open telemetry. In this example we will use the repository from opentelemetry-java-docs. Run the following commands to have it up and running.

  git clone https://github.com/open-telemetry/opentelemetry-java-docs
  cd opentelemetry-java-docs/otlp/docker
  docker-compose up

2. Configure Airbyte .env file.
   1. Change METRIC_CLIENT to otel to indicate Airbyte to use Open telemetry to emit metric data.
   2. Change OTEL_COLLECTOR_ENDPOINT to "http://host.docker.internal:4317" because Open Telemetry Collector has enabled port forward from localhost:4317 to container port 4317. To send data to Collector container port 4317, we want to need to export data to physical machine's localhost:4317, which in docker will be represented as http://host.docker.internal:4317.

   Do not use localhost:4317 or you will send data to the same container where Airbyte Worker is running.

   3. Start Airbyte server by running docker compose up under airbyte repository. Go to localhost:8000 to visit Airbyte and start a sync, then go to localhost:9090 to access Prometheus - you should be able to see the metrics there. Alternatively,

Run Opentelemetry and Airbyte on kubernetes

Prerequisite: Read https://github.com/airbytehq/airbyte/blob/master/docs/deploying-airbyte/on-kubernetes.md to understand how to start Airbyte on Kubernetes

We will use stable in this example.

Steps:

1. Run open telemetry collector in the same Kubernetes context. Here we follow example in OpenTelemetry doc
2. edit kube/overlays/stable/.env and add the following lines:

METRIC_CLIENT=otel
OTEL_COLLECTOR_ENDPOINT=<address>

If you started open telemetry collector in the link above, the address should be http://otel-collector:4317. Note the format - unlike the base .env, there is no quote in .env file under kubernetes.

Tutorial

Deploy the airbyte metric pod :

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-metrics
  namespace: {{ .Release.Namespace }}
  labels: {{ set . "component" "metrics" | include "labels" | nindent 4 }}
spec:
  selector:
    matchLabels: {{ set . "component" "metrics" | include "labels" | nindent 6 }}
  template:
    metadata:
      labels: {{ set . "component" "metrics" | include "labels" | nindent 8 }}
    spec:
      containers:
      - name: airbyte-metrics
        image: "airbyte/metrics-reporter:latest"
        imagePullPolicy: IfNotPresent
        env:
        - name: AIRBYTE_VERSION
          value: latest
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: {{ include "airbyte.database.secret.name" . }}
              key: DATABASE_PASSWORD
        - name: DATABASE_URL
          value: {{ include "airbyte.database.url" . | quote }}
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Name }}-secrets
              key: DATABASE_USER
        - name: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
          value: 0.35.15.001
        - name: METRIC_CLIENT
          value: otel
        - name: OTEL_COLLECTOR_ENDPOINT
          value: http://otel-collector:4317

Deploy an Open telemetry pod like below :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: {{ .Release.Namespace }}
  labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }} 
spec:
  selector:
    matchLabels: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }} 
  replicas: 1
  template:
    metadata:
      labels: {{ set . "component" "otel-collector" | include "labels" | nindent 8 }} 
    spec:
      containers:
      - command:
          - "/otelcol"
          - "--config=/conf/otel-collector-config.yaml"
        image: "otel/opentelemetry-collector:latest"
        name: otel-collector
        ports:
        - containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
        - containerPort: 8889 # Port for Prometheus instance
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
        - configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: config

WIth this Config Map :

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  namespace: {{ .Release.Namespace }}
  labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }} 
data:
  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
           endpoint: "0.0.0.0:4317"
    processors:
      batch:
      memory_limiter:
        limit_mib: 1500
        spike_limit_mib: 512
        check_interval: 5s
    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"
        namespace: "default"
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheus]

Then we need a service to be able to access both open telemetry GRPC and Prometheus

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: {{ .Release.Namespace }}
  labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }} 
spec:
  ports:
  - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: prometheus
    port: 8889
  selector: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }}

And finally We can add a service monitor to receive metrics in prometheus and optionally add some prometheus rules to generate alerts. You can replace with your prometheus name.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{ set . "component" "metrics" | include "labels" | nindent 4 }}
    prometheus: <your_prometheus_name>
spec:
  endpoints:
    - interval: 30s
      port: prometheus
      path: /metrics
      relabelings:
      - action: labeldrop
        regex: (service|endpoint|namespace|container)
  selector:
    matchLabels: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }}

One rule example :

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: {{ .Release.Name }}-rules
  namespace: {{ .Release.Namespace }}
  labels:
    {{ set . "component" "prometheus-rules" | include "labels" | nindent 4 }}
    prometheus: <your_prometheus_name>
spec:
  groups:
    - name: airbyte
      rules:
        - alert: AirbyteJobFail
          for: 0m
          expr: min(airbyte_job_failed_by_release_stage) > 0
          labels:
            priority: P2
          annotations:
            summary: {{ `"An Airbyte Job has failed"` }}     

Datadog

The set up for Datadog is pretty straightforward.

Set two env vars:

1. METRIC_CLIENT to datadog.
2. PUBLISH_METRICS to true.

Configure two additional env vars to the Datadog endpoint:

1. Set DD_AGENT_HOST to the IP where the Datadog agent is running.
2. Set DD_DOGSTATSD_PORT to the port the Datadog agent is using.

Metrics

Visit OssMetricsRegistry.java to get a complete list of metrics Airbyte is sending.

Additional information

Suppose you are looking for a non-production way of collecting metrics with dbt and Metabase, the tutorial Airbyte Monitoring with dbt and Metabase by accessing Airbyte's Postgres DB. The source code is open on airbytehq/open-data-stack. Think of it as an exploratory for data analysts and data engineers of building a dashboard on top of the existing Airbyte Postgres database versus the Prometheus more for DevOps engineers in production.