Dapr Observability with OpenTelemetry and Dynatrace
How to achieve observability in Dapr using OpenTelemetry and integrate it with Dynatrace for application monitoring and tracing capabilities.
Dapr (Distributed Application Runtime) is an open-source framework that simplifies building microservices-based applications. It provides various features to improve the development experience, including observability, which allows developers to monitor and understand their application’s behavior.
In this blog post, we will explore how to achieve observability in Dapr using OpenTelemetry and integrate it with Dynatrace for application monitoring and tracing capabilities.
Understanding Dapr and OpenTelemetry
Dapr is designed to be observability-agnostic, allowing users to choose their preferred observability tooling. OpenTelemetry, a CNCF project, provides a vendor-agnostic framework for collecting telemetry data, including logs, metrics, and traces.
By integrating Dapr with OpenTelemetry, we can enhance our application’s observability capabilities.
Using the Standard Otel Collector in Dapr
Dapr leverages the standard OpenTelemetry Collector to collect and process telemetry data. The OpenTelemetry Collector acts as an intermediary, receiving telemetry data from various sources and exporting it to different targets.
In the case of Dapr, the Collector collects logs, traces, and metrics from Dapr sidecars and forwards them to the desired observability backend.
Setting up DAPR to send distributed tracing
To configure DAPR to send the traces, a DAPR configuration needs to be added. This config yaml includes the configuration resource for tracing, specifically the sampling rate and the endpoint address for Zipkin.
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
name: appconfig
namespace: default
spec:
tracing:
samplingRate: "1"
zipkin:
endpointAddress: "http://otel-collector.default.svc.cluster.local:9411/api/v2/spans"The above Dapr Configuration resource will enable tracing and export the traces to a Zipkin endpoint provided by the OpenTelemetry Collector.
Configure the OpenTelemetry Collector in Dapr
Create a Kubernetes ConfigMap to define the Collector’s configuration. The provided code snippet shows an example of the otel-collector-config.yaml file. It includes configurations for receivers (zipkin, otlp), processors (batch, memory_limiter), exporters (logging, otlphttp, zipkin), and service telemetry settings.
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
namespace: default
labels:
app: opentelemetry
component: otel-collector-conf
data:
otel-collector-config: |
receivers:
zipkin:
otlp:
protocols:
grpc:
http:
processors:
batch:
memory_limiter:
limit_mib: 1500
spike_limit_mib: 512
check_interval: 5s
extensions:
health_check:
zpages: {}
pprof:
endpoint: :1888
memory_ballast:
size_mib: 683
exporters:
logging:
loglevel: debug
otlphttp:
endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
headers:
Authorization: "Api-Token ${OTEL_EXPORTER_OTLP_TOKEN}"
zipkin:
endpoint: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
service:
telemetry:
logs:
level: debug
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp, zipkin]
processors: [batch]
exporters: [otlphttp, zipkin, logging]
logs:
receivers: [otlp]
processors: [batch]
exporters: [logging]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging]
---
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: default
labels:
app: opentelemetry
component: otel-collector
spec:
ports:
- name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
port: 4317
protocol: TCP
targetPort: 4317
- name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
port: 4318
protocol: TCP
targetPort: 4318
- name: metrics # Default endpoint for querying metrics.
port: 8888
- name: zipkin-traces
port: 9411
protocol: TCP
selector:
component: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
namespace: default
labels:
app: opentelemetry
component: otel-collector
spec:
selector:
matchLabels:
app: opentelemetry
component: otel-collector
minReadySeconds: 5
progressDeadlineSeconds: 120
replicas: 1 #TODO - adjust this to your own requirements
template:
metadata:
labels:
app: opentelemetry
component: otel-collector
spec:
containers:
- image: otel/opentelemetry-collector-contrib:0.70.0
name: otel-collector
args:
- "--config=/conf/otel-collector-config.yaml"
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
ports:
- containerPort: 55679 # Default endpoint for ZPages.
- containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
- containerPort: 9411 # Default endpoint for Zipkin receiver.
- containerPort: 8888 # Default endpoint for querying metrics.
volumeMounts:
- name: otel-collector-config-vol
mountPath: /conf
- name: dynatrace-api-credentials
mountPath: /secrets
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
valueFrom:
secretKeyRef:
name: dynatrace-api-credentials
key: DT_API_ENDPOINT
- name: OTEL_EXPORTER_OTLP_TOKEN
valueFrom:
secretKeyRef:
name: dynatrace-api-credentials
key: DT_API_TOKEN
volumes:
- configMap:
name: otel-collector-conf
items:
- key: otel-collector-config
path: otel-collector-config.yaml
name: otel-collector-config-vol
- name: dynatrace-api-credentials
secret:
defaultMode: 420
secretName: dynatrace-api-credentialsThe above code snippet also creates a Kubernetes Deployment and Service. The Deployment ensures the Collector is running within our cluster, while the Service allows communication with the Collector’s various endpoints.
To understand more about the different components(receiver, processor, exporter) in above config, refer this link.
Integrating with Dynatrace
Dynatrace is a popular observability platform that provides advanced monitoring, analytics, and AI-driven insights.
To integrate Dapr with Dynatrace, we utilize the OpenTelemetry Exporter. The exporter allows us to send telemetry data from Dapr to our Dynatrace environment for analysis and visualization.
In the Deployment manifest, environment variables are set to configure the Dynatrace exporter. The OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_TOKEN variables are populated with the necessary values from a Kubernetes Secret, which stores the Dynatrace API credentials securely.
Conclusion
By combining Dapr, OpenTelemetry, and Dynatrace, we can achieve comprehensive observability for our microservices-based applications. Dapr’s integration with OpenTelemetry enables the collection and forwarding of telemetry data, while Dynatrace provides powerful monitoring and analysis capabilities.
The provided code samples and deployment instructions can help you get started with Dapr observability using OpenTelemetry and Dynatrace.
