Otel with collector works well

Author: royischoss

charts/mlrun-ce/README.md

@@ -107,20 +107,16 @@ helm --namespace mlrun \
     --set opentelemetry-operator.enabled=true \
     --set opentelemetry.namespaceLabel.enabled=true \
     --set opentelemetry.collector.enabled=true \
-    --set opentelemetry.collector.scrapeMode=otel \
     --set opentelemetry.instrumentation.enabled=true \
     mlrun/mlrun-ce
 ```
 
-> **Important:** When enabling OpenTelemetry, set `opentelemetry.collector.scrapeMode=otel` to collect metrics 
-> via the OTEL sidecar and prevent duplicate metrics. The default is `direct` (for when OTEL is disabled).
-
 The installation will:
 - Deploy the OpenTelemetry Operator
-- Create an OpenTelemetryCollector CR (sidecar mode)
+- Create an OpenTelemetryCollector CR (deployment mode — one collector per namespace)
 - Create an Instrumentation CR for Python auto-instrumentation
-- Label the namespace with `opentelemetry.io/inject=enabled`
-- Configure Prometheus to scrape OTEL sidecar metrics (port 8889)
+- Label and annotate the namespace so all Python pods are auto-instrumented automatically
+- Configure Prometheus to scrape OTEL collector metrics (port 8889)
 
 #### Step 5: Verify OpenTelemetry Installation
 
@@ -140,21 +136,14 @@ kubectl -n mlrun get instrumentations
 kubectl -n mlrun get pods | grep opentelemetry
 ```
 
-#### Step 6: Verify Jupyter has OTEL Sidecar Annotations
+#### Step 6: Verify OTel Pod Labels and Namespace Annotation
 
 ```bash
-kubectl -n mlrun get deployment -l app.kubernetes.io/component=jupyter-notebook \
-    -o jsonpath='{.items[0].spec.template.metadata.annotations}' | jq .
-```
+# Check that the namespace has the instrumentation annotation (enables auto-instrumentation for all Python pods)
+kubectl get namespace mlrun -o jsonpath='{.metadata.annotations}' | jq .
 
-You should see annotations like:
-```json
-{
-  "instrumentation.opentelemetry.io/inject-python": "my-mlrun-otel-instrumentation",
-  "prometheus.io/port": "8889",
-  "prometheus.io/scrape": "true",
-  "sidecar.opentelemetry.io/inject": "my-mlrun-otel-collector"
-}
+# Check pod labels — all chart-managed pods should have mlrun.io/otel=true
+kubectl -n mlrun get pods --show-labels | grep mlrun.io/otel
 ```
 
 ### Installing MLRun-ce on minikube
@@ -185,7 +174,7 @@ Override those [in the normal methods](https://helm.sh/docs/chart_template_guide
 ### Configuring OpenTelemetry (Observability)
 
 MLRun CE includes the OpenTelemetry Operator for collecting metrics and traces from your ML workloads. 
-The operator runs in **sidecar mode**, automatically injecting collector containers into annotated pods.
+The operator runs one collector **Deployment** per namespace. Instrumented pods send OTLP metrics to the collector, which exports them to Prometheus.
 
 > **Note:** OpenTelemetry is **disabled by default**. See below for how to enable it.
 
@@ -212,10 +201,10 @@ kubectl label namespace <your-namespace> opentelemetry.io/inject=enabled
 #### Default Configuration
 
 By default, OpenTelemetry is **disabled**. When enabled, it provides:
-- Namespace labeling for OTEL operator webhook targeting
-- Sidecar collector injection for instrumented pods
-- Python auto-instrumentation for Jupyter notebooks
-- Prometheus metrics export on port 8889
+- A single OTel Collector Deployment per namespace (OTLP receiver → Prometheus exporter on port 8889)
+- Namespace-level Python auto-instrumentation (all Python pods in the namespace are instrumented automatically)
+- `mlrun.io/otel: "true"` label on Jupyter, SeaweedFS, and Nuclio function pods
+- Prometheus scrapes the collector pod (not individual pods)
 
 #### Enabling OpenTelemetry
 
@@ -228,7 +217,6 @@ helm --namespace mlrun install my-mlrun \
     --set opentelemetry-operator.enabled=true \
     --set opentelemetry.namespaceLabel.enabled=true \
     --set opentelemetry.collector.enabled=true \
-    --set opentelemetry.collector.scrapeMode=otel \
     --set opentelemetry.instrumentation.enabled=true \
     mlrun/mlrun-ce
 ```
@@ -240,7 +228,6 @@ helm --namespace mlrun upgrade my-mlrun \
     --set opentelemetry-operator.enabled=true \
     --set opentelemetry.namespaceLabel.enabled=true \
     --set opentelemetry.collector.enabled=true \
-    --set opentelemetry.collector.scrapeMode=otel \
     --set opentelemetry.instrumentation.enabled=true \
     mlrun/mlrun-ce
 ```
@@ -253,13 +240,12 @@ helm --namespace mlrun upgrade my-mlrun \
     --set opentelemetry.collector.enabled=false \
     --set opentelemetry.instrumentation.enabled=false \
     --set opentelemetry.namespaceLabel.enabled=false \
-    --set opentelemetry.collector.scrapeMode=direct \
     mlrun/mlrun-ce
 ```
 
 #### Custom Resource Limits
 
-Configure collector sidecar resources:
+Configure collector resources:
 
 ```bash
 helm --namespace mlrun install my-mlrun \
@@ -282,63 +268,23 @@ helm --namespace mlrun install my-mlrun \
 
 #### Adding OpenTelemetry to Custom Workloads
 
-To instrument your own deployments with the OTEL sidecar and Python auto-instrumentation:
-
-1. Ensure your namespace has the OpenTelemetry label:
-   ```bash
-   kubectl label namespace <your-namespace> opentelemetry.io/inject=enabled
-   ```
-
-2. Add these annotations to your pod spec:
-   ```yaml
-   metadata:
-     annotations:
-       sidecar.opentelemetry.io/inject: "<release-name>-otel-collector"
-       instrumentation.opentelemetry.io/inject-python: "<release-name>-otel-instrumentation"
-       prometheus.io/scrape: "true"
-       prometheus.io/scrape-mode: "otel"
-       prometheus.io/port: "8889"
-   ```
-
-#### Preventing Prometheus/OTEL Metric Overlap
-
-To prevent duplicate metrics when using both Prometheus direct scraping and OpenTelemetry, 
-MLRun CE uses a **scrape-mode** annotation system:
-
-| Scrape Mode | Description | Use Case |
-|-------------|-------------|----------|
-| `direct` | Direct Prometheus scraping only | **Default** - When OTEL is disabled |
-| `otel` | Metrics collected via OTEL sidecar only | **Recommended when OTEL enabled** |
-| `both` | Both OTEL and direct scraping | Debugging/transition only |
-
-> **Note:** The default scrape mode is `direct`. When enabling OpenTelemetry, you must set 
-> `--set opentelemetry.collector.scrapeMode=otel` to collect metrics via the OTEL sidecar.
-
-**How it works:**
-- OTEL-collected metrics have the `mlrun_otel_` prefix and `metrics_source=otel_collector` label
-- Direct-scraped metrics have `metrics_source=direct_scrape` label
-- Prometheus scrape configs filter based on `prometheus.io/scrape-mode` annotation
+Python instrumentation is applied **namespace-wide** — any Python pod in the MLRun namespace is automatically instrumented when OTel is enabled. No per-pod annotations are required.
 
-**Configure scrape mode when enabling OTEL:**
+For pods in other namespaces, annotate the namespace directly:
 ```bash
-helm --namespace mlrun install my-mlrun \
-    --set opentelemetry-operator.enabled=true \
-    --set opentelemetry.collector.enabled=true \
-    --set opentelemetry.collector.scrapeMode=otel \
-    --set opentelemetry.instrumentation.enabled=true \
-    mlrun/mlrun-ce
+kubectl annotate namespace <your-namespace> \
+    instrumentation.opentelemetry.io/inject-python=<release-name>-otel-instrumentation
 ```
 
-**Query metrics by source in Prometheus:**
-```promql
-# OTEL-collected metrics only
-{metrics_source="otel_collector"}
-
-# Direct-scraped metrics only  
-{metrics_source="direct_scrape"}
+The `mlrun.io/otel: "true"` label is applied to: **Jupyter**, **SeaweedFS** (master, volume, filer, s3, admin), and **Nuclio function pods** (via `functionDefaults.metadata.labels`). This label is used for Prometheus metric filtering and enrichment.
 
-# OTEL metrics use prefix
+**Query OTEL-collected metrics in Prometheus:**
+```promql
+# OTEL metrics use the mlrun_otel_ prefix
 mlrun_otel_http_server_duration_seconds_bucket{...}
+
+# Filter by source
+{metrics_source="otel_collector"}
 ```
 
 #### Split Installation (Admin/Non-Admin)

charts/mlrun-ce/non_admin_installation_values.yaml

@@ -88,8 +88,8 @@ opentelemetry-operator:
   enabled: false
 
 # OpenTelemetry CRs - enabled for user namespace
-# The namespace will be labeled with opentelemetry.io/inject=enabled
-# so the operator can inject sidecars into pods
+# The namespace will be labeled and annotated for OTel deployment-mode collection
+# and namespace-wide Python auto-instrumentation.
 opentelemetry:
   namespaceLabel:
     enabled: true

charts/mlrun-ce/templates/NOTES.txt

@@ -134,47 +134,36 @@ OpenTelemetry Operator is enabled!
 -  Namespace selector: opentelemetry.io/inject=enabled
 {{- if .Values.opentelemetry.collector.enabled }}
 {{- "\n" }}
-OpenTelemetry Collector (sidecar mode):
--  Collector CR: {{ .Release.Name }}-otel-collector
+OpenTelemetry Collector (deployment mode):
+-  Collector CR: {{ include "mlrun-ce.otel.collector.fullname" . }}
 -  Mode: {{ .Values.opentelemetry.collector.mode }}
--  OTLP gRPC endpoint: localhost:{{ .Values.opentelemetry.collector.otlp.grpcPort }} (inside pod)
--  OTLP HTTP endpoint: localhost:{{ .Values.opentelemetry.collector.otlp.httpPort }} (inside pod)
--  Prometheus metrics port: {{ .Values.opentelemetry.collector.prometheus.port }}
--  Prometheus scrape mode: {{ .Values.opentelemetry.collector.scrapeMode }}
-{{- if eq .Values.opentelemetry.collector.scrapeMode "direct" }}
-
-⚠️  WARNING: Scrape mode is "direct" - OTEL sidecar metrics will NOT be collected!
-   To collect metrics via OTEL, reinstall with: --set opentelemetry.collector.scrapeMode=otel
-{{- end }}
+-  OTLP gRPC endpoint: {{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.grpcPort }}
+-  OTLP HTTP endpoint: {{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.httpPort }}
+-  Prometheus metrics port: {{ .Values.opentelemetry.collector.prometheus.port }} (scraped by Prometheus from the collector pod)
 {{- end }}
 {{- if .Values.opentelemetry.instrumentation.enabled }}
 {{- "\n" }}
 OpenTelemetry Auto-Instrumentation:
--  Instrumentation CR: {{ .Release.Name }}-otel-instrumentation
+-  Instrumentation CR: {{ include "mlrun-ce.otel.instrumentation.fullname" . }}
 {{- if .Values.opentelemetry.instrumentation.python.enabled }}
--  Python auto-instrumentation: enabled
+-  Python auto-instrumentation: enabled (namespace-wide via namespace annotation)
 {{- end }}
 {{- if .Values.opentelemetry.instrumentation.java.enabled }}
 -  Java auto-instrumentation: enabled
 {{- end }}
 {{- end }}
 {{- if .Values.opentelemetry.namespaceLabel.enabled }}
 {{- "\n" }}
-Namespace Label:
--  Namespace {{ .Release.Namespace }} is labeled with: {{ .Values.opentelemetry.namespaceLabel.key }}={{ .Values.opentelemetry.namespaceLabel.value }}
+Namespace OTel configuration:
+-  Label: {{ .Values.opentelemetry.namespaceLabel.key }}={{ .Values.opentelemetry.namespaceLabel.value }}
+{{- if .Values.opentelemetry.instrumentation.enabled }}
+-  Python instrumentation annotation applied to all pods in namespace {{ .Release.Namespace }}
 {{- end }}
+{{- end }}
+{{- if or .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
 {{- "\n" }}
-Prometheus Scrape Modes:
--  "otel"   : Metrics collected via OTEL sidecar only (recommended)
--  "direct" : Direct Prometheus scraping only (current: {{ .Values.opentelemetry.collector.scrapeMode }})
--  "both"   : Both methods active (for debugging)
-{{- "\n" }}
-To add OTEL instrumentation to your pods, add these annotations:
-  sidecar.opentelemetry.io/inject: "{{ .Release.Name }}-otel-collector"
-  instrumentation.opentelemetry.io/inject-python: "{{ .Release.Name }}-otel-instrumentation"
-  prometheus.io/scrape: "true"
-  prometheus.io/scrape-mode: "otel"
-  prometheus.io/port: "{{ .Values.opentelemetry.collector.prometheus.port }}"
+Pods labeled with mlrun.io/otel=true: Jupyter, SeaweedFS (master/volume/filer/s3/admin), and Nuclio function pods.
+{{- end }}
 {{- end }}
 
 Happy MLOPSing!!! :]

charts/mlrun-ce/templates/_helpers.tpl

@@ -423,7 +423,7 @@ OpenTelemetry helpers
 OpenTelemetry Collector name
 */}}
 {{- define "mlrun-ce.otel.collector.name" -}}
-{{- default "otel-collector" .Values.opentelemetry.collector.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- default "otel" .Values.opentelemetry.collector.nameOverride | trunc 63 | trimSuffix "-" }}
 {{- end }}
 
 {{/*
@@ -433,7 +433,7 @@ OpenTelemetry Collector fullname
 {{- if .Values.opentelemetry.collector.fullnameOverride }}
 {{- .Values.opentelemetry.collector.fullnameOverride | trunc 63 | trimSuffix "-" }}
 {{- else }}
-{{- $name := default "otel-collector" .Values.opentelemetry.collector.nameOverride }}
+{{- $name := default "otel" .Values.opentelemetry.collector.nameOverride }}
 {{- if contains $name .Release.Name }}
 {{- .Release.Name | trunc 63 | trimSuffix "-" }}
 {{- else }}
@@ -526,7 +526,7 @@ spec:
         endpoint: 0.0.0.0:{{ .Values.opentelemetry.collector.prometheus.port }}
         namespace: {{ .Values.opentelemetry.collector.prometheus.namespace }}
         const_labels:
-          collector_mode: sidecar
+          collector_mode: deployment
           metrics_source: otel_collector
         resource_to_telemetry_conversion:
           enabled: true
@@ -579,6 +579,8 @@ metadata:
   labels:
     {{- include "mlrun-ce.otel.labels" . | nindent 4 }}
 spec:
+  exporter:
+    endpoint: http://{{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.httpPort }}
   propagators:
     {{- toYaml .Values.opentelemetry.instrumentation.propagators | nindent 4 }}
   sampler:
@@ -589,24 +591,6 @@ spec:
       valueFrom:
         fieldRef:
           fieldPath: metadata.labels['app.kubernetes.io/name']
-    - name: OTEL_RESOURCE_ATTRIBUTES
-      value: >-
-        k8s.namespace.name=$(OTEL_RESOURCE_ATTRIBUTES_NAMESPACE),
-        k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),
-        k8s.container.name=$(OTEL_RESOURCE_ATTRIBUTES_CONTAINER_NAME),
-        service.namespace=$(OTEL_RESOURCE_ATTRIBUTES_NAMESPACE)
-    - name: OTEL_RESOURCE_ATTRIBUTES_NAMESPACE
-      valueFrom:
-        fieldRef:
-          fieldPath: metadata.namespace
-    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
-      valueFrom:
-        fieldRef:
-          fieldPath: metadata.name
-    - name: OTEL_RESOURCE_ATTRIBUTES_CONTAINER_NAME
-      valueFrom:
-        fieldRef:
-          fieldPath: metadata.name
     - name: OTEL_METRICS_EXPORTER
       value: otlp
     - name: OTEL_TRACES_EXPORTER
@@ -624,7 +608,7 @@ spec:
       - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
         value: "false"
       - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
-        value: ""
+        value: "aws_lambda"
   {{- end }}
   {{- if .Values.opentelemetry.instrumentation.java.enabled }}
   java:
@@ -636,3 +620,12 @@ spec:
         value: "true"
   {{- end }}
 {{- end }}
+..
+{{/*
+OTel pod label — marks a pod as OTel-monitored for metric enrichment and discovery.
+Namespace-level instrumentation annotation (set by namespace-label job) handles Python auto-instrumentation.
+Wrap usage with: {{- if and .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
+*/}}
+{{- define "mlrun-ce.otel.podLabels" -}}
+mlrun.io/otel: "true"
+{{- end }}

charts/mlrun-ce/templates/jupyter-notebook/deployment.yaml

@@ -14,22 +14,9 @@ spec:
     metadata:
       labels:
         {{- include "mlrun-ce.jupyter.selectorLabels" . | nindent 8 }}
-      {{- if and .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
-      annotations:
-        # OpenTelemetry sidecar injection
-        sidecar.opentelemetry.io/inject: "{{ include "mlrun-ce.otel.collector.fullname" . }}"
-        # Python auto-instrumentation injection
-        instrumentation.opentelemetry.io/inject-python: "{{ include "mlrun-ce.otel.instrumentation.fullname" . }}"
-        # Prometheus scraping configuration
-        # scrape-mode controls how metrics are collected to prevent duplicates:
-        #   "otel"   - Only OTEL sidecar metrics (recommended)
-        #   "both"   - Both OTEL and direct scraping (debugging)
-        #   "direct" - Only direct scraping (OTEL metrics ignored)
-        prometheus.io/scrape: "true"
-        prometheus.io/scrape-mode: {{ .Values.opentelemetry.collector.scrapeMode | quote }}
-        prometheus.io/port: "{{ .Values.opentelemetry.collector.prometheus.port }}"
-        prometheus.io/path: "/metrics"
-      {{- end }}
+        {{- if and .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
+        {{- include "mlrun-ce.otel.podLabels" . | nindent 8 }}
+        {{- end }}
     spec:
       {{- with .Values.jupyterNotebook.image.pullSecrets }}
       imagePullSecrets:

charts/mlrun-ce/templates/opentelemetry/namespace-label.yaml

@@ -9,7 +9,7 @@ metadata:
   labels:
 {{ include "mlrun-ce.otel.labels" . | indent 4 }}
   annotations:
-    "helm.sh/hook": post-install,post-upgrade
+    "helm.sh/hook": pre-install,pre-upgrade
     "helm.sh/hook-weight": "-10"
     "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
     "helm.sh/hook-timeout": "120s"
@@ -30,5 +30,11 @@ spec:
         - |
           echo "Labeling namespace {{ .Release.Namespace }} for OpenTelemetry..."
           kubectl label namespace {{ .Release.Namespace }} {{ .Values.opentelemetry.namespaceLabel.key }}={{ .Values.opentelemetry.namespaceLabel.value }} --overwrite
-          echo "Namespace labeled successfully!"
+          {{- if .Values.opentelemetry.instrumentation.enabled }}
+          echo "Annotating namespace for namespace-wide Python auto-instrumentation..."
+          kubectl annotate namespace {{ .Release.Namespace }} \
+            instrumentation.opentelemetry.io/inject-python={{ include "mlrun-ce.otel.instrumentation.fullname" . }} \
+            --overwrite
+          {{- end }}
+          echo "Namespace configured for OpenTelemetry successfully!"
 {{- end -}}