I am trying to get in memory storage of exemplars working:

time=2025-02-16T14:52:15.866Z level=INFO source=main.go:225 msg="Experimental in-memory exemplar storage enabled"

Even with - '--log.level=debug' turned on -- I see no evidence of scraping errors. The only error i can generate is when the version: otel/opentelemetry-collector-contrib:0.118.0 container instance is offline.

My prometheus instance version is: prom/prometheus:v3.1.0.

I can confirm that the /metrics endpoint exposed has metrics stored which have exemplars attached to them. I am working in a dotnet 8 web api with the open telemetry suite of nuget packages version: 1.11.x with auto instrumentation - seems to work well from what I can tell of it.

On my Docker container instance I exposed port 9091 to access the /metrics endpoint.

A span_id is highlighted

The trace_id can be seen in the same line.

I am convinced Prometheus is dropping the exemplars I can see in the collector - akin to not using Accept: application/openmetrics-text header -- that is what it feels like to me.

Another important question is, is this even the right way of doing this. Should metrics + exemplars be pushed to Prometheus or can they be scraped. On the latter so far, only metrics can be scraped successfully.

Solved

Will be posting an answer to cover everything I did to get this working.