I recently switched from using New Relic’s Java agent to OpenTelemetry (opentelemetry-javaagent.jar) to avoid vendor lock-in. In theory, OpenTelemetry should make it easier to migrate to another vendor in the future.

However, after making the switch, I noticed that service logs in New Relic have changed drastically compared to what I see locally when running the service.

Issues I’m Facing:

1. Missing exception stack traces – Error logs in New Relic do not include stack traces or error line numbers, making debugging difficult.

2. Log structure is different – The logs in New Relic look completely different from what I see locally in my terminal.

3. Some error logs are missing – Certain error logs that appear locally do not show up in New Relic at all.

4. Adding OpenTelemetry Logback appender didn’t change anything – I followed this guide: .0/library. I added the necessary Maven dependency and configured the OpenTelemetry Logback appender. However, this had no effect—logs were already appearing in New Relic even before adding the appender, and adding it did not fix the missing fields.

What I’ve Tried:

• I set OTEL_JAVAAGENT_LOGGING=application to ensure OpenTelemetry logs use my existing Logback configuration.
• I checked my logback-spring.xml, which uses LogstashEncoder to output structured JSON logs.
• I set OTEL_LOGS_INCLUDE_EXCEPTION_STACKTRACE=true, but stack traces still do not appear in New Relic logs.
• I tried querying logs in New Relic using NRQL (SELECT * FROM Log WHERE service.name = 'my-service' SINCE 1 hour ago LIMIT MAX) to check if logs were ingested correctly, but missing error logs still do not appear.

My Questions:

1. Why do logs look different in New Relic compared to my local logs?

2. Why are stack traces missing in New Relic logs, even when errors occur?

3. Do I need to explicitly configure OpenTelemetry to capture and forward stack traces? If so, how?

4. Is the OpenTelemetry Logback appender actually required for logs to be sent to New Relic, or is OpenTelemetry automatically capturing logs in a different way?

Here is the structure of newrelic log I am seeing right now:

{
  "entity.guid": "value”,
  "entity.guids": "value",
  "entity.name": "value",
  "entity.type": "SERVICE",
  "host.arch": "amd64",
  "host.name": "value",
  "instrumentation.provider": "opentelemetry",
  "message": “Error Log message, without stacktrace”,
  "newrelic.logPattern": "nr.DID_NOT_MATCH",
  "newrelic.source": "api.logs.otlp",
  "os.description": "Linux 6.1.119",
  "os.type": "linux",
  "otel.library.name": “package.and.ClassNameOfLogger”,
  "otel.library.version": "",
  "processmand_args": "[\"/usr/java/openjdk-22/bin/java\",\"-XX:+UseG1GC\",\"-XX:MaxRAMPercentage=75\",\"-XX:InitialRAMPercentage=75\",\"-XX:+ExitOnOutOfMemoryError\",\"-XX:OnOutOfMemoryError=echo OUT_OF_MEMORY_ALARM;\",\"-jar\",\"imp.jar\"]",
  "process.executable.path": "/usr/java/openjdk-22/bin/java",
  "process.pid": 1,
  "process.runtime.description": "Oracle Corporation OpenJDK 64-Bit Server VM 22-ea+21-1679",
  "process.runtime.name": "OpenJDK Runtime Environment",
  "process.runtime.version": "22-ea+21-1679",
  "service.instance.id": “instanceId”,
  "service.name": “serviceName”,
  "service.version": "0.0.1",
  "severity.number": 17,
  "severity.text": "ERROR",
  "span.id": "cb9a26c445504ce8",
  "telemetry.distro.name": "opentelemetry-java-instrumentation",
  "telemetry.distro.version": "2.12.0",
  "telemetry.sdk.language": "java",
  "telemetry.sdk.name": "opentelemetry",
  "telemetry.sdk.version": "1.46.0",
  "timestamp": 1739174011973,
  "trace.id": "f23ae673a91694c163fe87aefa950a0e",
  "w3c.flags": 1
}

Is there a way to configure or add additional information to newrelic log while using Spring logback with Opentelemetry java agent?

Main file:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.scheduling.annotation.EnableScheduling;

@EnableRetry
@EnableScheduling
@EnableFeignClients
@SpringBootApplication
public class SpringApplication {

    public static void main(String[] args) {
        SpringApplication.run(SpringApplication.class, args);
    }

}

maven dependency:

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.12.0-alpha</version>
</dependency>

Dockerfile (part of it related to opentelemetry):

ENV JAVA_TOOL_OPTIONS="-javaagent:/app/opentelemetry-javaagent.jar -Dotel.instrumentation.micrometer.enabled=true -Dotel.traces.sampler=parentbased_traceidratio -Dotel.exporter.otlppression=gzip -Dotel.javaagent.logging=application -Dotel.exporter.otlp.metrics.temporality.preference=delta -Dotel.exporter.otlp.metrics.default.histogram.aggregation=base2_exponential_bucket_histogram -Dotel.experimental.exporter.otlp.retry.enabled=true"

ENTRYPOINT ["./docker-entrypoint.sh"]

docker-entrypoint.sh:

#!/bin/sh

export OTEL_SERVICE_INSTANCE_ID=$HOSTNAME
export OTEL_EXPORTER_OTLP_HEADERS="api-key=$NEW_RELIC_LICENSE_KEY"
export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
export OTEL_SERVICE_NAME="service-name"
export OTEL_EXPORTER_OTLP_ENDPOINT=";

exec java -XX:+UseG1GC -XX:MaxRAMPercentage=75 -XX:InitialRAMPercentage=75 -XX:+ExitOnOutOfMemoryError -XX:OnOutOfMemoryError='echo OUT_OF_MEMORY_ALARM;' -jar imp.jar

logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>

    <!-- Read properties -->
    <springProperty name="springApplicationName" source="spring.application.name"/>
    <springProperty name="springApplicationVersion" source="spring.application.version"/>
    <springProperty name="teamName" source="team.name"/>

    <springProperty name="exceptionMaxLength" source="logging.exceptions.maxLength" defaultValue="2048"/>
    <springProperty name="maxDepthPerThrowable" source="logging.exceptions.maxDepthPerThrowable" defaultValue="10"/>

    <!-- Use the default logger if this profile is enabled -->
    <springProfile name="plain-text-logging">
        <include resource="org/springframework/boot/logging/logback/base.xml"/>
    </springProfile>

    <!-- Use the JSON logger by default -->
    <springProfile name="!plain-text-logging">
        <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
            <encoder class="net.logstash.logback.encoder.LogstashEncoder">
                <customFields>
                    {
                    "springApplicationName":"${springApplicationName}",
                    "springApplicationVersion":"${springApplicationVersion}",
                    "teamName":"${teamName}"
                    }
                </customFields>
                <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                    <maxDepthPerThrowable>${maxDepthPerThrowable}</maxDepthPerThrowable>
                    <maxLength>${exceptionMaxLength}</maxLength>
                    <shortenedClassNameLength>20</shortenedClassNameLength>
                    <exclude>sun\.reflect\..*\.invoke.*</exclude>
                    <exclude>net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
                    <rootCauseFirst>true</rootCauseFirst>
                    <inlineHash>true</inlineHash>
                </throwableConverter>
            </encoder>
        </appender>
        <appender name="OpenTelemetry"
                  class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
            <captureExperimentalAttributes>true</captureExperimentalAttributes>
            <captureMdcAttributes>*</captureMdcAttributes>
        </appender>

        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
            <appender-ref ref="OpenTelemetry"/>
        </root>
    </springProfile>

</configuration>