I'm trying to set up OpenTelemetry for an AWS Lambda function to view traces and metrics using AWS X-Ray and CloudWatch. I'm following the steps for adding the AWS OpenTelemetry Lambda Layer and configuring the necessary environment variables, but I'm not sure how to verify if the traces and metrics are being sent or how to view them.

Here's what I have set up so far:

Lambda Layer

I've added the OpenTelemetry Lambda layer with the following ARN:

arn:aws:lambda:<region>:901920570463:layer:aws-otel-nodejs-<architecture>-ver-1-30-1:1

Environment Variables

I've configured the following environment variables in my Lambda function:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_EXPORTER_XRAY_ENABLED=true
OTEL_EXPORTER_XRAY_ENDPOINT=xray.us-east-2.amazonaws
OTEL_LAMBDA_METRICS_ENABLED=true
OTEL_METRICS_EXPORTER=awsemf
OTEL_TRACES_SAMPLER=always_on

Lambda Permissions

I have granted the following permissions to the Lambda function:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords"
      ],
      "Resource": "*"
    }
  ]
}

Lambda Function Code

Here is a simplified version of my Lambda function where I am attempting to use OpenTelemetry for tracing and metrics collection:

import { APIGatewayProxyEvent, APIGatewayProxyHandler, APIGatewayProxyResult } from "aws-lambda";
import { trace, context, SpanStatusCode, metrics } from "@opentelemetry/api";

await initializeFunction();
const tracer = trace.getTracer("eventSearchViolation-tracer");

export const handler: APIGatewayProxyHandler = async (event, lambdaContext): Promise<APIGatewayProxyResult> => {
    const meter = metrics.getMeter("eventSearchViolation-metrics");

    const lambdaInvocationCounter = meter.createCounter("lambda_invocation", {
        description:"Counts the number of times the lambda function is invoked"
    });
    lambdaInvocationCounter.add(1);

    const parentSpan = tracer.startSpan("lambda_execution");
    try {
        parentSpan.setAttribute("aws.lambda.invocation_id", lambdaContext.awsRequestId);
        
        return await context.with(trace.setSpan(context.active(), parentSpan), async () => {
            const childSpan = tracer.startSpan("process_event");
            try {
                // Event handling logic goes here...

                return {
                    statusCode: 200,
                    body: JSON.stringify({ message: "Success" }),
                };
            } catch (error) {
                childSpan.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
                throw error;
            } finally {
                childSpan.end();
            }
        });
    } finally {
        parentSpan.end();
    }
};

Logs and Error Messages

I can see the following log entries, but I’m not sure if the traces are being sent:

2025-03-05T23:21:04.536Z
{
    "level": "info",
    "ts": 1741216864.5367267,
    "caller": "[email protected]/service.go:261",
    "msg": "Everything is ready. Begin running and processing data."
}

2025-03-05T23:21:05.421Z
Registering OpenTelemetry

2025-03-05T23:21:05.440Z
@opentelemetry/instrumentation-aws-lambda No handler file was able to resolved with one of the known extensions for the file /var/task/src/index

Question

1. How can I verify if my Lambda function is sending the traces and metrics correctly?
2. How do I view the traces in AWS X-Ray and metrics in CloudWatch? What steps should I follow to ensure the traces and metrics are visible?
3. Is there anything wrong with my setup that might be preventing the traces/metrics from being sent or displayed?

Any guidance on how to debug this issue or configure it properly to view OpenTelemetry traces and metrics would be greatly appreciated!