Copy page

Observability

Zuplo OpenTelemetry

Zuplo ships with an OpenTelemetry plugin that allows you to collect and export telemetry data from your Zuplo API. The OpenTelemetry plugin supports tracing and logging. Metrics support is planned for future releases.

Tracing

Tracing enables you to monitor performance, identify bottlenecks, and troubleshoot issues in your Zuplo API. The OpenTelemetry plugin automatically instruments your API to collect trace data. You can send trace data any OpenTelemetry service such as Honeycomb, Middleware, Dynatrace, Jaeger, and many more.

With tracing enabled on your Zuplo API you will see timings for each request as well as spans for plugins, handlers, and policies. The OpenTelemetry plugin supports trace propagation (W3C headers by default) so you can trace requests all the way from the client to your backend.

What's Traced?

By default, when the OpenTelemetry plugin is enabled, the following is traced:

• Request: The entire request lifecycle is traced, including the time taken to process the request and send the response.
• Inbound Policies: The time taken to execute all inbound policies as well as each policy is traced.
• Handler: The request handler is traced
• Outbound Policies: The time taken to execute all outbound policies as well as each policy is traced.
• Subrequests: Any use of fetch within your custom policies or handlers will be traced.

Limitations

One important limitation to keep in mind is that the clock will only increment when performing I/O operations (for example when calling fetch, using the Cache APIs, etc.). This is a limitation imposed as a security measure due Zuplo's serverless, multi-tenant architecture. In practice this shouldn't impact your ability to trace as virtually any code that isn't I/O bound is fast.

Custom Tracing

You can add custom tracing to your Zuplo API by using the OpenTelemetry API. The example below shows how to implement tracing in a custom policy.

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
import { trace } from "@opentelemetry/api";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const tracer = trace.getTracer("my-tracer");
  return tracer.startActiveSpan("my-span", async (span) => {
    span.setAttribute("key", "value");

    try {
      const results = await Promise.all([
        fetch("https://api.example.com/hello"),
        fetch("https://api.example.com/world"),
      ]);
      // ...

      return request;
    } finally {
      span.end();
    }
  });
}

This will result in a span that has the following spans:

Code
|--- my-policy
|    |
|    |--- my-span
|    |    |
|    |    |--- GET https://api.example.com/hello
|    |    |
|    |    |--- GET https://api.example.com/world

Logging

Logging can be enabled by configuring the logUrl property in the OpenTelemetry plugin configuration, as shown in Tracing and Logging Configuration. When enabled, logs will be exported to the specified endpoint in OpenTelemetry format.

To add OpenTelemetry logs in your Zuplo handlers and policies, you can use the context.log object as shown below:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.info("Hello World");
  return request;
}

You can also set additional custom log properties using context.log.setLogProperties!:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.setLogProperties!({ customProperty: "value" });
  context.log.info("Hello World");
  return request;
}

After setting a custom property, all subsequent log messages will include that property.

These logs will be exported to the configured log endpoint in OpenTelemetry format. The log message will be in the message field and the custom properties will be in the attributes field.

Setup

Adding OpenTelemetry to your Zuplo API is done by adding the OpenTelemetryPlugin in the zuplo.runtime.ts file as shown below.

Basic Tracing Configuration

For most providers you will set values for exporter.url and exporter.headers. It's common for providers to use a header for authorization.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: {
          "api-key": environment.OTEL_API_KEY,
        },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
    }),
  );
}

Advanced Tracing Configuration

The OpenTelemetry plugin supports additional configuration options for advanced use cases, including sampling and post-processing of spans.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: { "api-key": environment.OTEL_API_KEY },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      // Optional post processor to modify spans before export
      postProcessor: (spans) => {
        for (const span of spans) {
          (
            span as unknown as { attributes: Record<string, unknown> }
          ).attributes["post.processed"] = true;
        }
        return spans;
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}

Tracing and Logging Configuration

Logging is only enabled when specifically configured with its own endpoint using the logUrl property. When using both tracing and logging, you can configure them with separate endpoints.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      logUrl: "https://otel-collector.example.com/v1/logs",
      traceUrl: "https://otel-collector.example.com/v1/traces",
      headers: { "api-key": environment.OTEL_API_KEY },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}