Hey DevOps Engineer….. Here’s your Top 5 reasons why you need Distributed Tracing in your life!

https://giphy.com/

1. Distributed tracing tracks and observes service requests as they flow through distributed systems.

2. Distributed tracing provides end to end Observability.

Requests can be hard to track! Requests might pass through various services to reach completion, and these services could be in a variety of places: containers, serverless environments, virtual machines, different cloud providers, or on-premises.

giphy

3. Head-based sampling avoids storage issues!

In New Relic you can apply filters to individual spans before all spans in a trace arrive, which means decisions about whether to accept spans are made at the beginning (the “head”) of the filtering process. We use this sampling strategy to capture a representative sample of activity while avoiding storage and performance issues.

4. Infinite Tracing figures out which trace data is most important.

New Relic offers Infinite Tracing, you can send us 100% of your trace data from your application or third-party telemetry service, and Infinite Tracing will figure out which trace data is most important. And you can configure the sampling to ensure the traces important to you are retained.

5. Understanding the structure of a distributed trace can help you:

• Understand how traces are displayed in our UI
• Help you query trace data

A distributed trace has a tree-like structure, with “child” spans that refer to one “parent” span. This diagram shows some important span relationships in a trace:

This diagram above shows how spans in a distributed trace relate to each other.

This diagram shows several important concepts:

• Trace root. The first service or process in a trace is referred to as the root service or process.
• Process boundaries. A process represents the execution of a logical piece of code. Examples of a process include a backend service or Lambda function. Spans within a process are categorized as one of the following:
• Entry span: the first span in a process.
• Exit span: a span is a considered an exit span if it a) is the parent of an entry span, or b) has http. or db. attributes and therefore represents an external call.
• In-process span: a span that represents an internal method call or function and that is not an exit or entry span.
• Client spans. A client span represents a call to another entity or external dependency. Currently, there are two client span types:
• Datastore. If a client span has attributes prefixed with db. (like db.statement), it's categorized as a datastore span.
• External. If a client span has attributes prefixed with http. (like http.url) or has a child span in another process, it's categorized as an external span. This is a general category for any external calls that aren't datastore queries. If an external span contains http.url or net.peer.name, it's indexed on the External services page.
• Trace duration. A trace’s total duration is determined by the length of time from the start of the earliest span to the completion of the last span.

You can query span relationship data with the NerdGraph GraphiQL explorer at api.newrelic.com/graphiql.

gifphy.com

How trace data is stored

Understanding how we store trace data can help you query your trace data.

We save trace data as:

• Span: A span represents operations that are part of a distributed trace. The operations that a span can represent include browser-side interactions, datastore queries, calls to other services, method-level timing, and Lambda functions. One example: in an HTTP service, a span is created at the start of an HTTP request and completed when the HTTP server returns a response. Span attributes contain important information about that operation (such as duration, host data, etc.), including trace-relationship details (such as traceId, guid). For span-related data, see span attributes.
• Transaction: If an entity in a trace is monitored by an agent, a request to that entity generates a single Transaction event. Transactions allow trace data to be tied to other New Relic features. For transaction-related data, see transaction attributes.
• Contextual metadata. We store metadata that shows calculations about a trace and the relationships between its spans. To query this data, use the NerdGraph GraphiQL explorer.

gipfy.com

How trace context is passed between applications

We support the W3C Trace Context standard, which makes it easier to trace transactions across networks and services. When you enable distributed tracing, New Relic agents add HTTP headers to a service’s outbound requests. HTTP headers act like passports on an international trip: They identify your software traces and carry important information as they travel through various networks, processes, and security systems.

The headers also contain information that helps us link the spans together later: metadata like the trace ID, span ID, the New Relic account ID, and sampling information.

If you are a nonprofit organization, looking to level up your Observability tools, New Relic offers free tools and full platform features along with 1TB of free monthly data injest and 5 user accounts to all qualifying global nonprofit customers! Just email us at hgruber(at)newrelic(dot)com to find out more!