.svg?alt=media&token=0f6df7c7-5b98-4a62-98b8-a54e38a8329e){kind=logo}
Tracing with Tempo
Exploring traces in the DuploCloud Advanced Observability Suite (AOS)
Last updated
Exploring traces in the DuploCloud Advanced Observability Suite (AOS)
Last updated
© DuploCloud, Inc. All rights reserved. DuploCloud trademarks used herein are registered trademarks of DuploCloud and affiliates
DuploCloud's Advanced Observability Suite (AOS) leverages Tempo for tracing, with Alloy and Beyla as the data collectors. Beyla, powered by eBPF (extended Berkeley Packet Filter), enables seamless collection of observability data directly from the system kernel without requiring application-level instrumentation. By attaching to system events like network requests and function calls, Beyla efficiently gathers metrics and traces. For more detailed trace information, users can fine-tune their applications using the OTEL SDK, integrating traces into logs for enhanced observability.
Tracing with OpenTelemetry is useful in scenarios where you want to identify, analyze, and resolve performance issues or understand the flow of requests through a distributed system. Tracing is helpful for:
Debugging latency issues
End-to-end visibility of requests across services
Analyzing errors and failures
Capacity planning and optimization
Root Cause Analysis (RCA)
Performance tuning new features
Validating SLAs
Exposing misconfigurations
Understanding user behavior
Using OpenTelemetry with Grafana, you can collect and visualize traces alongside other observability data like metrics and logs, providing a unified view for effective troubleshooting and optimization.
In the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard (Administrator AOS Dashboard) or Observability -> Advanced -> Dashboard (Tenant AOS Dashboard). The AOS Dashboard displays.
Select your Infrastructure from the Infrastructure list box (Administrator AOS Dashboard) or Tenant from the Tenant list box (Tenant AOS Dashboard).
Click the Traces card button. The Grafana Tracing dashboard displays.
Use the following functions on the Grafana Tracing dashboard to find and view relevant trace data. See the Grafana documentation for detailed instructions.
Search Query Builder: Use the search query builder to filter and find traces based on parameters like service name, duration, status codes, or custom tags. This lets you quickly narrow down the traces you're interested in and dive into specific details.
TraceQL: TraceQL is a powerful query language designed to provide advanced filtering and exploration of your trace data. It lets you run more complex queries for pinpointing specific traces or span data based on conditions like service dependencies, error rates, or trace attributes.
Service Graph: The Service Graph visualizes the relationships between services in your application. It shows how services communicate with one another and provides a high-level view of trace flow across your system. This view helps you identify bottlenecks or failures in service-to-service interactions.
When you have located a trace of interest using the steps above, you can inspect specific metrics or individual spans to understand the operations that occurred during the trace.
From the Grafana Tracing dashboard, click on the trace of interest. The Trace Timeline Viewer displays on the right side of the screen. See the Grafana Tracing documentation for more details.
The Trace Timeline Viewer can be used to map traces with logs, metrics, and profiles. These details offer insights into timing, status, and metadata, which can help identify performance bottlenecks or failures.
Trace-to-Logs: Navigate from a trace directly to relevant logs, allowing you to correlate trace details with log data.
Trace-to-Metrics: Jump to metrics related to a particular exception/span to analyze rates, durations, and other key performance indicators.
Trace-to-Profiles: Link trace spans directly to profiling data for deeper analysis of resource usage and performance at the code level. This feature allows you to correlate traces with profiles, such as CPU or memory usage, providing fine-grained insights into your system’s behavior.
For detailed instructions on inspecting traces, spans, and associated logs, see this Grafana documentation page. For more information about logging with DuploCloud's AOS, see the DuploCloud Logging page.
