The DuploCloud OpenTelemetry (OTEL) stack requires specific resources for deployment. The implementation features node-level components, deployed as DaemonSets, for telemetry collection and central observability components for data aggregation, processing, and visualization. Object storage is the backbone for retaining metrics, logs, traces, and profiling data, ensuring scalability and long-term accessibility.

OTEL Resource Requirements

Node-Level Components (DaemonSet)

The following components run as DaemonSets on each node in the Kubernetes cluster.

Central Stack Components

Hardware Requirements

Core Services Resource Allocation

Supporting Components

Storage Configuration

The implementation uses object storage (e.g., S3, GCS, Azure Storage Account) for:

• Metrics storage (Mimir)

• Log storage (Loki)

• Trace storage (Tempo)

• Profile data storage (Pyroscope)

The DuploCloud AOS dashboards are a gateway to the detailed Grafana dashboards, serving two purposes:

• SSO and Authentication Proxy: The Grafana dashboards reside on a private network. DuploCloud acts as an authentication layer, connecting the same single sign-on (the DuploCloud login) to a Grafana session.

• Summarizing Links: While AOS contains many pre-configured Grafana dashboards, you can create quick links with descriptions of the ones you use most frequently. See Customizing Dashboards for more information.

Administrator and Tenant AOS Dashboards

Depending on your role (Administrator or User), you can access the Advanced Observability Suite dashboard from two locations in the DuploCloud Portal.

1. The Administrator AOS Dashboard displays cloud data across all resources and allows you to select DuploCloud Infrastructures. To use it, navigate to Administrator -> Observability -> Advanced -> Dashboard in the DuploCloud Portal.

2. The Tenant AOS Dashboard displays data by Tenant. To use it, navigate to Observability -> Advanced -> Dashboard in the DuploCloud Portal.

The Administrator AOS Dashboard displays cloud data across all resources and infrastructures. It is only accessible to administrators.

To access it, navigate to Administrator -> Observability -> Advanced -> Dashboard.

Viewing Admin Cloud Spend Data

The Cloud Spend area, on the left side of the Advanced AOS Dashboard, offers a comprehensive, real-time view of expenses across all resources.

• Fin Ops: The Fin Ops button, in the Cloud Spend header, opens the DuploCloud Billing dashboard, which displays billing details including billing summaries by month or Tenant, billing alerts, and DuploCloud license usage information.

• Current Month: Displays cloud expenditures for the current month.

• Monthly Spend: Displays spending by month. Use the Monthly Spend list box to display spending by week or day.

• Spend By Service: Displays a breakdown of spend by Cloud Service.

• Spend By Tenant: Highlights expenditures by Tenant.

Viewing Admin Observability Data

The Observability section, on the right side of the Advanced AOS Dashboard, gives real-time health and usage data across resources.

• Infrastructure: In the Observability header, the Infrastructure list box allows you to select the Infrastructure for which you wish to view observability details.

• Grafana: The Grafana button, in the Observability header, opens the Grafana console where you can add, customize, or edit your dashboards, query your logs, metrics, and traces, and more. For additional information, see the Grafana documentation.

Under the Observability header are data cards displaying the following metrics:

• Resources: Lists the type and number of DuploCloud resources, such as Tenants, Services, etc.

• K8s/Docker: Shows Kubernetes and Docker metrics, providing visibility into containerized workloads.

• Logs: Displays logs for troubleshooting and compliance checks across all resources.

• Metrics: Displays rate, errors, and duration metrics across Services.

• Traces: View traces to monitor request flows and latency, supporting application performance analysis.

• Profiles: Access profiling data for in-depth application insights, allowing performance tuning.

For Grafana-generated metrics (e.g., K8s/Docker, Logs, Metrics, Traces, Profiles), you can click on the card to open the corresponding detailed view in the Grafana console. Additionally, you can add custom links to the data cards.

Advanced Observability Suite (AOS) is based on OpenTelemetry. The following graphic shows the various components.

Viewing the DuploCloud Deployment of the OpenTelemetry Stack

To view the complete deployment of the OpenTelemetry stack:

1. In the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard.

2. In the Observability area, click the K8s/Docker card. The Grafana K8s Resource Monitoring dashboard launches, giving you a detailed view of resources and monitoring for Kubernetes nodes, Docker containers, and Pods.

Viewing OpenTelemetry Data in DuploCloud

OpenTelemetry data is stored in S3 Buckets, which you can view in the DuploCloud Portal.

1. In the DuploCloud Portal, select the OpenTelemetry_Tenant from the Tenant list box at the top of the Portal.

2. Navigate to Cloud Services -> Storage and select the S3 tab to view the data. This data setup is deployed and managed via Flux Helm release infrastructure.

To view a complete list of Kubernetes deployments, containers, and S3 buckets in an OpenTelemetry deployment, select the OpenTelemetry_Tenant from the Tenant list box at the top of the Portal and navigate to Kubernetes -> Services.

To view a complete list of Docker Containers in an OpenTelemetry deployment, select the OpenTelemetry_Tenant from the Tenant list box at the top of the Portal and navigate to Kubernetes -> Containers.

The Tenant AOS Dashboard provides cost and observability data by Tenant for granular infrastructure management. It is accessible to non-administrators who have access to the selected Tenant.

To access the Tenant AOS Dashboard, navigate to Observability -> Advanced -> Dashboard.

Selecting a Tenant

You can use the Tenant list box at the top of the Tenant AOS Dashboard to select the Tenant for which you want to view metrics.

Viewing Tenant Cloud Spend Data

The Cloud Spend area, on the left side of the Tenant AOS Dashboard, offers a comprehensive view of expenses for the selected Tenant. It includes the following expenditure categories:

• Current Month: Displays the current month’s spend for the selected Tenant.

• Spend By Service: Displays a breakdown of cloud spending by Service.

Viewing Tenant Observability Data

The Observability area, on the right side of the Tenant AOS Dashboard, gives health and performance data for a selected Tenant.

Grafana: The Grafana button in the Observability header opens the Grafana console, where you can add, customize, or edit your AOS dashboards, query your logs, metrics, and traces, and more. For additional information, see the Grafana documentation.

Under the Observability header are data cards displaying various metrics.

• Resources: This field lists the type and number of DuploCloud resources, such as Services, containers, and Ingresses, in the selected Tenant.

• K8s/Docker: Kubernetes and Docker metrics specific to the Tenant, assisting in container workload management.

• Logs: Access Tenant-specific logs for tracking, troubleshooting, and compliance.

• Metrics: Displays performance metrics relevant to the Tenant’s resources.

• Traces: View traces specific to the Tenant for performance and latency monitoring of the Tenant’s applications.

• Profiles: Access profiling data for in-depth application insights and performance tuning.

For Grafana-generated metrics (e.g., K8s/Docker, Logs, Metrics, Traces, Profiles), you can click on the card (header or visual data) to open the corresponding detailed view in the Grafana console. Additionally, you customize your data sources and dashboards.

DuploCloud AOS users can tailor observability for their Kubernetes clusters by managing OpenTelemetry stacks and configuring Grafana dashboards to meet specific needs. Additionally, custom links can be added to AOS dashboard cards, enabling seamless integration with external tools or data sources.

By default, each Kubernetes cluster in DuploCloud has its own dedicated OpenTelemetry (OTEL) stack. If you need to share an OTEL stack across multiple clusters, contact DuploCloud Support.

Configuring OpenTelemetry Cards

In the AOS dashboard, the OpenTelemetry section features five cards that link to Grafana dashboards. To configure these cards and their links, go to Administrator -> System Settings -> System Config in the DuploCloud Portal and search for otel as shown in the image below.

Each configuration entry corresponds to a card on the dashboard. Entries that start with <infraname>/ apply to the cards for that specific Infrastructure, (in the Administrator AOS Dashboard, all cards reference the Infrastructure selected from the Infrastructure list box), as shown in the table below.

Creating New Dashboards

For detailed instructions on customizing Grafana dashboards, refer to the official Grafana Documentation on Dashboards.

Adding Custom Dashboard Links

For custom dashboards or manually instrumented applications, you may need to add custom links to your AOS dashboards to lead to the specific data or external tools associated with your custom configurations. Follow these steps to add custom links to data cards as an administrator or non-administrator:

Adding Administrator Custom Links

1. From the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard.

3. Select the Admin tab to add a custom link to the Administrator AOS Dashboard or the Common tab to add a custom link to the Tenant AOS Dashboard dashboard.

4. Click Add. The Add DATA_CARD_NAME Custom Link pane displays.

5. Enter details for the custom link:

   • Name: The label for the link.

   • URL: The target location (e.g., <infraname>/logs/tenant for Tenant-specific logs or <infraname>/metrics/tenant for tenant-specific metrics).

   • Description: Optional details about the link.

6. Click Submit. The custom link is added to the data card.

Adding Tenant Custom Links

Add custom links to Tenant dashboards.

1. From the DuploCloud Portal, navigate to Observability -> Advanced -> Dashboard.

3. Click Add. The Add DATA_CARD_NAME Custom Link pane displays.

4. Enter a Name, URL, and Description for the custom link.

5. Click Submit. The custom link is added to the Tenant Dashboard.

DuploCloud's Advanced Observability Suite (AOS) metrics pipeline leverages AWS CloudWatch for capturing critical metrics such as CPU utilization, memory usage, and Lambda execution times, Grafana Mimir as the backend for scalable and high-availability metrics storage, Grafana as the visualization tool for intuitive dashboard creation, and Grafana Alloy for bundling tools like Beyla and OpenTelemetry SDKs to collect metrics from applications and infrastructure. Out-of-the-box metrics combine with custom application-level metrics, to create unified dashboards that correlate infrastructure and application performance for a comprehensive view of system health.

About RED Metrics

RED metrics (Rate, Errors, Duration) are critical for monitoring and understanding the health and performance of distributed systems and microservices. When integrated into a Grafana dashboard using OpenTelemetry, they provide essential insights for diagnosing issues and optimizing performance, such as:

• Rate: Tracks the number of requests per second, indicating the system's throughput. A sudden drop can highlight service degradation or bottlenecks.

• Errors: Measures the percentage or count of requests that fail. This helps quickly identify stability issues or breaking changes in your service.

• Duration: Monitors the time to process requests, providing insights into system latency. High durations can indicate inefficiencies or resource constraints.

Displaying Metrics in the DuploCloud Advanced Observability Suite (AOS)

1. In the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard (Administrator AOS Dashboard) or Observability -> Advanced -> Dashboard (Tenant AOS Dashboard). The AOS Dashboard displays.

2. Select your Infrastructure from the Infrastructure list box (Administrator AOS Dashboard) or Tenant from the Tenant list box (Tenant AOS Dashboard).

3. Click the Metrics card button. The Grafana APM -> RED Metrics Dashboard displays. The graphic below shows the RED (Request, Error, Duration) Dashboard.

Exploring Detailed Metrics

From the Grafana Beyla RED Dashboard, navigate to Grafana Dashboards in the left pane. Various out-of-box metrics and visualizations are available to you without creating specific queries.

For example, by selecting K8s Resource Monitoring from the Dashboards page, you can view numerous Kubernetes widgets depicting resource usage of Kubernetes Pods and containers, among other helpful performance data.

Click the Add button and select from the displayed list to add additional customizations and visualizations to a dashboard.

DuploCloud's Advanced Observability Suite (AOS) is an add-on service to boost your monitoring and troubleshooting abilities. Built on OpenTelemetry, our AOS leverages CloudWatch, LGTM (Loki, Grafana, Tempo and Mimir), and Pyroscope to deliver robust observability for your cloud infrastructure. It includes real-time anomaly detection and customizable alerts and unifies metrics, traces, logs, and profiles to track all aspects of your environment easily.

The Grafana-powered dashboards spotlight key metrics and visualize trends in real-time, empowering swift, data-driven decisions. Additionally, AOS leverages object storage solutions like S3, GCS, and Storage Accounts, making it extremely cost-effective. Overall DuploCloud’s AOS simplifies troubleshooting and enhances system health by providing a holistic view of your application and infrastructure performance.

About OpenTelemetry

a project under the Cloud Native Computing Foundation (CNCF), supports multiple programming languages and environments and provides a flexible, vendor-neutral framework for monitoring and analyzing application performance. Designed for high scalability and ideal for distributed applications, OpenTelemetry is the industry standard for observability. See the for more.

Advanced Observability Suite Use Cases

Real-Time Monitoring and Observability

• System Health Check: Track real-time metrics like CPU usage, memory consumption, network traffic, and error rates across services to ensure that applications are healthy.

• Latency Tracking: Monitor request latencies to identify performance bottlenecks, particularly in services where latency spikes might indicate an issue. Easily drill down to the traces and determine which specific module is the cause for latency, or switch to profile view for deeper utilization visibility.

• Error Rate Analysis: Track error counts and types to ensure services operate as expected and identify critical failure points. Drill down to the traces with just a click to determine where the error occurs.

End-to-End Tracing and Dependency Mapping

• Request Flow Visualization: Visualize the path of requests through different services to understand interdependencies and identify which services may contribute to slowdowns or failures.

• Service Dependency Mapping: See which services interact with each other, allowing for a clear view of critical service dependencies and helping identify potential cascading failures.

• Root Cause Analysis (RCA): Trace issues back to the source by identifying slow, failed, or error-prone transactions and drilling down to pinpoint problematic services or infrastructure.

Performance Optimization

• Identify Performance Bottlenecks: Detect slow components, such as long-running queries, network delays, or overloaded services, and make data-driven decisions to optimize them.

• Capacity Planning: Use metrics to analyze usage patterns, forecast demand, and optimize resource allocation, helping avoid over-provisioning or under-resourcing.

• Compare Release Impact: Measure the impact of new releases on system performance by comparing metrics before and after deployments.

Alerting and Incident Management

• Set Alerts for Key Metrics: Define thresholds and set alerts for anomalies in critical metrics, such as high error rates or slow response times, so that teams can act quickly.

• Incident Response and Remediation: Enable responders to access context-specific insights that accelerate incident response, diagnostics, and resolution times.

User Experience Insights

• Track User Journey Metrics: Analyze performance metrics specific to different user journeys (e.g., checkout flows) to understand the impact of backend performance on user experience.

• End-to-End Latency Perceptions: Collect latency and success metrics for critical user actions to help understand where improvements could enhance user satisfaction.

Custom Business and Operational Metrics

• Monitor Business KPIs: Track custom metrics like transaction success rates, revenue per minute, or user engagement rates to tie technical performance to business outcomes.

• Cost Optimization: Identify underused resources or inefficient services to optimize operational costs and improve infrastructure utilization.

Other Duplo-Specific Use Cases

• Request new capabilities via DuploCloud/Using the OpenTelemetry open-source API.

Additional Functions

Grafana's native Alert manager is used for alerting within the DuploCloud Advanced Observability Suite (AOS). DuploCloud's AOS includes a default set of alerts based on best practices and compliance standards. Custom alerting can be set up to meet your unique needs based on log strings, metrics, etc.

The following image shows an example set of alerts for nodes in Kubernetes:

Configuring Alerts

Set up alerts for key observability metrics (e.g., high CPU or memory usage) to be proactively notified of potential issues.

1. From the DuploCloud Portal, navigate to the Admin AOS Dashboard (Administrator -> Observability -> Advanced -> Dashboard) or the Tenant AOS Dashboard (Observability -> Advanced -> Dashboard).

A profile is essentially a snapshot of your application's performance metrics at a specific point in time. DuploCloud’s Advanced Observability Suite (AOS) uses to provide detailed application profiles. These profiles give you detailed insights into how resources like CPU and memory are being used and help you optimize performance and monitor resource usage.

Pyroscope integrates seamlessly with DuploCloud’s other observability tools, like Tempo (for tracing) and Beyla (for telemetry). This allows you to view profiling data alongside traces and logs in Grafana for a unified monitoring experience. The combination of automatic profiling with the option for manual fine-tuning using OpenTelemetry SDKs helps capture the most relevant data for optimizing application performance.

By default, CPU profiles are collected for all applications, while memory profiles are specifically collected for Go and Java applications, providing deeper insights into performance and resource utilization.

Enabling Profiling

To enable application profiling, you can push profiling data directly to Pyroscope by . For detailed instructions on configuring the client, refer to the .

What Profiles Can Tell You

Pyroscope is the backend for profiles that show the CPU and Memory profile for the application. Exploring profiles helps identify and resolve performance issues, optimize resource usage, and improve efficiency. Key benefits include:

• Identifying Performance Bottlenecks: Pinpointing slow functions or resource-heavy processes.

• Optimizing Resource Utilization: Analyzing CPU and memory usage for better scaling decisions.

• Debugging Latency Issues: Detecting areas with high latency that may affect user experience.

• Root Cause Analysis: Correlating profiling data with logs and traces to identify the cause of performance issues.

Exploring Profiles in the DuploCloud Advanced Observability Suite

1. In the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard (Administrator AOS Dashboard) or Observability -> Advanced -> Dashboard (Tenant AOS Dashboard). The AOS Dashboard displays.

2. Select your Infrastructure from the Infrastructure list box (Administrator AOS Dashboard) or Tenant from the Tenant list box (Tenant AOS Dashboard).

3. Click the Profiles card. The Grafana Profiles dashboard displays.

Viewing Profiles and Traces in Grafana

DuploCloud's Advanced Observability Suite (AOS) allows you to manage Grafana resources, such as dashboards and alerts, using Infrastructure as Code (IAC) through the Grafana Terraform provider. This integration simplifies the authentication process by providing a proxy that automatically handles authentication with DuploCloud’s session tokens, eliminating the need to manage Grafana API tokens manually (see the Terraform example below). For more information, see the .

DuploCloud's Advanced Observability Suite (AOS) leverages Tempo for tracing, with Alloy and Beyla as the data collectors. Beyla, powered by eBPF (extended Berkeley Packet Filter), enables seamless collection of observability data directly from the system kernel without requiring application-level instrumentation. By attaching to system events like network requests and function calls, Beyla efficiently gathers metrics and traces. For more detailed trace information, users can fine-tune their applications using the OTEL SDK, integrating traces into logs for enhanced observability.

What Tracing Can Tell You

Tracing with OpenTelemetry is useful in scenarios where you want to identify, analyze, and resolve performance issues or understand the flow of requests through a distributed system. Tracing is helpful for:

• Debugging latency issues

• End-to-end visibility of requests across services

• Analyzing errors and failures

• Capacity planning and optimization

• Root Cause Analysis (RCA)

• Performance tuning new features

• Validating SLAs

• Exposing misconfigurations

• Understanding user behavior

Using OpenTelemetry with Grafana, you can collect and visualize traces alongside other observability data like metrics and logs, providing a unified view for effective troubleshooting and optimization.

Exploring Traces in the DuploCloud Advanced Observability Suite

1. In the DuploCloud Portal, navigate to Administrator -> Observability -> Advanced -> Dashboard (Administrator AOS Dashboard) or Observability -> Advanced -> Dashboard (Tenant AOS Dashboard). The AOS Dashboard displays.

2. Select your Infrastructure from the Infrastructure list box (Administrator AOS Dashboard) or Tenant from the Tenant list box (Tenant AOS Dashboard).

3. Click the Traces card button. The Grafana Tracing dashboard displays.
4. Use the following functions on the Grafana Tracing dashboard to find and view relevant trace data. See the Grafana documentation for detailed instructions.

• Search Query Builder: Use the search query builder to filter and find traces based on parameters like service name, duration, status codes, or custom tags. This lets you quickly narrow down the traces you're interested in and dive into specific details.

• TraceQL: TraceQL is a powerful query language designed to provide advanced filtering and exploration of your trace data. It lets you run more complex queries for pinpointing specific traces or span data based on conditions like service dependencies, error rates, or trace attributes.

• Service Graph: The Service Graph visualizes the relationships between services in your application. It shows how services communicate with one another and provides a high-level view of trace flow across your system. This view helps you identify bottlenecks or failures in service-to-service interactions.

Inspecting Traces

When you have located a trace of interest using the steps above, you can inspect specific metrics or individual spans to understand the operations that occurred during the trace.

1. From the Grafana Tracing dashboard, click on the trace of interest. The Trace Timeline Viewer displays on the right side of the screen. See the Grafana Tracing documentation for more details.

Mapping Traces to Logs, Metrics, and Profiles for Debugging

The Trace Timeline Viewer can be used to map traces with logs, metrics, and profiles. These details offer insights into timing, status, and metadata, which can help identify performance bottlenecks or failures.

• Trace-to-Logs: Navigate from a trace directly to relevant logs, allowing you to correlate trace details with log data.

• Trace-to-Metrics: Jump to metrics related to a particular exception/span to analyze rates, durations, and other key performance indicators.

• Trace-to-Profiles: Link trace spans directly to profiling data for deeper analysis of resource usage and performance at the code level. This feature allows you to correlate traces with profiles, such as CPU or memory usage, providing fine-grained insights into your system’s behavior.

For detailed instructions on inspecting traces, spans, and associated logs, see this Grafana documentation page. For more information about logging with DuploCloud's AOS, see the DuploCloud Logging page.

DuploCloud leverages automated instrumentation with Grafana Beyla and the OpenTelemetry Operator to provide robust monitoring and tracing without extensive manual configuration. With Grafana Beyla, users get automatic instrumentation out of the box for a wide range of languages, capturing key metrics and trace spans with minimal setup. For languages and use cases not covered by Beyla, users can manually instrument their applications using OpenTelemetry, offering full flexibility to capture detailed telemetry data where automated options are unavailable.

Automated instrumentation using Beyla

• Grafana Beyla is an open-source, eBPF-based auto-instrumentation tool designed to simplify the collection of key observability data for applications written in Go, C/C++, Rust, Python, Ruby, Java, NodeJS, .NET, and more.

• It uses eBPF technology to capture RED metrics (Rate, Error, Duration) and basic trace spans for HTTP/S and gRPC services running on Linux, without requiring code changes or configuration updates.

• Beyla is ideal for getting started quickly with observability and provides a low-overhead way to monitor application performance.

For more information, see the Beyla documentation.

OpenTelemetry Operator for Advanced Tracing

While Beyla captures foundational metrics and spans out of the box, it does not provide distributed tracing or detailed trace spans. To address this, DuploCloud integrates the Kubernetes OpenTelemetry Operator.

• This operator enables auto-instrumentation for services written in .NET, Java, Node.js, Python, and Go based on Pod Annotations.

• DuploCloud ships an Instrumentation object in the OTEL namespace by default, forwarding telemetry data to an OpenTelemetry Collector (Alloy).

• The following annotations are used for different programming languages to enable automatic instrumentation:

  • .NET: instrumentation.opentelemetry.io/inject-dotnet: "duploservices-<<Opentelemetry_tenant>>/otel-instrumentation"

  • Go: instrumentation.opentelemetry.io/inject-go: "duploservices-<<Opentelemetry_tenant>>/otel-instrumentation"

  • Java: instrumentation.opentelemetry.io/inject-java: "duploservices-<<Opentelemetry_tenant>>/otel-instrumentation"

  • Node.js: instrumentation.opentelemetry.io/inject-nodejs: "duploservices-<<Opentelemetry_tenant>>/otel-instrumentation"

  • Python: instrumentation.opentelemetry.io/inject-python: "duploservices-<<Opentelemetry_tenant>>/otel-instrumentation"

• Users can also customize or add OpenTelemetry environmental variables for their Pods, as needed. See the OpenTelemetry documentation for details.

Manual instrumentation

For unsupported languages or use cases not covered by automated tools, manual instrumentation is recommended.

OpenTelemetry provides libraries and documentation to help developers instrument applications manually for a wide range of languages and frameworks. See the OpenTelemetry documentation for more information.