Set up logging

Set up logging for the DuploCloud Portal

Prerequisites

• If you need to make changes to the Control Plane Configuration, follow this procedure to do so, before enabling logging. Note that you cannot modify the Control Plane Configuration after you set up logging.

• Docker applications use stdout for writing log files, collecting logs, placing them in the Host directory, mounting them into Filebeat containers, and sending them to AWS Elasticsearch. If you need to customize the log collection and you use folders other than stdout, for example, follow this procedure. Note that you cannot customize the log collection after you set up logging.

Setting up logging

1. In the DuploCloud Portal, navigate to Administrator -> Observability -> Settings -> Logging.

2. From the Tenant list box at the top of the DuploCloud Portal, select the Default Tenant.

3. Click the Create Logging link. The Enable Logging page displays.

   

   Create Logging link

4. Use the Enable Logging page to deploy logging for the Control Plane, which uses OpenSearch and Kibana to retrieve and display log data for the Default Tenant. In the Cert ARN field, enter the ARN certificate for the Default Tenant. Find the ARN by selecting the Default Tenant from the Tenant list box at the top of the DuploCloud Portal; navigating to Administrator -> Plans; selecting the Plan that matches your Infrastructure Name; and clicking the Certificates tab.

   

   Enable Logging page

5. Click Submit. Data gathering takes about fifteen (15) minutes. When data gathering is complete, graphical logging data is displayed in the Logging tab.

6. After logging has been enabled for the Control Plane, finish the logging setup by enabling the Log Collector to collect logs per Tenant. This feature is especially useful for Tenants that are spread across multiple regions. In the DuploCloud Portal, navigate to Administrator -> Observability -> Settings -> Logging.

7. In the Logging tab, on the Logging Infrastructure Tenants page, click Add.

   

   Add button on the Logging Infrastructure Tenants page
8. Select the Tenants for which you want to configure logging, using the Select Tenants to enable logging area, as in the example below. The Control Plane configuration is deployed for each Tenant that you select in the Infrastructure, specified in Infrastructure Details.

Logging tab with logging setup complete

The Log Collector uses Elastic Filebeat containers that are deployed within each Tenant.

When you enable a Tenant for logging, the Filebeat service starts up and begins log collection. View the Filebeat containers by navigating to Kubernetes -> Containers in the DuploCloud Portal. In the row of the container for which you want to view the logs, click on the menu icon and select Logs.

How DuploCloud configures logging for you

When you perform the steps above to configure logging, DuploCloud does the following:

Control Plane deployment

1. An EC2 Host is added in the Default tenant, for example, duploservices-default-oc-diagnostics.

2. Services are added in the Default tenant, one for OpenSearch and one for Kibana. Both services are pinned to the EC2 host using allocation tags. Kibana is set up to point to ElasticSearch and exposed using an internal load balancer.

3. Security rules from within the internal network to port 443 are added in the Default Tenant to allow log collectors that run on Tenant hosts to send logs to ElasticSearch.

Log Collector deployment

1. A Filebeat service (filebeat-duploinfrasvc) is deployed for each Tenant where central logging is enabled.

2. The /var/lib/docker/Containers are mounted from the Host into the Filebeat container. The Filebeat container references ElasticSearch, which runs in the Default Tenant. Inside the container, Filebeat is configured so that every log line is added with metadata information consisting of the Tenant name, Service names, Container ID, and Hostname, enabling ease of search using these parameters with ElasticSearch.