Logging for AWS in the DuploCloud Platform
The DuploCloud Platform performs centralized logging for Docker-based applications. For the native and Kubernetes container orchestrations, this is implemented using OpenSearch and Kibana with Elastic Filebeat as the log collector. For ECS Fargate, AWS Lambda, and AWS SageMaker Jobs, the platform integrates with CloudWatch, automatically setting up Log Groups and making them viewable from the DuploCloud Portal.
No setup is required to enable logging for ECS Fargate, Lambda, or AWS SageMaker Jobs. DuploCloud automatically sets up CloudWatch log groups and provides a menu next to each resource.
To maintain optimal performance and cost-efficiency, it's crucial to manage logging resources effectively. If you find yourself with unnecessary monitoring hosts or logging instances, specific steps should be taken to clean them up without affecting essential services.
To terminate unnecessary monitoring hosts in DuploCloud, it's recommended that a designated user, referred to as Person 0, performs the termination. This approach ensures that essential services, such as Prometheus, are not inadvertently removed, which could lead to loss of data or configurations.
Cleaning up a logging instance involves several steps, starting with remote access into DuploMaster. From there, navigate to the appropriate directories to edit and delete specific files related to the unintended tenant. This includes removing entries from the logging_config.json and deleting tenant-specific JSON files. Additionally, tenant services related to OpenSearch, Kibana, and Elastic Filebeat need to be deleted, followed by the termination of the oc-diagnostics host. It's also necessary to remove specific entries from the DuploCloud portal related to reverse proxy settings and platform services.
When a host or a Load Balancer (LB) is no longer required, consider stopping or deleting them as part of cost optimization measures. Before taking such actions, ensure they do not contain or support essential services that could impact your infrastructure's operation.
By following these guidelines, you can ensure that your logging resources in DuploCloud are managed efficiently, contributing to both operational effectiveness and cost savings.
Set up central logging for the DuploCloud Default Tenant
The Default Tenant in DuploCloud is the central management space for platform-wide resources and configurations, including monitoring and logging. Enabling logging in the Default Tenant deploys comprehensive Control Plane monitoring. This deployment uses OpenSearch and Kibana to retrieve and display log data. Once logging is enabled for the Default Tenant, you can enable logging for non-Default Tenants and configure logging per Tenant.
Central logging is typically set up during DuploCloud onboarding. Contact DuploCloud Support if you have questions about this process.
If needed, make changes to the Control Plane Configuration. You cannot modify the Control Plane Configuration after you set up logging.
If needed, customize Elastic Filebeat logging. Docker applications use stdout to write log files, collect logs, place them in the Host directory, mount them into Filebeat containers, and send them to AWS Elasticsearch. If you need to customize log collection using folders other than stdout, follow this procedure. Log collection cannot be customized after logging is set up.
From the Tenant list box at the top of the DuploCloud Portal, select the Default Tenant.
In the DuploCloud Portal, navigate to Administrator -> Observability -> Basic -> Settings, and select the Logging tab.
Click the Enable Logging link. The Enable Logging page displays.
In the Select Tenant list box, select Default.
In the Cert ARN field, enter the ARN certificate for the Default Tenant.
Find the ARN certificate by selecting the Default Tenant from the Tenant list box at the top of the DuploCloud Portal, navigating to Administrator -> Plans, selecting the Plan that matches your Infrastructure Name, clicking the Certificates tab, and copying the ARN from the Certificate ARN column.
Enter the number of days to retain logs in the Log Retention in Index (Days) field.
Click Submit. Data gathering takes about fifteen (15) minutes. When data gathering is complete, graphical logging data is displayed on the Logging tab.
When you enable logging for a Tenant, an Elastic Filebeat Service starts and begins log collection. The Elastic Filebeat Service must be running for log collection to occur.
To view the Filebeat Service, navigate to Kubernetes -> Services. To view the Filebeat containers, navigate to Kubernetes -> Containers. In the row of the container, click on the menu icon and select Logs.
Once logging is enabled for the Default Tenant, you can enable logging for other Tenants.
When you perform the steps above to configure logging, DuploCloud does the following:
An EC2 Host is added in the default Tenant, for example, duploservices-default-oc-diagnostics.
Services are added in the default Tenant, one for OpenSearch and one for Kibana. Both services are pinned to the EC2 host using allocation tags. Kibana is set up to point to ElasticSearch and exposed using an internal load balancer.
Security rules from within the internal network to port 443 are added in the default Tenant to allow log collectors that run on Tenant hosts to send logs to ElasticSearch.
A Filebeat service (filebeat-duploinfrasvc) is deployed for each Tenant where central logging is enabled.
The /var/lib/docker/Containers are mounted from the Host into the Filebeat container. The Filebeat container references ElasticSearch, which runs in the Default Tenant. Inside the container, Filebeat is configured so that every log line is added with metadata information consisting of the Tenant name, Service names, Container ID, and Hostname, enabling ease of search using these parameters with ElasticSearch.
Configure log collection per Tenant in the DuploCloud Portal
Tailor your logging data to your specific needs by configuring log collection per Tenant.
Before configuring logging for each Tenant, enable logging for the Default Tenant and enable logging for non-Default Tenants, if needed.
If a Tenant is not included in the Enable/Disable logs collection for tenants area, ensure that you have completed the listed prerequisites.
From the DuploCloud Portal, navigate to Administrator -> Observability -> Basic -> Settings, and select the Logging tab.
In the Enable/Disable logs collection for tenants area, select the Tenants for which you want to enable log collection.
Click Update. Elastic Filebeat Service begins log collection for the selected Tenants.
When you enable logging for a Tenant, an Elastic Filebeat Service starts and begins log collection. The Elastic Filebeat Service must be running for log collection to occur.
To view the Filebeat Service, navigate to Kubernetes -> Services.
Enable log collection for non-Default DuploCloud Tenants
Enable logging to deploy AWS Log Collector to collect logs for selected Tenant(s). Once logging is enabled for Tenant(s), you can configure log collection, tailoring your log data to display only relevant information.
Before configuring logging per Tenant, set up central logging for the Default Tenant.
Configure AWS Log Collector to collect logs for non-Default Tenants.
From the DuploCloud Portal, navigate to Administrator -> Observability -> Basic -> Settings.
Select the Logging tab, and click Add. The Enable Logging pane displays.
In the Select Tenant list box, select the Tenant for which you want to enable log collection.
In the Cert ARN list box, select the correct ARN.
In the Log retention in Index(Days) field, enter the number of days logs should be retained.
Click Submit. Log collection for the selected Tenant deploys based on the Control Plane configuration.
Once logging is enabled, you can configure logging per Tenant.
Display logs for the DuploCloud Portal, components, services, and containers
The central logging dashboard displays detailed logs for Service and Tenant. The dashboard uses Kibana and preset filters that you can modify.
In the DuploCloud Portal, navigate to Observability -> Logging.
Select the Tenant from the Tenant list box at the top of the DuploCloud Portal.
Select the Service from the Select Service list box.
Modify the DQL to customize Tenant selection, if needed.
Adjust the date range by clicking Show dates.
Add filters, if needed.
DuploCloud pre-filters logs per Tenant. All DuploCloud logs are stored in a single index. You can see any Tenant or combination of Tenants (using the DQL option) but the central logging control plane is shared, with no per-Tenant access.
Confirm that your Hosts and Services are running or runnable to view relevant log data.
See Kubernetes Containers for information on displaying logs per container.
Change configuration for the Control Plane, customize Platform Services
There are several use cases for customized log collection. The central logging stack is deployed within your environment, as with any other application, streamlining the customization process.
The version of OpenSearch, the EC2 host size, and the control plane configuration are all deployed based on the configuration you define in the Service Description. Use this procedure to customize the Service Description according to your requirements.
You must make Service Description changes before you enable central logging. If central logging is enabled, you cannot edit the description using the Service Description window.
In the DuploCloud Portal, navigate to Administrator -> System Settings.
In the Service Description tab, in the Name column, select duplo_svd_logging_opensearch. The Service Description window displays.
Edit the YAML in the Service Description window as needed.
Click Update when the configuration is complete to close the window and save your changes.
You can update the Control Plane configuration by editing the Service Description. If the control plane is already deployed using the Service Description specification, then updating the description is similar to making a change to any application.
Note that Control Plane Components are deployed in the DuploCloud Default Tenant. Using the Default Tenant, you can change instance size, Docker images, and more.
You can update the log retention period using the OpenSearch native dashboard by completing the following steps.
From the DuploCloud portal, navigate to Administrator -> Observability -> Logging.
Click Open New Tab to access the OpenSearch dashboard.
Navigate to Pancake -> Index management -> State management policies.
Edit the FileBeat YAML file and update the retention period.
For more information see the OpenSearch documentation.
The new retention period settings will only apply to logs generated after the retention period was updated. Older logs will still be deleted according to the previous retention period settings.
You can modify Elastic Filebeat logging configurations, including mounting folders other than /var/lib/docker for writing logs to folders other than stdout.
You need to customize the log collection before enabling logging for a Tenant.
If logging is enabled, you can update the Filebeat configuration for each tenant by editing the Filebeat Service Description (see the procedure in Defining Control Plane Configuration).
Alternately, delete the Filebeat collector from the Tenant and the platform automatically redeploys based on the newest configuration.
In the DuploCloud Portal, navigate to Administrator -> System Settings.
Select the Platform Services tab.
Click the Edit Platform Services button. The Platform Services window displays. Select the appropriate Filebeat service. For native container management, select filebeat; for Kubernetes container management, select filebeat-k8s.
Edit the YAML in the Platform Services window as needed.
Click Update to close the window and save your changes.
With DuploCloud, you have the choice to deploy third-party tools such as Datadog, Sumo Logic, and so on. To do this, deploy Docker containers that act as collectors and agents for these tools. Deploy and use these third-party app containers as you would any other container in DuploCloud.
