Parsing Structured JSON Logs Embedded in Text Using Filebeat Pod Annotations

🧩 Problem A customer reported that they were unable to search or filter logs based on specific JSON fields (e.g., request_id, user_id, company_id) from their application logs in Elasticsearch. The reason? The logs were not pure JSON . Instead, each log line started with a prefix like this: I, [2025-04-27T11:37:24.732923 #2464] INFO -- : [f8650-4140-af6b-8499a3c6] {"method":"GET", ... } This prefix prevented Filebeat from directly parsing the JSON structure. 🛠 Solution To address this, we used Filebeat pod annotations to first dissect the log line and extract the embedded JSON string into a separate field (json_message). Then we used the decode_json_fields processor to parse and flatten the JSON into searchable fields. ✅ Working Filebeat Pod Annotations: PodAnnotations: co.elastic.logs/processors.0.dissect.tokenizer: '%{log_level}, [%{timestamp} #%{pid}] %{severity} -- : [%{request_id}] %{json_message}' co.elastic.logs/processors.0.dissect.field: message co.elastic.logs/processors.0.dissect.target_prefix: '' co.elastic.logs/processors.0.dissect.ignore_failure: 'true' co.elastic.logs/processors.1.decode_json_fields.fields.0: json_message co.elastic.logs/processors.1.decode_json_fields.process_array: 'false' co.elastic.logs/processors.1.decode_json_fields.max_depth: 1 co.elastic.logs/processors.1.decode_json_fields.target: '' co.elastic.logs/processors.1.decode_json_fields.overwrite_keys: 'true' co.elastic.logs/processors.2.drop_fields.fields.0: json_message 🔍 Results • After applying the above annotations, all JSON fields in the logs are now parsed and indexed by Elasticsearch. • Fields like request_id, session_id, method, path, etc. are searchable in Kibana . • Note: Fields with null values (e.g., user_id, company_id) may not appear in results since Elasticsearch does not index null fields by default .