CloudWatch Logs Insights – Difference Between Display and Fields Directives

amazon-cloudwatchamazon-web-services

What is the difference between display and fields directives in CloudWatch Logs Insights query syntax?

These are descriptions from the AWS documentation that look very similar to me :

display: Specifies which fields to display in the query results.

fields: Retrieves the specified fields from log events for display.

As an example, I have logs stored in Cloudwatch in this structure (with these fields):

  • @timestamp
  • @message
  • stream (stdout|stderr)
  • kubernetes.namespace_name
  • …

Here are examples of valid queries that confuse me:

  1. I can display any non-retrieved field:
limit 8
| display @message, stream
  1. I can display a field even if I haven't specified it in fields.
fields @message, stream
| limit 8
| display @message, stream, kubernetes.namespace_name
  1. It doesn't matter if I specify a field in fields when parsing:
fields @message
| parse @message "[*] *" as loggingType, loggingMessage
| display loggingMessage

parse @message "[*] *" as loggingType, loggingMessage
| display loggingMessage

What is the meaning of the fields directive? Wouldn't it be enough to just use display?

Best Answer

The difference in fields and display commands is that fields behavior is cumulative and display is not (replace-like behavior).

From the CloudWatch Logs Insights query syntax guideline:

If your query contains multiple fields commands and doesn't include a display command, you'll display all of the fields that are specified in the fields commands.

So a display command would replace the output defined by any preceding display or fields commands (or any other command that defines ephemeral fields), and the fields would add to the currently defined output.

Examples:

  1. Returns @timestamp, @message, @logStream, @log fields
fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| limit 20
  1. Returns @message
fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| limit 20
| display @message
  1. Returns @message, requestId, text, hasRequestId
fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| limit 20
| display @message
| parse @message "(*) *" as requestId, text
| fields !isblank("requestId") as hasRequestId

The last example defines ephemeral fields using parse command (these do not have @ in the beginning) and expression field defined by last fields command

Related Topic