Linux – How to filter (out) certain output in bash

bashlinuxloggingunix

I have a process that tails our AWS CloudWatch logs. It works great, except for the fact that every line is prepended by the CloudWatch group and container ID. For example:

core-api 04a7ce3daf83aa018a92eb4a613c87354695dc8219f213697f7f786b81d4d [PROD] - INFO: GET 200 - 5ms core-api 04a7ce3daf83aa018a92eb4a613c87354695dc8219f213697f7f786b81d4d [PROD] - INFO: POST 200 - 7ms

Is there a way to pipe my logs command I use with some type of filter to only show the line info after [PROD] -?

Best Answer

You can try using cut to cut out the columns you are interested in.

tail -f whatever | cut -d ' ' -f 3-

[PROD] - INFO: GET 200 - 5ms
[PROD] - INFO: POST 200 - 7ms

This -d sets a space as the field delimiter, and -f specifies to display only the third and subsequent fields.

Specifying fields can get more complex, too. Suppose you only really want to get rid of that annoying useless second field. You can then specify -f 1,3-.

core-api [PROD] - INFO: GET 200 - 5ms
core-api [PROD] - INFO: POST 200 - 7ms

See the cut man page for more things you can do with this command.

Related Solutions

Bash Scripting – How to Determine if a Bash Variable is Empty

This will return true if a variable is unset or set to the empty string ("").

if [ -z "${VAR}" ];

Difference Between Double and Single Square Brackets in Bash – Scripting Shell POSIX

There are several differences. In my opinion, a few of the most important are:

[ is a builtin in Bash and many other modern shells. The builtin [ is similar to test with the additional requirement of a closing ]. The builtins [ and test imitate the functionality /bin/[ and /bin/test along with their limitations so that scripts would be backwards compatible. The original executables still exist mostly for POSIX compliance and backwards compatibility. Running the command type [ in Bash indicates that [ is interpreted as a builtin by default. (Note: which [ only looks for executables on the PATH and is equivalent to type -P [. You can execute type --help for details)
[[ is not as compatible, it won't necessarily work with whatever /bin/sh points to. So [[ is the more modern Bash / Zsh / Ksh option.
Because [[ is built into the shell and does not have legacy requirements, you don't need to worry about word splitting based on the IFS variable to mess up on variables that evaluate to a string with spaces. Therefore, you don't really need to put the variable in double quotes.

For the most part, the rest is just some nicer syntax. To see more differences, I recommend this link to an FAQ answer: What is the difference between test, [ and [[ ?. In fact, if you are serious about bash scripting, I recommend reading the entire wiki, including the FAQ, Pitfalls, and Guide. The test section from the guide section explains these differences as well, and why the author(s) think [[ is a better choice if you don't need to worry about being as portable. The main reasons are:

You don't have to worry about quoting the left-hand side of the test so that it actually gets read as a variable.
You don't have to escape less than and greater than < > with backslashes in order for them not to get evaluated as input redirection, which can really mess some stuff up by overwriting files. This again goes back to [[ being a builtin. If [ (test) is an external program the shell would have to make an exception in the way it evaluates < and > only if /bin/test is being called, which wouldn't really make sense.

Best Answer

Related Solutions

Bash Scripting – How to Determine if a Bash Variable is Empty

Difference Between Double and Single Square Brackets in Bash – Scripting Shell POSIX

Related Topic