Practical Examples for Using Awk Command in Text Processing in Linux

In this post, we’ll show you some practical examples  to use awk command for processing text files in Linux Systems. Those examples I used them many times for many practical situations and will share them with you. I’ll use question and answer for examples in this post.

Awk is a scripting language used for manipulating data and generating reports.The awk command programming language requires no compiling, and allows the user to use variables, numeric functions, string functions, and logical operators.

Let’s start

Q1. How can we remove duplicate lines from a text file without sorting?

One time, I needed to remove the duplicated lines from an ordered text file ‘my_text_file’ and I must keep the orders of lines. I keep only the first occurrence/appearance of each line and remove the other duplicated lines.

Simply, run this command:

cat my_text_file | awk '!x[$0]++'

You can save the results to another file by running this command:

cat my_text_file | awk '!x[$0]++' > new_unique_without_sorting

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.