Linux Command: uniq

by mike on March 8, 2011

Use the uniq utility with files that have consecutive, identical lines.   When you look at the output of /var/log/secure in this example you see a list of IP Addresses that are the same.

You can get  rid of the duplicates that were identical in case.  But if you still have some consecutive words that are the same, except for case.  Use the “-i” switch to take care of that.

You can get a count of how many consecutive duplicate lines there are with the “-c” switch.  Combine it with the “-i” switch to make the count case-insensitive.
grep  "Aug 13" /var/log/secure | uniq -c
1 Aug 13 22:19:28 mail sshd[1872]: Server listening on :: port 22.
1 Aug 13 22:19:28 mail sshd[1872]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
1 Aug 13 22:20:22 mail sshd[2112]: Accepted password for root from 192.168.5.103 port 51208 ssh2
1 Aug 13 22:20:22 mail sshd[2112]: pam_unix(sshd:session): session opened for user root by (uid=0)
1 Aug 13 23:40:18 mail sshd[1888]: Server listening on :: port 22.
1 Aug 13 23:40:18 mail sshd[1888]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
1 Aug 13 23:45:32 mail sshd[2128]: Accepted password for root from 192.168.5.103 port 43125 ssh2
1 Aug 13 23:45:32 mail sshd[2128]: pam_unix(sshd:session): session opened for user root by (uid=0)
1 Aug 13 23:47:50 mail sshd[2128]: pam_unix(sshd:session): session closed for user root
1 Aug 13 23:47:57 mail sshd[2163]: Accepted password for root from 192.168.5.103 port 38934 ssh2
1 Aug 13 23:47:57 mail sshd[2163]: pam_unix(sshd:session): session opened for user root by (uid=0)
1 Aug 13 23:50:43 mail sshd[2163]: pam_unix(sshd:session): session closed for user root
1 Aug 13 23:50:51 mail sshd[2209]: Accepted password for root from 192.168.5.103 port 38935 ssh2
1 Aug 13 23:50:51 mail sshd[2209]: pam_unix(sshd:session): session opened for user root by (uid=0)

The “-u” switch will allow you to only display lines that aren’t repeated.  You can also make this case-insensitive by combining it with the “-i” switch.

Use the “-d” switch to show one copy of each line that is repeated, and to not show any line that isn’t repeated.  Again, you can combine this with the “-i” switch.
grep  "Aug 13" /var/log/secure | uniq -d

Use the “-f” switch to tell uniq to ignore fields in its comparison.  In this case, a field is a word in a sentence.  The number after the “-f” tells uniq how many fields to ignore.

grep  "Aug 13" /var/log/secure | uniq -f16
Aug 13 22:19:28 mail sshd[1872]: Server listening on :: port 22.
Aug 13 22:19:28 mail sshd[1872]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 13 22:20:22 mail sshd[2112]: Accepted password for root from 192.168.5.103 port 51208 ssh2
Aug 13 23:40:18 mail sshd[1888]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Aug 13 23:45:32 mail sshd[2128]: Accepted password for root from 192.168.5.103 port 43125 ssh2

Comments on this entry are closed.

Previous post:

Next post: