Linux uniq Command: 6 Log Analysis Tricks SOC Analysts Use

Linux uniq Command Tutorial: Find Attack Patterns in Log Files Fast

Quick Answer: The Linux uniq command collapses adjacent duplicate lines. Paired with sort and uniq -c, it ranks repeated IPs, usernames, and errors in seconds, which makes it a core threat detection tool.

Last verified: September 20, 2026 (tested against GNU coreutils behavior; always confirm flags with man uniq on your distribution).

Picture a 2 a.m. alert: an authentication log has ballooned to 40,000 lines overnight. Nobody is going to read that by hand. In practical Linux log analysis, the fastest question is not "what happened?" but "what happened the most?" One short pipeline built around uniq answers that question, and it works on a jump box, a forensic image, or a production server where you cannot install anything.

This guide walks through every common uniq option, then applies each to real SOC scenarios: brute-force detection, web log triage, duplicate account checks, and rare-event hunting.

Table of Contents

• How uniq Actually Works (and the Mistake Everyone Makes)
• Quick Reference: uniq Options at a Glance
• Basic uniq Options With Examples
• The sort + uniq Pipeline: Where the Power Is
• Comparing Part of a Line: -w, -f, and -s
• Real-World Scenarios for SOC Analysts
• Saving Results Safely
• Turning Counts Into Incident Response Actions
• Expert Tips From the Analyst Desk
• FAQ
• Conclusion

How uniq Actually Works (and the Mistake Everyone Makes)

Per the GNU coreutils documentation, uniq only compares each line with the line directly before it. It does not search the whole file for duplicates.

That is the single most common beginner mistake. If the same IP address appears on line 3 and line 900, plain uniq will not treat them as duplicates. That is why almost every real-world use pairs it with sort, which pushes identical lines next to each other first.

For a file containing these lines in this order:

alice
alice
bob
alice

uniq outputs alice, bob, alice. Only the first two are adjacent, so only they collapse.

Quick Reference: uniq Options at a Glance

Command	What It Does
uniq file.txt	Removes adjacent duplicate lines
uniq -c file.txt	Prefixes each line with its occurrence count
uniq -d file.txt	Shows only lines repeated consecutively
uniq -u file.txt	Shows only lines with no adjacent duplicate
uniq -i file.txt	Ignores case when comparing
uniq -w5 file.txt	Compares only the first 5 characters
uniq -f1 file.txt	Skips the first field when comparing
uniq -s3 file.txt	Skips the first 3 characters when comparing

Basic uniq Options With Examples

Remove adjacent duplicates

uniq file.txt

What it does: Prints the file with consecutive repeated lines collapsed into one.
When to use it: On data that is already ordered, such as a log where a service repeats the same message back to back.
Expected output: The same file, with runs of identical lines shortened to a single line.

Count occurrences

uniq -c file.txt

What it does: Adds a count before each line, showing how many times that line repeated consecutively.
When to use it: To measure how noisy a repeated event is.
Expected output: Lines like 5 Connection reset by peer.

Show only duplicated lines

uniq -d file.txt

What it does: Prints one copy of each line that appears more than once in a row.
When to use it: When you only care about what repeats, not what is unique.

Show only non-repeated lines

uniq -u file.txt

What it does: Prints only lines that have no adjacent duplicate.
When to use it: To isolate one-off events. After sorting, this becomes a strong way to surface rare activity.

Ignore case

uniq -i file.txt

What it does: Treats Admin, ADMIN, and admin as the same line.
When to use it: When comparing usernames, hostnames, or domains, where attackers and applications vary capitalization. Sort with sort -f first so case variants land next to each other.

The sort + uniq Pipeline: Where the Power Is

Sorting first fixes the adjacency limitation. These are the patterns you will use most.

sort file.txt | uniq

Removes all duplicate lines, not just adjacent ones. This is equivalent to sort -u file.txt.

sort file.txt | uniq -c

Shows how many times each distinct line occurs anywhere in the file.

sort file.txt | uniq -d

Shows every line that occurs more than once in the whole file.

sort file.txt | uniq -u

Shows only lines that occur exactly once in the whole file.

sort file.txt | uniq -c | sort -nr

Counts each line, then sorts numerically in reverse so the most frequent entries appear first. This is the pattern analysts call a "top talkers" view.

sort file.txt | uniq -c | sort -nr | head

Limits the output to the top 10 entries. Add -n 20 to head for more.

Comparing Part of a Line: -w, -f, and -s

Real log lines carry timestamps, PIDs, and other noise that makes otherwise identical events look different. These three flags tell uniq what to ignore. According to the GNU documentation, they are applied in this order: fields skipped first, then characters skipped, then the width limit.

Compare only the first N characters

uniq -w5 -c file.txt

If each log line begins with an HH:MM timestamp, this groups events by minute. Because logs are already chronological, no sort is needed. A sudden jump in one minute's count is a fast way to spot a burst of activity. Note that BSD-based systems such as macOS may not support -w, so check man uniq there.

Skip the first N fields

sort -k2 app.log | uniq -f1 -c

Ignores the first whitespace-separated field (often a timestamp) and counts identical messages. Sorting with -k2 keeps the sort key aligned with what uniq compares.

Skip the first N characters

uniq -s3 file.txt

Ignores a fixed-width prefix, such as a three-character severity tag, when deciding whether two lines match.

Real-World Scenarios for SOC Analysts

The examples below use documentation-reserved IP ranges (203.0.113.x, 198.51.100.x, 192.0.2.x) and illustrative numbers. Your own output will differ.

Scenario 1: Ranking failed SSH logins by source

grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

What it does: Pulls failed SSH login lines, extracts the source IP, and ranks IPs by attempt count. On standard OpenSSH log lines, the IP sits four fields from the end, just before port, the port number, and ssh2. Verify the field position against your own log format first.
When to use it: When investigating suspected password guessing.
Illustrative output:

   1842 203.0.113.45
    611 198.51.100.23
     12 192.0.2.77

A few IPs with thousands of failures and a long tail of single attempts is a classic guessing pattern. On RHEL-family systems the log is typically /var/log/secure instead of auth.log.

Scenario 2: Web server traffic triage

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10

What it does: Counts requests per client IP. In the common Apache and Nginx combined log formats, the client IP is the first field.
When to use it: When a site slows down, or when you suspect scanning or scraping. A single address dominating the list is worth a closer look, though large corporate proxies and CDNs can also produce high counts legitimately.

Scenario 3: Repeated error messages

sort errors.log | uniq -c | sort -nr

What it does: Groups identical error messages and ranks them.
When to use it: During incident response to separate background noise from a new, high-volume error. If timestamps make every line unique, strip them first with uniq -f or cut.

Scenario 4: Auditing local accounts for duplicates

cut -d':' -f1 /etc/passwd | sort | uniq -c

What it does: Lists each username with its count.
Expected output: Every count should be 1. Anything higher deserves investigation.

cut -d':' -f3 /etc/passwd | sort -n | uniq -d
awk -F: '$3==0 {print $1}' /etc/passwd

The first command shows any UID that appears more than once. The second lists every account with UID 0. On a healthy system, only root should appear. Additional UID 0 accounts are a well-known persistence indicator that defenders check during compromise assessments.

Scenario 5: Hunting the rare, not the common

sort process_list.txt | uniq -c | sort -n | head

What it does: Reverses the ranking so the least frequent entries come first.
Why it matters: Threat hunters often use this "least frequency" approach across process names, scheduled task names, or service names collected from many hosts. Legitimate software tends to appear on many machines, while an item found on one or two systems is a lead worth checking. It is a lead, not proof, since custom internal tools also appear rarely.

Scenario 6: Cleaning indicator lists and CSVs

sort ip.txt | uniq -d
sort users.csv | uniq

The first shows IPs that appear more than once in your list. The second deduplicates a CSV export. Note that sort reorders rows, so the header line will move. Save it separately if you need it kept on top.

Saving Results Safely

sort file.txt | uniq > unique.txt

This writes the deduplicated result to unique.txt.

Warning: never redirect output back into the file you are reading, as in sort file.txt | uniq > file.txt. The shell truncates the file before the pipeline reads it, and you can lose your data. When working with evidence or original logs, write to a new filename and keep the source untouched.

Turning Counts Into Incident Response Actions

Counting is only the first step. Here is how analysts typically act on what uniq -c reveals.

• Validate first. Check whether a high-count IP belongs to a known scanner, VPN, or corporate gateway before escalating.
• Correlate. Search for the same source in other logs, such as firewall, VPN, and web logs. A pipeline result on one host is a lead for a SIEM query, not a conclusion.
• Contain proportionally. Depending on policy, options include firewall rules, rate limiting, or temporary blocks. Document each action for the incident record.
• Harden. Common defensive measures include key-based SSH authentication, multi-factor authentication, disabling direct root login, and limiting exposure of management ports. No single control guarantees protection, so layer them.
• Centralize. Command-line pipelines are excellent for triage, but sustained monitoring belongs in a SIEM or log platform. NIST SP 800-92 provides general guidance on log management practices.

Also remember that logs can contain personal data such as IP addresses and usernames. Handle and retain them according to your organization's privacy and compliance obligations under frameworks like GDPR or HIPAA, where they apply.

Expert Tips From the Analyst Desk

• Sort consistently. Set LC_ALL=C before sort and uniq for predictable ordering and faster performance on large files.
• Pair -i with sort -f. Otherwise, case variants may not end up adjacent, and uniq -i will miss them.
• Trim noise first. Use awk or cut to keep only the field you care about before counting. Whole-line comparison rarely groups log data well.
• Read the leading spaces. uniq -c right-aligns counts with padding. If you feed the result into another tool, normalize it with awk '{print $1","$2}'.
• Know when to switch tools. For huge datasets or complex grouping, awk associative arrays, jq, or a SIEM query will scale better than a shell pipeline.
• Keep a snippet library. The five or six pipelines above cover a large share of everyday triage.

Related Cybersecurity Topics You Should Explore

• Brevo Hack Exposed 100,000+ Sites to ClickFix: Check Now
  
• Linux sort Command Guide: Rank Attacker IPs in Seconds
  
• Settra Ransomware: The Log Attackers Forgot to Clear
  
• UAE's AI Lab Tests Every Model for Hidden Risks
  
• Check Point CVE-2026-91843: Root Access, No Login Needed (Patch Now)
  
• Why 40% of Indian SMEs Got Breached Despite Rising Cybersecurity Budgets
  
• cPanel's EmailTrack Bug: How Low-Privilege Users Reach Root Access

FAQ

Why does uniq not remove all my duplicates?

Because it only compares adjacent lines. Run sort first, or use sort -u.

What is the difference between uniq -d and uniq -u?

-d prints one copy of lines that are repeated. -u prints only lines that are not repeated. They are opposites.

How do I sort uniq -c output by most frequent?

Pipe it to sort -nr. Use sort -n instead to see the least frequent entries first.

Is sort | uniq the same as sort -u?

For simple duplicate removal, yes. But sort -u cannot count occurrences, so sort | uniq -c is needed when counts matter.

Can uniq compare only part of a line?

Yes. Use -f to skip fields, -s to skip characters, and -w to limit the number of characters compared. Support for -w varies on non-GNU systems.

Is uniq suitable for large log files?

It streams its input and handles large files well, but sorting large files is the slow part. Filter with grep or awk first to reduce volume.

Can uniq replace a SIEM?

No. It is a fast triage tool for a single file or stream. A SIEM adds correlation across sources, retention, alerting, and dashboards.

Conclusion

The uniq command is small, but in the hands of a SOC analyst it turns a wall of log text into a ranked list of leads. Remember the core rule: uniq only sees adjacent lines, so sort first. Then build from there with -c, -d, -u, and the field and character options.

Try the SSH and web log pipelines on your own systems this week and see what floats to the top. If this guide helped, share it with a teammate, and leave a comment with your favorite one-liner for log triage.

Analysis based on SOC monitoring practice, GNU coreutils documentation, and public log-management guidance.