Hunt Failed Logins in SSH Logs With Splunk

On Debian and Ubuntu, SSH logins are written to /var/log/auth.log. Take a copy you own (so you never upload a live file Splunk holds open) and hand that copy to Splunk:

$ sudo cp /var/log/auth.log ~/auth-$(date +%F).log
$ sudo chown $USER ~/auth-*.log

In the Splunk web UI: Settings → Add Data → Upload, pick that file, and on the Set Source Type screen choose linux_secure — Splunk’s built-in type for auth logs. Send it to an index you can find again (the default main is fine for a lab). For a server you want to watch continuously, use Settings → Add Data → Monitor instead and point it at the file.

On a systemd box — Fedora, RHEL, Arch — there is often no auth.log at all; sshd logs to the journal. Export it first with journalctl _COMM=sshd --since today > ssh-$(date +%F).log, then upload that. Red Hat’s file equivalent is /var/log/secure. — Nyx

Confirm it landed. In the search bar, with an All time window:

> index=main sourcetype=linux_secure

You should see your auth lines in the events list. If you see nothing, widen the time range — a copied log keeps its original dates, not today’s.

Why this matters. A SIEM is only as good as the data in it, and the first failure of every Splunk beginner is an empty search because the sourcetype or time range was wrong. Getting one known file in cleanly, and proving it is there, is the step everything else stands on. For reading the same log with no SIEM at all, see Lab 09 and the journalctl cheat sheet.

Narrow to the failures, then teach Splunk to read the source address and the username out of each line with rex:

> index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| rex "for (?:invalid user )?(?<user>\S+) from"
| table _time, user, src

An auth line reads Failed password for invalid user admin from 192.0.2.44 port 40569 ssh2. The first rex captures the IP after from into a field called src; the second captures the username after for, skipping the optional “invalid user” so it works whether or not the account exists. You now have a clean table of time, user and source for every failed login.

The built-in linux_secure sourcetype extracts some of these on its own, and versions disagree about what they are called. Writing the rex yourself takes ten seconds and works the same on every Splunk, every log, forever. Do that. — Nyx

Why this matters. Raw log text is unsearchable in aggregate — you cannot count or group a sentence. The moment a value becomes a named field, every command after this (stats, top, timechart) can slice by it. Field extraction is the whole game; the detections are just arithmetic on good fields.

Count failures per source address, loudest first. This is stats, the command you will use more than any other:

> index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| stats count by src
| sort - count

On the sample log, that returns a table that reads like this — the same numbers grep gives on the raw file, which is how you can trust the query:

src             count
192.0.2.44      430
203.0.113.9     194
198.51.100.200   60
198.51.100.7      1

Four addresses, and the shape of the day is already visible: one address failed 430 times, a second 194, a third 60, and one address failed exactly once. For the short version — just the worst offenders — top does the count, sort and percentage in a single word:

> index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| top limit=10 src

Why this matters. Volume is the cheapest signal there is and it is usually right: the address at the top of this list is the one to look at first. But hold the thought about that single failure at the bottom — one failed password is not an attack, it is a human mistyping a password. Volume finds the noisy attackers. It will not find the quiet one, and step 6 is about the quiet one.

A count tells you how much; a timechart tells you when. Bucket the failures by hour:

> index=main sourcetype=linux_secure "Failed password"
| timechart span=1h count

Switch the result to a column chart under Visualization. On the sample log the hours stack up like this — one enormous bar and the rest background noise:

hour   failures
02:00     430
03:00      39
04:00     155
09:00       1
12:00      33
13:00      27

The busiest single minute of the whole day had 23 failures. A real user does not fail to log in 23 times in a minute. Split the chart by source to see each attacker as its own line:

> index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| timechart span=1m count by src

Why this matters. Time is context. A hundred failures spread over a week is a tired user or a broken cron job; a hundred in ninety seconds is a script. The same timechart is also how you answer “is it still happening right now?” — the question that decides whether this is an incident or a report. For the full timechart toolkit — spans, trendlines, big log files — see Lab 14.

Volume alone cannot tell a password-spray (many accounts, one guess each) from a targeted hammering of one account. Count the distinct usernames per source with dc():

> index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| rex "for (?:invalid user )?(?<user>\S+) from"
| stats dc(user) as users_tried count as attempts by src
| where users_tried > 5
| sort - attempts

On the sample log:

src             users_tried  attempts
192.0.2.44              20       430
198.51.100.200          20        60

Two addresses each cycled through 20 different usernames — root, admin, oracle, postgres and the rest of the standard list every bot tries. That is a spray. Notice the where users_tried > 5 filter quietly dropped the address from step 6 that only ever went after one or two accounts — a different pattern, caught by a different query.

The 198.51.100.200 one is the one people miss: only 60 attempts, slow enough to slide under a volume alert, but still 20 accounts from one address. “Low and slow” is just a spray with patience. Count accounts, not only attempts. — Nyx

Why this matters. The account count changes the response. Twenty usernames from one address is a bot reading a wordlist — block the address and move on. Five hundred attempts against one real username is someone who knows that account exists and wants it — a different, more worrying conversation.

A thousand failed logins are a nuisance. One success after a thousand failures is an incident. Ask Splunk for any source that both failed a lot and succeeded at least once — count each outcome per address in one pass:

> index=main sourcetype=linux_secure ("Failed password" OR "Accepted password")
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| rex "for (?:invalid user )?(?<user>\S+) from"
| stats count(eval(searchmatch("Failed password"))) as fails
        count(eval(searchmatch("Accepted password"))) as wins
        values(user) as users by src
| where fails > 20 AND wins > 0

One row comes back, and it is the whole point of the lab:

src           fails  wins  users
203.0.113.9     194     1  backup root

That address never showed up in the spray list — it went after just backup and root, failed 194 times, and then one attempt on backup succeeded. To see the attack as a single timeline instead of two counts, stitch the run together with transaction — open a group on a failure, close it on the success:

> index=main sourcetype=linux_secure ("Failed password" OR "Accepted password")
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| transaction src maxspan=1h startswith="Failed password" endswith="Accepted password"
| table _time, src, duration, eventcount

You get one row holding the 190 failures and the login that ended them — a single clickable event with a start time, a duration, and an eventcount. That is the brute force and its payoff in one line.

The defender’s move. This query is the alert worth having. Failed logins alone are too noisy to page anyone; “failures over a threshold followed by a success from the same address” almost never fires by accident, and when it does fire it means someone is already inside. The fix for the hole it found is older than Splunk: key-only SSH, no password auth, and backup should never have been able to log in over SSH at all.

A successful login from an address or country you have never used before is worth a second look even when nothing failed. Splunk’s iplocation command adds City, Country, Region and coordinates to any field holding an IP:

> index=main sourcetype=linux_secure "Accepted password"
| rex "from (?<src>\d+\.\d+\.\d+\.\d+)"
| rex "for (?:invalid user )?(?<user>\S+) from"
| iplocation src
| stats count by src, Country, user

On the sample log the successes group into just three source-and-user pairs, and one of them should stop you:

src           Country  user     count
198.51.100.7           steven     8
198.51.100.7           deploy     2
203.0.113.9            backup     1

The Country column is empty here on purpose: these are reserved documentation addresses, and Splunk’s own docs say an address with no location simply gets no fields added. On a real server that same column fills in — and a backup login from a country you do not operate in is the tell. Even blank, the table earns its place: 198.51.100.7 is your admin logging in as steven and deploy, as always; 203.0.113.9 is an address that had never succeeded before today. A brand-new source for a successful login is the thing to flag.

Why this matters. Geography and “first time seen” are context that volume and timing miss entirely. The strongest version of this alert is not “login from country X” — it is “login from any source we have not seen succeed before”, which catches the attacker whether they come from the next town or the next continent.

A query you run by hand catches yesterday. An alert catches tonight. Take the step 6 search — failures over a threshold followed by a success — and in the search view choose Save As → Alert.

Set it to run on a schedule (every 15 minutes over the last 15 minutes is a sane start), trigger if the number of results is greater than 0, and add a throttle on the src field so one noisy attacker does not alert you sixty times an hour. The action can be as simple as an email to yourself while you are learning.

Resist alerting on step 3’s raw failed-password count. It will fire every single night, you will mute it inside a week, and a muted alert catches nothing. Alert on the rare, meaningful thing — the success after the failures — not the loud, constant thing. — Nyx

Why this matters. The gap between “I can find it when I look” and “it finds me when I am not looking” is the whole difference between log reading and detection. One good scheduled alert, tuned so it almost never cries wolf, is worth more than a dashboard nobody opens. Which SIEM you build it in barely matters once you can write the query — see Which SIEM Should You Learn.