YARA rules lab: write, tune and test rules that find files

yara is the scanner. gcc-mingw-w64-x86-64-win32 is a compiler that builds Windows programs on Linux; you use it once, in step 5, to make a harmless hello.exe to study. Together they take about 250 MB of disk.

$ sudo apt update
$ sudo apt install -y yara gcc-mingw-w64-x86-64-win32
$ yara --version
$ x86_64-w64-mingw32-gcc --version | head -1

...
4.5.0
x86_64-w64-mingw32-gcc (GCC) 13-win32

Why not pip? pip install yara-python gives you YARA inside Python scripts, which is handy later. The apt package gives you the yara and yarac commands this lab uses, with the extra modules (pe, math, hash) already switched on. Ubuntu 24.04 ships YARA 4.5.0.

Nothing in this lab is real malware. You will make five small files: two ordinary text files, one “suspicious” script that only looks bad (it is plain text with a made-up web address and a made-up mutex name), a copy of that mutex name saved the way Windows programs store text, and a file of random bytes. Copy the whole block.

$ mkdir -p ~/yaralab/samples/notes ~/yaralab/samples/drop && cd ~/yaralab
$ cat > samples/invoice_march.txt <<'EOF'
Invoice 1043 for March. Amount due: 120.00 USD.
Pay online at https://billing.example.com/pay
EOF
$ cat > samples/notes/readme.txt <<'EOF'
Project notes. Help pages live at https://docs.example.org/start
EOF
$ cat > samples/drop/updater.ps1 <<'EOF'
# SGLAB practice file. Harmless text, never run it.
$c2 = "http://sg-lab-beacon.invalid/check-in"
$mutex = "SGLAB_Mutex_7731"
Write-Output "sleeping 300 seconds between check-ins"
EOF
$ printf 'mutex=SGLAB_Mutex_7731\n' | iconv -t UTF-16LE > samples/drop/config.dat
$ head -c 4096 /dev/urandom > samples/blob.bin
$ find samples -type f | sort

samples/blob.bin
samples/drop/config.dat
samples/drop/updater.ps1
samples/invoice_march.txt
samples/notes/readme.txt

Why these files? They stand in for what an analyst actually gets handed: a folder from a suspect machine, mostly normal, with one or two things that matter. The .invalid ending is reserved so it can never be a real website. A mutex is a name a program grabs so only one copy of it runs at a time; malware authors often hard-code one, which makes it a handy indicator of compromise (IoC). iconv -t UTF-16LE stores the text with a zero byte after every letter, the way Windows programs usually keep strings. Stay in ~/yaralab for the rest of the lab.

A YARA rule has three parts: meta (notes for humans), strings (what to look for) and a condition (when to call it a match). This one looks for the fake web address and the fake mutex.

$ cat > first.yar <<'EOF'
rule SGLAB_Fake_Beacon
{
    meta:
        author = "you"
        description = "Practice rule: made-up beacon domain and mutex from the lab samples"
    strings:
        $domain = "sg-lab-beacon.invalid" nocase
        $mutex  = "SGLAB_Mutex_7731" ascii wide
    condition:
        any of them
}
EOF
$ yara -s first.yar samples/drop/updater.ps1

SGLAB_Fake_Beacon samples/drop/updater.ps1
0x42:$domain: sg-lab-beacon.invalid
0x6c:$mutex: SGLAB_Mutex_7731

Reading it. The first line is “rule name, file”: a hit. -s adds which strings matched and where, as a byte offset in hex (0x42 is byte 66). nocase ignores upper and lower case. ascii wide means “look for the normal spelling and the Windows two-bytes-per-letter spelling”. any of them means one string is enough. No output at all means no match; YARA stays quiet about files that are clean.

-r walks every folder under the one you name. Then look at the second hit up close, and finally make a stricter copy of the rule that needs both strings.

$ yara -r first.yar samples
$ yara -s first.yar samples/drop/config.dat
$ sed 's/any of them/all of them/' first.yar > strict.yar
$ yara -r strict.yar samples

SGLAB_Fake_Beacon samples/drop/updater.ps1
SGLAB_Fake_Beacon samples/drop/config.dat
SGLAB_Fake_Beacon samples/drop/config.dat
0xc:$mutex: S\x00G\x00L\x00A\x00B\x00_\x00M\x00u\x00t\x00e\x00x\x00_\x007\x007\x003\x001\x00
SGLAB_Fake_Beacon samples/drop/updater.ps1

Reading it. The folder scan found the script and config.dat. The close-up shows why: the mutex is there with a \x00 (zero byte) after every letter. Without the wide keyword that file would have slipped past, a classic false negative. The strict copy, all of them, matches only the script, because config.dat has the mutex but not the web address. You can also write 2 of them or any of ($domain*) to count or group strings. The order of results in a folder scan can differ on your machine; YARA scans with several threads.

Programs are bytes, not sentences. Build a harmless Windows program, then write a rule that recognizes any 32- or 64-bit Windows program by its byte patterns.

$ cat > hello.c <<'EOF'
#include <stdio.h>
int main(void) { puts("Hello from the YARA lab"); return 0; }
EOF
$ x86_64-w64-mingw32-gcc -O2 -s -o samples/hello.exe hello.c
$ cat > hex.yar <<'EOF'
rule SGLAB_Windows_Program
{
    strings:
        $dos_msg = "This program cannot be run in DOS mode"
        $pe_sig  = { 50 45 00 00 ( 64 86 | 4C 01 ) }
    condition:
        uint16(0) == 0x5A4D and $pe_sig and $dos_msg
}
EOF
$ yara -r -s hex.yar samples

SGLAB_Windows_Program samples/hello.exe
0x4e:$dos_msg: This program cannot be run in DOS mode
0x80:$pe_sig: 50 45 00 00 64 86

Reading it. Hex strings go in curly brackets. 50 45 00 00 is “PE” plus two zero bytes, the marker every Windows program carries. The brackets with | mean “either”: 64 86 is the code for a 64-bit program, 4C 01 for 32-bit. You can also use ?? for “any byte” and [2-4] for “skip 2 to 4 bytes”. uint16(0) == 0x5A4D reads two bytes at the very start of the file: “MZ”, the first two letters of every Windows program (stored backwards, which is why it reads 5A4D). Checking the start is fast and stops the rule matching a text file that merely mentions these words.

Byte patterns work, but YARA can also understand Windows programs. The pe module reads the file’s headers and gives you named facts: CPU type, number of sections, and which Windows functions it imports. The console module prints a value while scanning.

$ cat > pe.yar <<'EOF'
import "pe"
import "console"

rule SGLAB_PE_Facts
{
    condition:
        pe.is_pe and
        pe.machine == pe.MACHINE_AMD64 and
        pe.number_of_sections > 3 and
        pe.imports("kernel32.dll", "VirtualProtect") and
        console.log("imphash: ", pe.imphash())
}
EOF
$ yara pe.yar samples/hello.exe
$ yara -D pe.yar samples/hello.exe | grep -E 'library_name|number_of_sections|machine ='

imphash: 8feeb4b8639b9197e7245e82d60db06f
SGLAB_PE_Facts samples/hello.exe
            library_name = "KERNEL32.dll"
            library_name = "msvcrt.dll"
    number_of_sections = 10
    machine = 34404

Reading it. -D dumps everything the module learned (over 500 lines; the grep keeps four). 34404 is 0x8664, the 64-bit code you matched by hand in step 5. VirtualProtect is a normal Windows function that the compiler’s start-up code uses; it is also popular with malware that unpacks itself in memory, which is why analysts write rules about imports. The imphash is a fingerprint of the import list: programs built the same way from the same code share it, so it groups related samples even when their file hashes differ. Your imphash may differ if your compiler version differs. kernel32.dll matched KERNEL32.dll because the DLL name check ignores case.

Real indicators change: tomorrow the address will be different. A regular expression (a pattern, written between slashes) can match the shape of a thing instead of one exact value. #var counts how many times a string appears, and filesize keeps a rule cheap and focused.

$ cat > regex.yar <<'EOF'
rule SGLAB_Script_Calling_Odd_Domain
{
    strings:
        $url = /https?:\/\/[a-z0-9.-]+\.(invalid|test)\/[a-z0-9\/-]*/ nocase
        $var = /\$[a-z0-9_]{1,20} = "/
    condition:
        filesize < 2KB and $url and #var >= 2
}
EOF
$ yara -r -s regex.yar samples

SGLAB_Script_Calling_Odd_Domain samples/drop/updater.ps1
0x3b:$url: http://sg-lab-beacon.invalid/check-in
0x34:$var: $c2 = "
0x62:$var: $mutex = "

Reading it. $url matches any http or https address ending in .invalid or .test (both reserved, never real), whatever the name in front. The slashes inside are written \/ because a plain slash would end the pattern. $var matches a PowerShell variable being set to a quoted value, and the condition wants at least two of them. The invoice and readme have web addresses too, but on .com and .org, so they stay clean. Keep regular expressions tight: a loose one such as /.*http.*/ can make scans crawl on a big folder.

Here is the mistake almost everyone makes once: a rule for something that is in nearly every file. Run it on the samples and on Ubuntu’s own documentation folder, which you know is harmless (analysts call a known-clean collection like this goodware).

$ cat > broad.yar <<'EOF'
rule SGLAB_Too_Broad
{
    strings:
        $http = "http"
    condition:
        $http
}
EOF
$ yara -r broad.yar samples
$ yara -r broad.yar /usr/share/doc 2>/dev/null | wc -l

SGLAB_Too_Broad samples/invoice_march.txt
SGLAB_Too_Broad samples/notes/readme.txt
SGLAB_Too_Broad samples/drop/updater.ps1
1248

Three hits in the samples, two of them innocent, and 1,248 false alarms in the docs folder (your number will differ; it depends on what is installed). Now require something that is actually unusual:

$ cat > tuned.yar <<'EOF'
rule SGLAB_Tuned
{
    strings:
        $http   = "http"
        $badtld = /\.invalid\//
        $mutex  = "SGLAB_Mutex_7731" ascii wide
    condition:
        filesize < 50KB and $http and ($badtld or $mutex)
}
EOF
$ yara -r tuned.yar samples
$ yara -r tuned.yar /usr/share/doc 2>/dev/null | wc -l

SGLAB_Tuned samples/drop/updater.ps1
0

Why this matters. A rule that fires on 1,248 clean files is worse than no rule: people stop reading its alerts, and the one real hit drowns. Tuning means pairing a common string with something rare and keeping a size limit, then testing against goodware before you deploy. Real teams keep a large folder of known-clean files for exactly this check. 2>/dev/null hides “can’t open file” messages for files your user can’t read.

Two more ways to describe a file. Entropy measures how random the bytes look, from 0 (all the same) to 8 (pure noise); packed or encrypted malware scores near 8. A hash is the file’s fingerprint, the most common IoC in threat-intel feeds.

$ cat > entropy.yar <<'EOF'
import "math"

rule SGLAB_High_Entropy
{
    condition:
        filesize > 1KB and math.entropy(0, filesize) >= 7.5
}
EOF
$ yara -r entropy.yar samples
$ cat > show.yar <<'EOF'
import "math"
import "console"
rule show { condition: console.log("entropy ", math.entropy(0, filesize)) }
EOF
$ for f in samples/blob.bin samples/hello.exe samples/drop/updater.ps1; do echo "$f $(yara show.yar $f | head -1)"; done

SGLAB_High_Entropy samples/blob.bin
samples/blob.bin entropy 7.951796
samples/hello.exe entropy 4.658528
samples/drop/updater.ps1 entropy 5.046648

Now turn the script’s SHA-256 hash into a rule. Note the heredoc here is <<EOF without quotes, so the shell fills in $H.

$ H=$(sha256sum samples/drop/updater.ps1 | cut -d' ' -f1); echo "$H"
$ cat > ioc.yar <<EOF
import "hash"

rule SGLAB_Known_Bad_Hash
{
    condition:
        filesize < 1MB and hash.sha256(0, filesize) == "$H"
}
EOF
$ yara -r ioc.yar samples

936a992f088363f3e8eac4facedb8d22a8c8437fe30c09c7f068d50196bd23dc
SGLAB_Known_Bad_Hash samples/drop/updater.ps1

Reading it. Random bytes score about 7.95 (yours will differ slightly, they are random), the small program 4.66, the script 5.05. High entropy alone proves nothing (zip files and photos score high too), so in practice you combine it with other clues, for example “a Windows program and high entropy”. The hash rule matches exactly one file, and would miss it if a single byte changed. That is the bottom of the Pyramid of Pain: hashes are trivial for an attacker to change, while strings, imports and behavior cost them more. Your hash will be the same as ours, because the script’s text is identical. The filesize < 1MB check skips hashing big files, which keeps the scan fast.

EICAR is a 68-byte text file that every antivirus product agrees to detect. It is not a virus and does nothing; it exists so you can test detection without real malware. The command builds it from two halves so this web page itself never contains the full string. Keep it inside your Linux home folder: if you copy it into a Windows folder, Microsoft Defender will (correctly) quarantine it.

$ mkdir -p samples/test
$ printf '%s%s' 'X5O!P%@AP[4\PZX54(P^)7CC)7}$' 'EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' > samples/test/eicar.com
$ wc -c samples/test/eicar.com; sha256sum samples/test/eicar.com
$ cat > eicar.yar <<'EOF'
rule SGLAB_EICAR_Test_File
{
    meta:
        description = "Standard antivirus test file. Harmless by design."
        reference = "https://www.eicar.org/download-anti-malware-testfile/"
    strings:
        $start = "X5O!P%@AP[4\\PZX54"
        $name  = "EICAR-STANDARD-ANTIVIRUS-TEST-FILE!"
    condition:
        $start at 0 and $name and filesize < 128
}
EOF
$ yara -r -m eicar.yar samples

68 samples/test/eicar.com
275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f  samples/test/eicar.com
SGLAB_EICAR_Test_File [description="Standard antivirus test file. Harmless by design.",reference="https://www.eicar.org/download-anti-malware-testfile/"] samples/test/eicar.com

Reading it. 68 bytes and that SHA-256 are the published values for the standard test file, so you know it was built correctly. $start at 0 means “this string must be at the very first byte”, mirroring the EICAR rule that the file must begin with the string. Inside a YARA text string, a backslash is written twice (\\). -m prints the rule’s meta, which is how a good rule tells the next analyst what it is and where it came from.

Before a rule goes anywhere near production it has to pass YARA’s own syntax check. First, a deliberately broken rule; then compile all your good rules into one file and run that.

$ cat > broken.yar <<'EOF'
rule SGLAB_Broken
{
    strings:
        $a = "hello"
    condition:
        $a and $b
}
EOF
$ yara broken.yar samples; echo "exit $?"
$ yarac first.yar hex.yar pe.yar regex.yar tuned.yar entropy.yar ioc.yar eicar.yar lab.yarc && ls -l lab.yarc
$ yara -C -r lab.yarc samples | sort

error: rule "SGLAB_Broken" in broken.yar(7): undefined string "$b"
exit 1
-rw-rw-r-- 1 steven steven 11179 Nov  5 20:04 lab.yarc
SGLAB_EICAR_Test_File samples/test/eicar.com
SGLAB_Fake_Beacon samples/drop/config.dat
SGLAB_Fake_Beacon samples/drop/updater.ps1
SGLAB_High_Entropy samples/blob.bin
SGLAB_Known_Bad_Hash samples/drop/updater.ps1
SGLAB_PE_Facts samples/hello.exe
SGLAB_Script_Calling_Odd_Domain samples/drop/updater.ps1
SGLAB_Tuned samples/drop/updater.ps1
SGLAB_Windows_Program samples/hello.exe
imphash: 8feeb4b8639b9197e7245e82d60db06f

Reading it. YARA refuses the broken rule and exits with 1, naming the rule and the problem (it uses $b in the condition but never defines it). That non-zero exit is what an automated pipeline checks before it ships a rule. yarac compiles many rule files into one fast-loading file; -C tells yara it is compiled. Rule names must be unique across the set, which is why strict.yar (same name as first.yar) is left out. Every sample except the two innocent text files now has at least one hit. The full rule language is in the YARA documentation.

Delete the lab folder, including the EICAR file, the program and all rules.

$ cd ~ && rm -rf ~/yaralab
$ ls ~/yaralab

ls: cannot access '/home/steven/yaralab': No such file or directory

YARA stays installed; you will want it again. To remove the Windows compiler (about 250 MB): sudo apt remove gcc-mingw-w64-x86-64-win32 && sudo apt autoremove.