Is This Model File Safe? Scan a Booby-Trapped Model

Model files are pickles, and a pickle can run code the instant you load it. picklescan reads the file without running it and tells you whether it would. Install it:

$ pip install picklescan

It is a few kilobytes and pulls in nothing heavy. You now have a tool that opens the box and reads the wiring instead of plugging it in.

You need something dangerous-shaped to scan, and you are not going to download real malware to get it. This lab ships a tiny generator (make-demo-model.py, in the lab files) that writes a pickle whose load instruction calls eval() — exactly the shape a malicious model uses — but with a payload that only reads the current year. Harmless even if you slipped and loaded it. Save the file and run it:

$ python3 make-demo-model.py

Wrote demo-model.pkl -- do NOT run pickle.load / torch.load on it.
Scan it instead:  picklescan -p demo-model.pkl

The one thing that matters here: the trap fires on load, not on scan. From here on you only ever scan it.

A real attacker does not use “read the year”. They use “download this and run it”. Same mechanism, one string different. That is why you treat every unscanned model as live. — Nyx

Point the scanner at the file:

$ picklescan -p demo-model.pkl

demo-model.pkl: dangerous import 'builtins eval' FOUND
----------- SCAN SUMMARY -----------
Scanned files: 1
Infected files: 1
Suspicious globals: 0
Dangerous globals: 1

There it is: dangerous import ‘builtins eval’. The scanner found that loading this file would reach for eval — a function that runs arbitrary code — and it told you without running anything. It also sets its exit code to 1, which is what lets you wire it into a script or a pipeline.

So you know what “safe” looks like, make an ordinary pickle — just data, no code — and scan that too:

$ python3 -c "import pickle; pickle.dump({'weights':[0.1,0.2,0.3]}, open('clean-model.pkl','wb'))"
$ picklescan -p clean-model.pkl

----------- SCAN SUMMARY -----------
Scanned files: 1
Infected files: 0
Suspicious globals: 0
Dangerous globals: 0

Zero dangerous globals, exit code 0. A model that is only weights has nothing to call. The difference between the two scans is the entire skill.

A clean result is “I did not find a known-dangerous call”, not “this is safe”. Two real limits:

• Scanners can be dodged. Researchers have found real malicious models on Hugging Face that slipped past pickle scanners by hiding the payload in a deliberately broken pickle stream, which the scanner gave up on but the loader still partly ran. The host pulled them, but the lesson stands: a pass is a filter, not a guarantee.
• A second opinion helps. Other scanners (ModelScan is the common one) read the same files with slightly different rules. On anything you actually care about, run more than one.

Why this matters. This is AI supply chain (OWASP LLM04). A downloaded model is downloaded code. You would not curl | bash a stranger’s script; loading their .pkl or .bin is the same act with the risk hidden inside a “model”.

The real fix is to stop using a format that can carry code at all. safetensors stores only numbers — there is no load-time code path to abuse. The habits that follow from that:

• Prefer safetensors over .pt/.pkl/.bin whenever the model offers both.
• Scan before you load, every time, and fail closed on a non-zero exit.
• Load anything untrusted in a throwaway VM — the isolated range from the next lab is exactly the place — never on your daily machine.

Delete the demo files when you are done. They were only ever a target for the scanner:

$ rm demo-model.pkl clean-model.pkl