KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

KaliBench. A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards. By Pengfei Li and three others, from Khalifa University and the University of Western Australia. Published on the first of October. Its code repository is public and real: about thirty source files.

Language models are increasingly used in cybersecurity workflows, where the job is to translate an analyst's intent into a tool invocation. Current evaluations either test security knowledge or run end-to-end agentic tasks. Neither directly measures whether a model can write an executable command for a real cybersecurity tool. That matters because these operations run through strict command-line interfaces. A minor syntax error, a flag bound to the wrong value, or arguments in the wrong order can each be enough to make the command fail. A model can understand exactly what the analyst wants and still produce something the tool refuses to run, and knowledge tests would never notice.

KaliBench is a fine-grained benchmark and dataset for translating natural language into command-line invocations on Kali Linux. It assesses two things precisely and reproducibly: whether the model picks the right tool, and whether it builds the arguments correctly. Those are the two places where a command line goes wrong, and the benchmark keeps them separate so that a failure can be traced to one or the other.

It contains eight thousand five hundred and four query and command pairs, spanning one thousand six hundred and forty-two tools, grouped into twenty-three capability dimensions and five security phases. Construction follows a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation. Canonicalization and alias handling mean that two commands written differently but meaning the same thing can be scored as the same, which is what lets the grading stay exact without being brittle.

To make sure each command is both semantically correct and actually runs, a multi-stage verification pipeline combines validation by a language model, execution in a sandboxed terminal, and refinement with a human in the loop. Because the resulting signals are fine-grained and deterministic, KaliBench can also supply verifiable rewards for training without needing to run the commands at all.

Evaluation covers three modes and twenty-four configurations of general-purpose and security-focused open-weight models. In the unrestricted setting, no open-weight model exceeds forty-two percent exact-command accuracy, which shows how hard this is without explicit hints about which tool to use. Training on KaliBench, with supervised fine-tuning and with reinforcement learning on its verifiable rewards, significantly improves an eight billion parameter model, bringing it to performance comparable with a six hundred and eighty-five billion parameter mixture-of-experts model.

KaliBench is both a test and a training signal. As a test it isolates the narrow step where intent becomes a command line, and grades it exactly. As a signal it lets a small model close much of the distance to one many times its size, without a live runtime in the loop.