Research status / 01

SoK: Understanding Security Issues in LLM-based Vulnerability Detection

Project status · snapshot 2026-10-06 · Anonymous submission (double-blind)ANONYMOUS

NDSS-style IEEEtran conference paper · target NDSS 2027

Status at a glance

Compiles cleanly14 pages · no undefined references

86References

4 / 6Research questions / takeaways

7 + 1LLMs evaluated + non-LLM baseline

2026-05-06Last content edit

pp. 1–3Main body

pp. 4–6References

pp. 7–14Appendix A (Prompts) + B (Experiments)

Open items

Some newly written passages (most of Experiment B's write-up, the experiment motivation paragraph) are still marked as proposed edits awaiting co-author review.
Camera-ready items pending: author block, NDSS publication-ID block (still template placeholder), PDF title metadata.

Paper outline

I. Bug and Vulnerability Detection Landscape

Survey · detection tasks · technique evolution · benchmarks & evaluation gaps · metrics

A. Security Dimensions — threats, paradigm-specific weaknesses, open challenges

II. New Insights in Vulnerability Detection

Experiments

A. Robustness of Vulnerability Detection — RQ1–RQ3, T1–T3

B. Factors Affecting Vulnerability Detection — RQ4, T4–T6

Appendix A: Prompts · Appendix B: detailed experiment results

Experiments

A — Robustness

Section II-A · RQ1–RQ3

BigVul · shared base set of 100 vulnerable + 100 non-vulnerable functions for each of C and C++ · zero-shot prompt · six semantic-preserving attacks NT1–NT6; NT5 and NT6 run on C only.

RQ1 clean performance · RQ2 attack effectiveness and direction · RQ3 robustness ranking

Non-LLM transformers outperform LLMs on clean performance, and C is easier than C++. UniXcoder weighted F1 ≈ 0.85 vs. 0.61–0.66 for the seven LLMs; C > C++ (p = 0.0234).

Renaming and fake-sanitization attacks are broadly effective. NT2 and NT3 hit all models; NT5 degrades most models except Gemini and Opus4.7; NT1 and NT6 generally fail on LLMs; NT4 causes model-specific bias.

Robustness varies widely and is orthogonal to clean performance. Most robust: GPT-4o (~4.4% mean relative change), UniXcoder (~6.1%). Least robust: Opus4.7 (~22.9%), Gemini (~19.1%), Llama4 (~18.7%).

Attack descriptions and interpretation

NT1 — Rename variables to vulnerability-related keywords
NT2 — Rename a safe function to a "vulnerable"-sounding name
NT3 — Rename an unsafe function to a "non-vulnerable"-sounding name
NT4 — Insert a dangerous library call (strcpy / strcat) used safely
NT5 — Fake sanitization (e.g., realpath) that does not fix the bug (CWE-22, CWE-787)
NT6 — Macro that names a safe function but expands to an unsafe one (e.g., fgets → gets)

GPT-4o's positive C++ value is an artifact of its near-zero NT3 clean accuracy (tiny denominator).

B — What drives detection

Section II-B · RQ4

7 LLMs × 2,357 MegaVul functions (C, C++, Java) = 16,499 predictions; CWEs mapped to CWE-699 → 11 analysis categories; two Bayesian mixed-effects logistic regressions (complexity proxy = LOC or CCN), 95% credible intervals.

RQ4: how do language, complexity, vulnerability position and CWE category jointly affect recall on vulnerable code?

Models are biased toward "vulnerable", trading higher recall for lower specificity. Adjusted association of the vulnerability label ≈ 2.66 (logit scale; 95% CrI [2.25, 3.02]), consistent across LOC and CCN.

Higher code complexity biases models more strongly toward "vulnerable" across all languages. On benign code, complexity lowers correctness in C++ and Java; on vulnerable code the interaction is positive in all three languages.

CWE category drives detection; the hardest categories are those whose bugs are not local to a single function. CWE random-effect SD: LOC 0.45 [0.24, 0.78]; CCN 0.48 [0.26, 0.86]. Harder: Concurrency Issues, Authorization Errors. Easier: Memory Buffer Errors, Data Neutralization Issues, Numeric Errors.

Status: write-up still pending co-author review. Other factors: language is not a credible factor; vulnerability position has a credible but very weak negative association.

Models

Model	Provider	Version	Open	Reasoning
Llama4	Meta	Llama-4-Scout-17B-16E-Instruct	yes	no
Qwen3	Alibaba	Qwen3-235B-A22B (experts_int8)	yes	yes
Sonnet4	Anthropic	Claude-Sonnet-4-20250514	no	no (thinking off)
Gemini	Google	gemini-2.5-pro	no	yes
GPT-4o	OpenAI	gpt-4o-2024-11-20	no	no
o3	OpenAI	o3-2025-04-16	no	yes
Opus4.7	Anthropic	claude-opus-4-7 (via Amazon Bedrock, max reasoning effort)	no	yes

Non-LLM baseline: UniXcoder, fine-tuned on the BigVul training split for 10 epochs. Temperature 0 except o3 and Opus4.7 (default temperature).

Mean relative accuracy change (%) under NT1–4

	Llama4	Qwen3	o3	Sonnet4	GPT-4o	Gemini	Opus4.7	UniXcoder
C	-21.7	-17.3	-14.0	-17.4	-6.3	-29.1	-37.9	-2.5
C++	-20.3	-10.6	-16.9	-19.3	+6.7	-31.5	-29.1	-6.9

Table caveat

GPT-4o's positive C++ value is an artifact of its near-zero NT3 clean accuracy (tiny denominator).

Survey highlights · Section I

Three task types: binary detection (vulnerable vs. clean); localization / root-cause; classification & severity (CWE / CVE / CVSS).

Technique evolution: static analysis (CodeQL, Coverity, Infer) · dynamic / fuzzing (AFL++, OSS-Fuzz) · symbolic execution (KLEE, angr) → deep learning (DeepBugs, VulDeePecker, DeepDFA) → pretrained transformers (CodeBERT, UniXcoder, LineVul) → LLM-based and agentic detectors.

Benchmark gaps: severe class imbalance; narrow CWE coverage relative to the 940+ known CWE categories; strong C/C++ skew; dataset size ranges from 185 examples to millions of functions.

The benchmark table lists 12 benchmarks (3 bug, 9 vulnerability). Vulnerability rows:

Dataset	Vulnerable	Non-Vulnerable	CWEs
DiverseVul	18.9K funcs	330.5K funcs	150
BigVul	11.8K funcs	253.1K funcs	91
CVEFixes	50.3K methods	18.2K files	180
PrimeVul	6.9K funcs	228.8K funcs	140
CrossVul	13.7K files	13.7K files	168
VulnPatchPairs	13.1K pairs	13.1K pairs	–
SVEN	803 funcs	803 funcs	9
CodeXGLUE (Devign)	13.2K funcs	14.2K funcs	38
SARD	Synthetic	Synthetic	100+

Bug rows: Defects4J, BugSwarm, ManyBugs (no vulnerability counts).

Threat model: training-time data poisoning; inference-time semantic-preserving evasion (renaming, dead code, obfuscation); zero-day limits; forging attacks that flood analysts with false positives; privacy leakage / memorization; supply-chain risk from compromised CVE databases or benchmarks.

“techniques that substantially improve detection capability also enlarge the attack surface”