Structured vulnerability extraction from security reports, powered by LLMs

How it works

An adaptive, scanner-aware extraction pipeline

Eight stages turn a raw PDF into standardized, multi-format output, with adaptive re-splitting whenever a chunk fails.

{
  "host": "192.168.1.10",
  "name": "SSL/TLS Deprecated",
  "severity": "High",
  "cvss": 9.8,
  "cve": "CVE-2011-3389",
  "scanner": "openvas"
}

1. 1

   Scanner PDF report

   Text and a layout reconstruction are read in two passes, recovering header context, rejoining lines broken across pages and normalizing ligatures and Unicode artifacts.

2. 2

   Vulnerability-aware segmentation

   A start regex declared in the scanner profile materializes one block per finding, so no vulnerability is ever cut in half.

3. 3

   Chunking and prompt assembly

   Each block is sliced inside the token budget of the target model, using its native tokenizer. The prompt carries an explicit output contract, the JSON schema and two complete examples.

4. 4

   LLM extraction

   Any configured model runs the extraction at temperature zero. Cloud providers and local runtimes are declared the same way.

5. 5

   Validation and repair

   Every response goes through progressive JSON repair, budget verification and truncation detection, with re-splitting proportional to the error before any retry.

6. 6

   Consolidation

   A merge rule declared per scanner removes duplicates and keeps base findings joined to their instances, which matters for Tenable WAS pairs.

7. 7

   Multi-format export

   The same canonical records are written to JSON, CSV, TSV and XLSX, ready to query, diff or load into a database.

8. 8

   Field-by-field evaluation

   Optional. Scores the extraction against a reference on omission, hallucination, exact record match, text similarity and severity classification.

Practical guidance

If you are building an extraction pipeline, do it in this order

The ordering matters more than any individual item. It is the main practical recommendation the research leaves behind.

1. Invest in segmentation before investing in the model

   Identify the semantic unit of the document, here the vulnerability, and guarantee that no record is ever cut in half. This produced more gain than any model swap observed.

2. Tune the token budget and tokenizer per model

   A single set of parameters calibrated for every model disproportionately penalizes the ones with the largest reasoning overhead.

3. Anchor the prompt in an explicit schema

   Give it an output contract and complete examples, and repair malformed output before falling back on retries.

4. Measure coverage, not just similarity

   Otherwise you will declare successful an extraction that simply ignored part of the document. Lexical metrics say nothing about what was never extracted.

5. Only then, choose the model

   With the pipeline stabilized, the dominant criteria become cost, latency and data residency policy.

What comes next

Six directions the dissertation leaves open, in the order the research would take them.

1. Extend the scale evaluation to Tenable WAS and to the other scanners already supported by profile, such as Nessus, Qualys and Rapid7, conditional on obtaining a comparable volume of data.
2. Widen the set of curated references, in number and in application category, with parallel annotation on at least one target and a reported agreement statistic, putting the reliability of the reference on firmer ground.
3. Run the full protocol with local providers such as Ollama, Hugging Face and LM Studio, a prerequisite for deployments in network-isolated environments.
4. Build a fourth version of the pipeline that attacks the residual omission in interpretive fields through field-oriented re-extraction.
5. Report processing time and cost per model systematically, folding operating cost into the selection criteria.
6. Validate the principle in other document domains with repetitive structure, such as technical assessment and audit reports.

Publications

Three papers, one trajectory

Feasibility, evidence and artifact. None of the three would be sufficient on its own.

Each stage answers a concrete limitation of the one before it.

🏆

Best Paper award (2nd place), WRSeg 2025

Recognized at the 9th Workshop Regional de Segurança da Informação e de Sistemas Computacionais (WRSeg), ERRC 2025.

View certificate See all awards

1. 1

   Feasibility

   Structured Extraction of Vulnerabilities in OpenVAS and Tenable WAS Reports Using LLMs

   Machado, B.; Lautert, D.; Kapelinski, C.; Kreutz, D.

   WRSeg, proceedings of the XXII Escola Regional de Redes de Computadores (ERRC 2025), p. 144-150.

   Inventories the fields of both scanners, defines the field-by-field mapping to a unified schema and demonstrates that LLM-based extraction is feasible. ROUGE-L above 0.7 for GPT-4.1 and DeepSeek, with four recurring failure modes characterized, all tied to segmentation and truncation.

   Published Open in the SBC library

2. 2

   Evidence

   MulitaMiner: A Multi-Version Evaluation of LLM-Based Vulnerability Report Extraction

   Machado, B.; Lautert, D.; Kapelinski, C.; Kreutz, D.; Ferrão, I. G.; Bof, A.

   Submitted to SBSeg 2026, under review.

   Builds three successive pipeline versions and measures, over 450 independent runs on the same five models, how much each refinement contributes, isolating the effect of the pipeline from the effect of the model.

   Under review

3. 3

   Artifact

   MulitaMiner: An LLM-Based Tool for Structuring Vulnerability Scanner Reports

   Lautert, D.; Machado, B.; Kapelinski, C.; Kreutz, D.

   Submitted to the open-source track of SBSeg 2026, under review.

   Consolidates the method into the open-source tool and submits it to a real institutional archive: 129 reports and 6,343 findings, measured against the scanner's native export, with the financial cost of every run reported.

   Under review See the repository

@inproceedings{machado2025structured,
  title     = {Structured Extraction of Vulnerabilities in OpenVAS
               and Tenable WAS Reports Using LLMs},
  author    = {Machado, Beatriz and Lautert, Douglas and
               Kapelinski, Cristhian and Kreutz, Diego},
  booktitle = {Anais da XXII Escola Regional de Redes de Computadores (ERRC)},
  publisher = {Sociedade Brasileira de Computa\c{c}\~{a}o (SBC)},
  pages     = {144--150},
  year      = {2025},
  url       = {https://sol.sbc.org.br/index.php/errc/article/view/39195}
}

@mastersthesis{lautert2026mulitaminer,
  title  = {MulitaMiner: engenharia de pipeline para extra\c{c}\~{a}o estruturada
            de relat\'{o}rios de scanners de vulnerabilidades com modelos
            de linguagem},
  author = {Lautert, Douglas},
  school = {Universidade Federal do Pampa},
  type   = {Disserta\c{c}\~{a}o de Mestrado},
  note   = {Programa de P\'{o}s-Gradua\c{c}\~{a}o em Engenharia de Software (PPGES)},
  address = {Alegrete, RS, Brasil},
  year   = {2026},
  url    = {https://mulitaminer.github.io}
}

Presentation

Presentation decks

Two decks cover the project: the dissertation defense, which threads the three studies into a single timeline, and the earlier TCC I defense.

Dissertation · v1

TCC I