LLM Security Research Pipeline
The data collection infrastructure behind one of the first large-scale studies of AI-generated vulnerability reports in open-source software.
- Research engineering. Sole implementer of the full pipeline, supervised by Dr. Ahmed Lekssays
- Qatar Computing Research Institute · Cybersecurity Department · HBKU
- May 2026 — September 2026
- repositories across GitHub and GitLab
- 4,646
- repositories across GitHub and GitLab
- tables in the PostgreSQL schema
- 17
- tables in the PostgreSQL schema
- output files written crash-safe
- ~2M
- output files written crash-safe
- CWE classes auto-tagged
- 10
- CWE classes auto-tagged
The problem
In February 2026, curl shut down its bug bounty program. This is software installed on billions of devices, and fewer than 5% of the AI-submitted vulnerability reports it was receiving were valid. libxml2, SQLite, CPython, and Kubernetes Ingress NGINX have reported the same pattern: plausible-sounding, machine-written reports describing functions that don't exist, eating the volunteer time that keeps open source running.
Practitioners talk about this constantly. The research literature had almost nothing empirical. Measuring the problem at scale needs a dataset of security-relevant issues, pull requests, commits, and comments across thousands of repositories. That dataset did not exist. My job was to build the system that creates it.
The engineering problem
Collecting from 4,646 repositories across two platforms means millions of API calls against hard rate limits (GitHub: 5,000 requests/hour; GitLab: 2,000/minute), weeks of continuous running on a remote server, and roughly two million output files. At that scale, everything that can fail, will: the process crashes mid-write, the filesystem slows down, the API bans you for patterns you didn't know were abusive.
So the pipeline is designed around one assumption (it will be interrupted) and one requirement: when it resumes, it loses nothing and repeats nothing.
Architecture
A single CSV of 4,646 repositories drives a Python scraper that normalises GitHub REST v3 and GitLab API v4 into one collection model: issues, PRs and merge requests (with per-PR detail calls for diff stats), repository commits, PR-linked commits, and four kinds of comments. Each item is written as JSON plus human-readable Markdown through atomic temp-file-then-rename writes. A CWE classifier tags security-relevant items across 10 vulnerability classes as they're collected. A loader then maps the files into a 17-table PostgreSQL schema (UUID keys, named enums, JSONB, upsert logic) that I implemented from my mentor's design document. Scraper, database, and loader each run as separate Docker containers on a remote Linux research server.
repos.csv — 4,646 repositories
One CSV drives everything: GitHub and GitLab projects selected for the study
Python collector, built to be interrupted
Issues, PRs/MRs, commits, and four kinds of comments — normalised across both platform APIs
~2M JSON + Markdown files
Every write is temp-file-then-rename — a crash can never leave a truncated file
Schema mapper with upsert logic
Maps per-item files onto the relational schema; ON CONFLICT keeps reruns idempotent
17 tables — UUID keys, enums, JSONB
Implemented from the research design document; queryable the moment data lands
$ ssh research-server && tail -f scrape.log[06:41:12] repo 1893/4646 — issues page 14[06:41:13] saved issue #41283 security=true cwe=[CWE-89][06:41:15] github rate 4821/5000 — window resets 06:58[06:41:17] gitlab 429 — backing off 42s (+jitter)[06:41:59] resumed repo 1893 at issue #41284 — nothing lost$ jq .totals state/progress.json{ "repos_done": 1892, "issues": 1249, "comments": 3258 }
17 tables total · UUID primary keys · named enums · JSONB payloads · ON CONFLICT upserts keep the loader idempotent
Engineering decisions
Atomic writes for every file
A crash mid-write leaves a truncated JSON file that poisons the loader later. So every write goes to a temp file first, then renames, which the filesystem guarantees is all-or-nothing. After that change, corrupted output simply stopped being a failure mode.
State tracking at the item level, not the repo level
Tracking only completed repositories means a crash inside a 3,000-issue repository re-scrapes all of it. The state file tracks completed issue numbers, PR numbers, and commit SHAs per repo, so a resume picks up at the exact item where it stopped.
Three rate-limit regimes, handled differently
GitHub primary limits announce themselves in X-RateLimit headers. GitHub secondary abuse detection hides in 403 message bodies. GitLab sends plain 429s. Treat them identically and you either waste hours sleeping or get the scraper banned, so each gets its own detection and its own backoff-with-jitter strategy.
SHA-level deduplication
The same commit arrives through repository history and through PR-linked commits. A per-repo seen-SHA set guarantees each commit is written exactly once, keeping the dataset clean at the source instead of cleaning it later.
One container per service
Scraper, PostgreSQL, and loader are separate Docker containers, so the same composition runs identically on my laptop and the research server, and each piece restarts independently.
What was hard
Roughly two million output files degrade filesystem performance and make any non-resumable process effectively unrunnable.
Atomic writes plus item-level state plus SHA dedup turned a fragile long-running job into one that survives crashes, restarts, and server maintenance without data loss.
I had never used PostgreSQL, and the schema needed enums, UUID keys, JSONB columns, indexes, and conflict-safe upserts.
Implemented the full 17-table schema from the design document, learning PostgreSQL by building the real thing: DDL, foreign keys, ON CONFLICT logic, and a loader that maps millions of JSON files onto it.
Long-running jobs on a remote server fail silently if nobody is watching.
Background jobs under nohup, structured logging, and a state file that doubles as a progress report. Every morning starts with tail -f and a count of repos done, items saved, errors hit.
Outcome
The collection infrastructure for the entire study is built and running in production on the research server, with 1,249+ issues and 3,258+ comments collected and counting. CWE tagging lets the team put annotation effort into security-relevant items first, and the schema and loader mean data is queryable the moment it lands.
The research is targeting a top-tier security venue. Every downstream study (measurement, case studies, maintainer interviews, a detection classifier) stands on this pipeline.
- SQL injection
- XSS
- Path traversal
- Command injection
- Auth bypass
- Privilege escalation
- Buffer overflow
- CSRF
- Open redirect
- Insecure deserialisation
I went in thinking AI security meant attacks on models. This project reframed it: AI is also a tool that erodes the human attention holding open-source infrastructure together. That's a threat model too.