LLM Security Research Pipeline
The data collection infrastructure behind one of the large-scale studies of AI-generated vulnerability reports in open-source software.
- Research engineering. Sole implementer of the full pipeline, supervised by Dr. Ahmed Lekssays
- Qatar Computing Research Institute · Cybersecurity Department · HBKU
- May 2026 - July 2026
- repositories across GitHub and GitLab
- 4,646
- repositories across GitHub and GitLab
- tables in the PostgreSQL schema
- 17
- tables in the PostgreSQL schema
- output files written crash-safe
- ~ 2M
- output files written crash-safe
The problem
In February 2026, curl shut down its bug bounty program. This is software installed on billions of devices and fewer than 5% of the AI-submitted vulnerability reports it was receiving were valid. libxml2, SQLite, CPython and Kubernetes Ingress NGINX have reported the same pattern: plausible-sounding, machine-written reports describing functions that don't exist, eating the volunteer time that keeps open source running.
Practitioners talk about this constantly.Measuring the problem at scale needs a dataset of security-relevant issues, pull requests, commits and comments across thousands of repositories. My job was to build the system that creates the database to train AI model to identify which reports are AI generated and which ones are written by human.
The engineering problem
Collecting from 4,646 repositories across two platforms means millions of API calls against hard rate limits (GitHub: 5,000 requests/hour), hours of continuous running on a remote server and roughly two million output files. At that scale, everything that can fail, will and the process crashes mid-write, the filesystem slows down, the API bans you for patterns you didn't know were abusive.
So the pipeline is designed around one assumption (it will be interrupted) and one requirement: when it resumes, it loses nothing and repeats nothing.
Architecture
A single CSV of 4,646 repositories drives a Python scraper that normalises GitHub REST v3 and GitLab API v4 into one collection model: issues, PRs and merge requests (with per-PR detail calls for diff stats), repository commits, PR-linked commits and four kinds of comments. Each item is written as JSON plus human-readable Markdown through atomic temp-file-then-rename writes. A CWE classifier tags security-relevant items across vulnerability classes as they're collected. A loader then maps the files into a 17-table PostgreSQL schema (UUID keys, named enums, JSONB, upsert logic) that I implemented. Scraper, database and loader each run as separate Docker containers on a remote Linux research server.
repos.csv — 4,646 repositories
One CSV drives everything: GitHub and GitLab projects selected for the study
Python collector, built to be interrupted
Issues, PRs/MRs, commits and four kinds of comments — normalised across both platform APIs
~2M JSON + Markdown files
Every write is temp-file-then-rename — a crash can never leave a truncated file
Schema mapper with upsert logic
Maps per-item files onto the relational schema; ON CONFLICT keeps reruns idempotent
17 tables — UUID keys, enums, JSONB
Implemented from the research design document; queryable the moment data lands
17 tables total
Engineering decisions
Atomic writes for every file
A crash mid-write leaves a truncated JSON file that poisons the loader later. So every write goes to a temp file first, then renames, which the filesystem guarantees is all-or-nothing. After that change, corrupted output simply stopped being a failure mode.
State tracking at the item level, not the repo level
Tracking only completed repositories means a crash inside a 4,000+ issue repository re-scrapes all of it. The state file tracks completed issue numbers, PR numbers and commit SHAs per repo, so a resume picks up at the exact item where it stopped.
Three rate-limit regimes, handled differently
GitHub primary limits of 5,000 requests per hour. GitHub secondary abuse detection hides in 403 message bodies. GitLab sends plain 429s. Treat them identically and you either waste hours sleeping or get the scraper banned, so each gets its own detection and its own backoff-with-jitter strategy.
SHA-level deduplication
The same commit arrives through repository history and through PR-linked commits. A per-repo seen-SHA set guarantees each commit is written exactly once, keeping the dataset clean at the source instead of cleaning it later.
One container per service
Scraper, PostgreSQL and loader are separate Docker containers, so the same composition runs identically on my laptop and the research server and each piece restarts independently.
What was hard
Roughly two million output files degrade filesystem performance and make any non-resumable process effectively unrunnable.
Atomic writes plus item-level state plus SHA dedup turned a fragile long-running job into one that survives crashes, restarts and server maintenance without data loss.
I had never used PostgreSQL and the schema needed enums, UUID keys, JSONB columns, indexes and conflict-safe upserts.
Implemented the full 17-table schema from the design document, learning PostgreSQL by building the real thing: DDL, foreign keys, ON CONFLICT logic and a loader that maps millions of JSON files onto it.
Long-running jobs on a remote server fail silently if nobody is watching.
Background jobs under nohup, structured logging and a state file that doubles as a progress report. Every morning starts with tail -f and a count of repos done, items saved, errors hit.
Outcome
The collection infrastructure for the entire study is built and running in production on the research server, with 1,249+ issues and 3,258+ comments collected and counting. CWE tagging lets the team put annotation effort into security-relevant items first and the schema and loader mean data is queryable the moment it lands.
The research is targeting a top-tier security venue. Every downstream study (measurement, case studies, maintainer interviews, a detection classifier) stands on this pipeline.
- SQL injection
- XSS
- Path traversal
- Command injection
- Auth bypass
- Privilege escalation
- Buffer overflow
- CSRF
- Open redirect
- Insecure deserialisation
We usually frame AI security around adversarial attacks. This work highlighted a subtle threat that showed AI as an operational wedge that strains open-source maintainers and degrades the human layer of infrastructure.