ASAysha Shafique
← Back to all projects
LLM AGENTS · SECURITY

Agentic Coding Reliability · NSF Cybersecurity-Funded

An NSF-funded study of whether software written by LLM coding agents is actually safe to ship. I rebuilt a classic UNIX reliability fuzzer into faster research tooling, tested a 12-program corpus of utilities including grep, make, and ptx, and root-caused real memory-safety and denial-of-service bugs.

0112 programs · grep, make, ptx + more
024× faster evaluation tooling
0310+ source defects investigated
04uutils ptx · 29× faster · 19% less peak memory
01

The research question

As coding agents take on larger software tasks, teams need objective evidence about whether the resulting systems are secure and dependable. This work develops a repeatable way to evaluate those properties without relying on subjective code-quality judgments.

  • Studies both security failures and reliability failures in systems software.
  • Uses consistent, reproducible evaluation conditions across a 12-program corpus.
02

Modernizing the Miller UNIX reliability fuzzer

I upgraded the classic Miller et al. UNIX reliability fuzzer into faster, more dependable research infrastructure for long-running campaigns and reproducible failure analysis.

  • Parallelized independent fuzz inputs to improve evaluation throughput by 4×.
  • Made hang detection 2× faster with more reliable process monitoring.
  • Reaped detached process trees to eliminate resource leaks during long-running campaigns.
03

Diagnosing crashes and security defects

The evaluation infrastructure also exposed failures in mature Unix source implementations. I reduced those failures to reproducible cases, traced their causes, and prepared fixes for upstream submission.

  • Root-caused more than 10 crashes, memory-safety bugs, and denial-of-service defects across five widely used implementations.
  • Traced failures to exact source locations and CWE classes using AFL++, AddressSanitizer, and GDB.
  • Eliminated quadratic prefix rescans in uutils ptx with cached character offsets, cutting a 2 MB single-line workload from 10.02 seconds to 0.34 seconds (~29× faster).
  • Replaced per-word filename clones with indexes into the existing file map, reducing average peak memory from 66.6 MiB to 54.0 MiB (19%).
Where to?