The Evolution of Web Application Firewalls: From RegEx Rules to AST Tokenizers and Wasm Filters
For two decades, web application firewalls sat in front of application servers, inspecting HTTP requests before traffic reached backend code. The operational goal remained constant: drop malicious payloads without breaking legitimate traffic. The underlying mechanics of how these engines evaluate bytes, manage memory, and handle structured protocols have changed completely across four distinct architectural phases.
String matching treats structured languages as arbitrary text. Attackers exploit the discrepancy between what the regular expression engine evaluates and how the backend SQL interpreter parses the token stream.
2018 – Regular expression engines and ReDoS vulnerabilities
In 2018, ModSecurity v2 and v3 paired with the OWASP Core Rule Set (CRS 3.x) represented the standard deployment model for HTTP inspection. These engines evaluated incoming URIs, request headers, and buffered POST bodies against large collections of regular expressions. Each rule ran sequentially against the request context inside Nginx or Apache worker processes.
This design suffered from severe architectural flaws. Regular expressions with overlapping branches and nested quantifiers exposed inspection engines to Regular Expression Denial of Service (ReDoS). An attacker sending a crafted string could force the PCRE engine into exponential backtracking, pinning CPU cores at 100% and starving legitimate connections.
Evasion was trivial because string-based rules lacked awareness of backend grammar. Attackers bypassed pattern checks by splitting SQL keywords across inline comments, using multibyte character truncation, or exploiting nested URL and Unicode encoding differences between the proxy and the target database parser.
# Legacy ModSecurity regex matching against SQL injection patterns
SecRule ARGS "@rx (?i:(?:union\s+all\s+select|select\s+.*\s+from))" \
"id:1001,phase:2,deny,status:403,log,msg:'SQL Injection Attempt'"
# Trivial bypass using inline SQL comments and MySQL variable concatenation
# Target payload: UN/**/ION/**/SEL/**/ECT 1,user(),32020 – Abstract syntax tree tokenization and libinjection
By 2020, security engineering moved away from maintaining thousands of brittle regular expressions. Libinjection, developed by Nick Galbreath, changed how detection engines analyzed SQL injection and cross-site scripting payloads. Rather than looking for specific keywords or substrings, libinjection implemented a lightweight lexical analyzer and finite state machine.
The parser transformed arbitrary input strings into five-character grammar fingerprints. For example, a payload like 1' OR '1'='1 mapped to the fingerprint s&1c (string, binary operator, literal, comment). If the resulting token structure matched known SQL evaluation branches, the engine flagged the request regardless of how many comments, whitespace variations, or obfuscated characters the attacker inserted.
This structural approach reduced false positive rates and cut inspection latency. However, libinjection had strict limitations. It focused exclusively on SQL and XSS grammars, failed to handle modern NoSQL syntax, and could not parse structured JSON documents or nested microservice RPC payloads without external preprocessors.
2022 – WebAssembly filters in proxy ingress pipelines
The rise of service mesh architectures and Envoy proxies displaced monolithic C modules. Compiling third-party C/C++ security modules directly into ingress proxies created stability and maintenance liabilities. A single null-pointer dereference or buffer overflow in a legacy rule parser would crash the entire gateway process.
The Proxy-Wasm standard established an Application Binary Interface (ABI) for running sandboxed inspection filters inside WebAssembly runtimes like V8 and Wasmtime. Security teams wrote rules in Rust, compiled them to WebAssembly bytecode, and loaded them dynamically into running Envoy instances without restarts.
# Envoy proxy-wasm filter configuration for dynamic rule loading
http_filters:
- name: envoy.filters.http.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
config:
name: "coraza-wasm-filter"
root_id: "coraza_root"
vm_config:
runtime: "envoy.wasm.runtime.v8"
code:
local:
filename: "/etc/envoy/filters/coraza.wasm"Isolation protected the host proxy: if a WebAssembly filter panicked on a malformed request, the runtime terminated only that specific sandbox instance. The trade-off came down to memory throughput. Copying request bodies across the host-to-Wasm memory boundary introduced latency on large payloads, pushing engineers to optimize zero-copy buffer sharing.
2024 – Memory-safe engines and streaming body analysis
In 2024, memory safety and streaming execution became the baseline requirements for HTTP inspection engines. OWASP Coraza demonstrated that a native Go implementation of the SecLang standard could replace legacy C codebases in Caddy, Traefik, and Kubernetes ingress controllers without CGo bindings or foreign function interface penalties.
Traditional inspection engines buffered entire multi-megabyte request bodies in memory before evaluating rules, creating memory spikes during high concurrency. Modern engines implement streaming parsers for JSON, YAML, and multipart form-data. Tokens pass through inspection state machines as TCP chunks arrive from the network interface, discarding processed bytes immediately.
// Inspecting incoming body streams without full memory buffering in Go
func ProcessStream(reader io.Reader, engine *coraza.WAF) error {
tx := engine.NewTransaction()
defer tx.Close()
buf := make([]byte, 4096)
for {
n, err := reader.Read(buf)
if n > 0 {
if interrupted, _ := tx.WriteRequestBody(buf[:n]); interrupted != nil {
return errors.New("request blocked by WAF rule")
}
}
if err == io.EOF {
break
}
if err != nil {
return err
}
}
return nil
}Where we are now
Modern application security operates at the transport and protocol decoding layers. Instead of treating HTTP traffic as flat character streams, modern inspection pipelines hook directly into HTTP/2 and HTTP/3 multiplexed frames. Decompression, QPACK header decoding, and stream reassembly happen in memory-safe, zero-allocation buffers.
The goal is parser parity between edge inspection layers and backend application runtimes. When the edge proxy and backend microservice parse JSON keys, multipart boundaries, and Unicode normalizations identically, parser differential attacks fail. Security enforcement has shifted from guessing malicious patterns to verifying structural syntax at wire speed.