ARPA47: Passive Measurement of QUIC, ECH, and Post-Quantum Cryptography at the ISP Edge

Going Dark at the Edge: A 30-Day Empirical Measurement of QUIC, Encrypted Client Hello, and Post-Quantum Cryptography Adoption at a South Asian Residential ISP.

This repository contains the measurement infrastructure, analytical scripts, and processing pipelines for an empirical network measurement study examining the co-adoption of TLS 1.3, Encrypted Client Hello (ECH), HTTP/3 (QUIC), and Post-Quantum Cryptography (PQC) at a live residential ISP edge in Chattogram, Bangladesh.

---

Published Dataset

The fully anonymized Zeek telemetry dataset generated by this project is publicly available on Zenodo:

Field	Value
Title	ARPA47: 30-Day Zeek Telemetry from a South Asian Residential ISP
DOI	10.5281/zenodo.22974122
URL	https://doi.org/10.5281/zenodo.22974122
License	Creative Commons Attribution 4.0 International (CC BY 4.0)
Version	1.0.0
Published	September 26, 2026
Collection Window	August 26, 2026 – September 26, 2026 (30 days)
Size	1.66 GB (12 files)
Format	Apache Parquet (partitioned by date) + CSV aggregates

Dataset Contents

File	Size	Description
conn.zip	959.60 MB	Zeek conn.log: TCP/UDP/ICMP flow metadata, byte counts, durations
dns.zip	369.29 MB	Zeek dns.log: DNS queries, HTTPS Resource Records (ECH key delivery)
quic.zip	227.71 MB	Zeek quic.log: QUIC/HTTP3 session metadata, SNI obfuscation
ssl.zip	90.34 MB	Zeek ssl.log: TLS handshakes, PQC curves, ECH indicators
http.zip	10.94 MB	Zeek http.log: Legacy cleartext HTTP transactions
statistics.zip	3.30 KB	Daily aggregate CSV summaries (4 files)
README.md	4.94 KB	Full dataset documentation and Python usage guide
DATASET_SCHEMA.md	4.37 KB	Field-level schema dictionary for all log types
ETHICAL_COMPLIANCE_AND_DATA_SANITIZATION.pdf	112.74 KB	Signed Ethics Certificate (Menlo Report aligned)
LICENSE	1.58 KB	CC BY 4.0 license
requirements.txt	62 bytes	Python dependencies for reproducing the analysis
zenodo_metadata.json	1.16 KB	Machine-readable dataset metadata

Key Findings

Metric	Value
Total connections	31,338,265
TLS handshakes parsed	2,712,288
QUIC sessions	4,236,252
QUIC share of port-443 traffic	61.0%
TLS 1.3 with Post-Quantum (X25519MLKEM768)	23.08%
TLS sessions with hidden/empty SNI	14.16%
Confirmed genuine ECH flows	0.10% (2,822 flows)
QUIC sessions with hidden SNI	77.4%
DNS HTTPS Resource Record queries	3.39% (258,899 queries)

---

Repository Structure

ARPA47/
├── scripts/                    # All Python data processing and analysis scripts
│   ├── build_dataset.py        # Main pipeline: Zeek TSV → anonymized Parquet
│   ├── patch_dataset.py        # Post-processing fixes (type casting, schema alignment)
│   ├── generate_detailed_stats.py  # Computes daily CSV aggregate summaries
│   ├── verify_integrity.py     # Validates dataset metrics against paper claims
│   ├── validate_and_summarize.py   # End-to-end dataset validation report
│   └── fix_citations.py        # Manuscript citation formatting utility
├── Sample_scripts/             # Standalone Zeek/NetFlow utility scripts
│   └── test_data.py            # Quick sanity-check loader for Parquet files
├── Mikrotik Scripts/           # RouterOS scripts (gitignored)
│   └── ...                     # FastTrack bypass, TZSP mangle rules, CPU watchdog
├── Info/                       # Engineering notes and architecture documentation
│   └── project.md              # Original research blueprint and project plan
├── Data/                       # Local data scratch space (not committed)
├── Paper 1/                    # Main research paper drafts (gitignored)
├── Paper 2/                    # Secondary paper drafts (gitignored)
├── .env                        # Environment variables (gitignored)
├── .gitignore
└── README.md

---

Collection Architecture

The data pipeline operates on two distinct parallel layers:

Layer 1: Macroscopic (IPFIX / NetFlow)

• Scope: All 4,000 active residential subscribers
• Tool: MikroTik IPFIX export → nfcapd (nfdump toolchain)
• Data: Flow-level byte/packet counts, port distribution, session durations
• Purpose: Proves the volumetric shift from TCP/443 to UDP/443 (QUIC) across the full ISP

Layer 2: Microscopic (TZSP + Zeek)

• Scope: A statically assigned /24 PPPoE pool (225 active subscribers)
• Tool: MikroTik TZSP mangle → Linux virtual TAP → Zeek v8.0.10 LTS
• Data: TLS handshake metadata, QUIC SNI fields, DNS HTTPS RR queries, PQC key exchange
• Purpose: Deep protocol-level analysis of ECH, PQC, and QUIC header obfuscation

Engineering Safeguards

Two MikroTik-specific issues required custom solutions:

1. FastTrack Bypass: RouterOS FastTrack optimization bypasses mangle rules for established connections, causing Zeek to see only one-sided traffic. Fixed by injecting two accept rules at positions 0 and 2 on the forward chain, above the FastTrack rules at positions 12–14.

2. CPU Overload Watchdog: TZSP encapsulation adds CPU load. A Python daemon polls the router CPU via the RouterOS API every 10 seconds. If load exceeds 85% for 10+ consecutive seconds, it automatically removes the TZSP mirror rules to protect subscriber QoS.

---

Privacy & Ethical Compliance

All data collection adheres to the Menlo Report ethical principles for ICT research.

• No payload capture: Packet payloads were truncated at the interface level. Only connection metadata and cryptographic handshake headers were processed.
• Cryptographic IP anonymization: All subscriber IPs in the 10.60.1.0/24 pool were anonymized using a salted, prefix-preserving CryptoPan HMAC-SHA256 algorithm. The cryptographic salt was destroyed after the collection window.
• No MAC/PPPoE identifiers: Layer-2 hardware identifiers (orig_l2_addr, resp_l2_addr, mac) were actively filtered and discarded.
• Credential and resource stripping: Cleartext credentials and specific path endpoints in HTTP and Weird logs (username, password, uri, referrer, addl) were fully dropped from the dataset to eliminate residual PII risks.
• Domain truncation: All domain string fields (server_name, host, query) were truncated to their effective Top-Level Domain plus one (eTLD+1) using Mozilla's Public Suffix List (e.g. user-laptop.example.com becomes *.example.com).
• Formal Ethics Certificate: A signed Ethics Certificate is included in the Zenodo dataset: ETHICAL_COMPLIANCE_AND_DATA_SANITIZATION.pdf.

---

Reproducing the Analysis

Prerequisites

pip install -r requirements.txt
# pandas>=2.2, pyarrow>=16.0, duckdb>=1.1, numpy>=1.26

Quick Start

1. Download and extract the dataset from Zenodo:

   https://doi.org/10.5281/zenodo.22974122
   

2. Extract the ZIP archives into a local directory:

   unzip conn.zip && unzip ssl.zip && unzip quic.zip && unzip dns.zip

3. Load QUIC sessions with Pandas:

   import pyarrow.dataset as ds
   
   dataset = ds.dataset("quic/", format="parquet", partitioning="hive")
   df = dataset.to_table().to_pandas()
   hidden_sni = df[df['server_name'] == '-']
   print(f"QUIC sessions with hidden SNI: {len(hidden_sni):,}")

4. Analyze PQC adoption with DuckDB (no RAM required):

   import duckdb
   
   result = duckdb.query("""
       SELECT date, COUNT(*) AS pqc_sessions
       FROM read_parquet('ssl/*/*.parquet')
       WHERE curve = 'X25519MLKEM768'
       GROUP BY date ORDER BY date
   """).df()
   print(result)

5. Verify dataset integrity against paper claims:

   python scripts/verify_integrity.py

---

Citation

If you use this dataset or codebase in your research, please cite:

@dataset{uddin_arpa47_2026,
  author    = {Uddin, Jalal},
  title     = {{ARPA47}: 30-Day Zeek Telemetry from a South Asian Residential ISP},
  year      = {2026},
  publisher = {Zenodo},
  version   = {1.0.0},
  doi       = {10.5281/zenodo.22974122},
  url       = {https://doi.org/10.5281/zenodo.22974122}
}

---

License

The dataset is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. The code in this repository is available for research and academic use.