🕵️ NetGuard – Full-Stack NIDS & Security Observability Engine

A full end-to-end real-time Network Intrusion Detection System (NIDS).
Combines a Multi-threaded Python (Scapy) capture and analysis engine with DPI (Aho-Corasick), Sliding-Window anomaly detection, Active Defense mechanisms, and a fully code-managed monitoring stack (Dashboard as Code) on Docker (Grafana + Loki + Promtail).

---

🔎 Overview & Architecture

NetGuard provides a complete solution for monitoring, analyzing, and responding to network security events across OSI layers 3, 4, and 7.
The architecture is built on a continuous Producer-Consumer pipeline that separates low-level packet capture, real-time threat analysis, and structured observability shipping:

graph TD
    %% Traffic Input
    NIC[📡 Network Interface] -->|Raw Packets| SnifferThread[🐍 Sniffer Thread - Scapy store=0]
    
    %% Core Engine
    subgraph Engine [Python NIDS Core Engine]
        SnifferThread -->|Non-blocking Put| Queue[📦 Bounded Queue maxsize=10000]
        Queue -->|Get Packet| Worker[⚙️ Worker Threads Pool]
        
        subgraph Detection [Detection & Active Defense]
            Worker -->|L3/L4 Sliding Window| Anomaly[🛡️ DoS / Port Scan Detector]
            Worker -->|L7 Raw Payload| DPI[🔍 DPI Engine Aho-Corasick]
            Anomaly & DPI -->|Check/Update| State[🔒 Lock-Guarded State & Blacklist]
        end
        
        GC[🧹 Background GC Thread] -->|Synchronous Clean Every 30s| State
    end

    %% Logging & Observability
    Worker -->|Write JSON Log| LogFile[📄 logs/network_security.json]
    Promtail[🔄 Promtail Container] -->|Tail & Ship| LogFile
    Promtail -->|HTTP/Push| Loki[🗄️ Loki DB Container]
    Loki -->|PromQL/LogQL| Grafana[📊 Grafana Dashboard as Code]

Loading

---

⚡ Performance & Resilience Analysis

• Memory Backpressure & Drop Policy — The engine utilizes a bounded Queue(maxsize=10000) combined with store=0 in Scapy to ensure zero in-memory packet buffering by the sniffer thread. Under high-throughput conditions, excess packets are dropped safely rather than causing Out-Of-Memory (OOM) fatal crashes.
• Concurrency & C-Level Unlocking — Low-level packet capture executes within native socket primitives (C-level libpcap/WinPcap), releasing Python's Global Interpreter Lock (GIL) and allowing the background worker thread and garbage collector thread to execute processing tasks concurrently.
• Thread Safety & Granular Locking — Multi-threaded access to volatile state structures (syn_history, port_history, blacklist) is protected using explicit threading.Lock primitives to guarantee atomic read/write state transitions without data races.
• Deterministic Resource Cleanup (Garbage Collector) — Dormant IP records and expired blacklist entries are purged every 30 seconds by a background garbage collection thread in bounded $O(N)$ time, ensuring steady memory utilization under sustained traffic.

---

📂 Project Structure

python_sniffer/
├── grafana/
│   └── dashboards/                 # Standard JSON Dashboards (Git Version-Controlled)
│       ├── dashboard-Live Security Log Stream.json
│       ├── dashboard-Security Events Distribution.json
│       ├── dashboard-Threat Timeline & Severity Levels.json
│       ├── dashboard-Top Suspicious Source IPs.json
│       └── dashboard-Total Security Alerts.json
├── provisioning/                   # Grafana Automated Provisioning Configs
│   ├── dashboards/
│   │   └── dashboards.yml
│   └── datasources/
│       └── datasources.yml
├── logs/                           # Runtime Log Directory (Ignored by Git)
├── .env.example
├── .gitignore
├── docker-compose.yml
├── main.py                         # NIDS Core Engine (Thread-Safe & GC Refactored)
├── promtail-config.yml
├── requirements.txt
└── test_attack.py                  # Traffic Simulator

---

🚀 Core Features

Domain	Feature	Status	Description	Performance Indicator
📡 Network	Real-time L2-L7 Sniffing	✅	Real-time capture and analysis of IP, TCP, UDP, and DNS traffic while preventing memory overflow (store=0).	Zero-copy capture, O(1) enqueue
🛡️ Cyber Security	Sliding-Window & Stealth Detection	✅	Detects DoS (SYN Flood), standard port scans, and Stealth Scans (NULL, FIN, XMAS) via moving time windows.	O(1) queue operations
🧬 DNS Security	DNS Tunneling Detection	✅	Shannon Entropy calculation & query length evaluation to catch exfiltration over DNS.	O(N) entropy check
⚡ Active Defense	Dynamic IP Isolation	✅	Active mitigation mechanism that isolates attacking addresses for a limited time (Blacklist with automatic expiry).	O(1) blacklist check
🔍 DPI Engine	Deep Packet Inspection	✅	Byte-level Raw Payload scanning leveraging an Aho-Corasick Automaton to search for multiple credential/injection patterns in parallel, detecting credentials & command injection.	O(N+M) string matching
⚙️ Architecture	Producer-Consumer & Thread-Safety	✅	Bounded Queue, threading.Lock locks, and a dedicated background Garbage Collector thread to prevent memory leaks.	Bounded queue, backpressure-safe
📊 Observability & IaC	Dashboard as Code (Grafana + Loki)	✅	Five pre-defined dashboards in standard JSON format, automatically loaded on container startup via Provisioning files.	Instant provisioning on boot
📝 Logging	Structured JSON Dual-Stream	✅	Colorized console output alongside structured JSON log writes (logs/network_security.json), tailored for collection by Promtail.	Low-overhead async writes
🧪 Testing	Traffic Attack Simulator	✅	Simulation script (test_attack.py) that generates synthetic attack traffic to validate detection mechanisms.	Configurable synthetic load

---

🛠️ Technologies & Architectural Highlights

• Python & Scapy — Raw-socket-level packet capture, protocol parsing, and deep payload-level inspection (DPI).
• Producer-Consumer Architecture — Full separation between packet capture and analysis via queue.Queue(maxsize=10000), preventing packet loss under load.
• Thread-Safety & Active Defense — Whitelist/Blacklist state management and anomaly detection guarded by threading.Lock to prevent data races, alongside dynamic, time-limited blocking of attacking IP addresses.
• Background Garbage Collector — A dedicated background thread that cleans up stale data structures (Sliding Window History & Blacklist) from memory every 30 seconds, synchronously and thread-safely, ensuring zero memory leaks from dormant IP addresses.
• Promtail & Grafana Loki — Shipping of structured JSON logs from the local logs directory and indexing them in Loki.
• Dashboards as Code (IaC) — Full version control of 5 dashboards in Git under grafana/dashboards/, automatically loaded into Grafana on container startup.
• Docker Compose Stack — One-click deployment of the entire observability infrastructure.

---

📋 Prerequisites

• Docker & Docker Compose — For running Loki, Promtail, and Grafana.
• Python 3.10+ — Required for running the NIDS engine and test suite.
• Administrator / Root Privileges — Required to capture raw socket traffic via Scapy (or use the least-privilege setcap option below on Linux).
• Npcap (Windows only) — Required for Scapy to capture raw packets on Windows network adapters.

---

📝 JSON Log Structure (Structured Logging)

{
  "timestamp": "2026-08-06T10:30:15.123456",
  "level": "WARNING",
  "message": "[PORT SCAN DETECTED] Host 10.0.0.4 scanned 18 unique ports",
  "logger": "NetworkGuardian",
  "src_ip": "10.0.0.4",
  "event_type": "PORT_SCAN",
  "details": "18 ports scanned"
}

---

⚙️ Installation & Quick Start

# 1. Clone the repository
git clone https://github.com/RazEini/python_sniffer.git
cd python_sniffer

# 2. Environment Setup
cp .env.example .env  # Set your Grafana password in .env

# 3. Start Observability Stack (Grafana, Loki, Promtail)
# Grafana will automatically provision all dashboards from grafana/dashboards/
docker compose up -d

# 4. Setup Python Environment
python -m venv .venv
.\.venv\Scripts\activate     # On Windows
source .venv/bin/activate    # On Linux/Mac
pip install -r requirements.txt

# 5. Run NIDS Engine
# On Linux (Principle of Least Privilege - grant raw socket capability without full sudo):
sudo setcap cap_net_raw,cap_net_admin=eip $(readlink -f .venv/bin/python)
.venv/bin/python main.py

# Or run directly with root (Linux / Mac):
sudo .venv/bin/python main.py

# On Windows (Run PowerShell / CMD as Administrator):
python main.py

# 6. Run Attack Simulator (in a separate terminal)
# Automatically targets local active IP:
python test_attack.py

# Or target a specific IP address explicitly:
python test_attack.py <TARGET_IP>

📊 Accessing Grafana: Open your browser to http://localhost:3000 (username: admin, password set in .env). All dashboards will already be loaded and ready to use!

---

📄 License

This project is distributed under the MIT license – free to use and modify for educational and research purposes.

---

👨‍💻 Raz Eini (2026)