Add SentinelX project proposal and full SRS baseline docs

docs/PROJECT_PROPOSAL.md

@@ -0,0 +1,84 @@
+# SentinelX Project Proposal
+
+## 1. Project Title
+**SentinelX: Intelligent AI DevOps & Network Defense Platform**
+
+## 2. Executive Summary
+SentinelX is a cybersecurity platform that combines real-time network monitoring, AI-based intrusion detection, distributed processing, API services, dashboard visualization, and a domain-specific query language (DSL). The project will deliver an integrated, production-oriented system that captures packet traffic, detects threats, and presents actionable insights through APIs and a monitoring dashboard.
+
+## 3. Problem Statement
+Network infrastructures generate high-volume traffic, and manual or rule-only monitoring cannot reliably detect modern attacks in time. Organizations need a system that can:
+- monitor traffic continuously,
+- detect suspicious behavior with AI,
+- expose results through reliable services,
+- and support quick analytical queries for operators.
+
+## 4. Project Objectives
+1. Build a live packet-capture and feature-extraction pipeline.
+2. Integrate ML-based intrusion/anomaly detection.
+3. Provide robust backend APIs for health, prediction, and analytics.
+4. Develop a user-friendly monitoring dashboard.
+5. Implement a mini DSL for threat/traffic queries.
+6. Enable deployable architecture with container/cloud readiness.
+7. Maintain software quality through structured documentation and validation.
+
+## 5. Scope
+
+### In Scope
+- Real-time and batch-oriented network traffic analysis.
+- ML inference for attack classification.
+- API layer for prediction and reporting.
+- Persistent storage for traffic/prediction records.
+- Dashboard visualizations and operational monitoring.
+- DSL query execution over stored and live data.
+
+### Out of Scope
+- Full SIEM replacement and enterprise SOC orchestration.
+- Fully autonomous mitigation actions in production.
+- Unlimited scale guarantees without infrastructure tuning.
+
+## 6. Proposed Architecture (High-Level)
+1. Packet Sniffer Module (traffic collection)
+2. Feature Aggregation Layer (model-ready features)
+3. ML Detection + Validation Layer (classification + confidence gating)
+4. Database Layer (predictions, stats, audit-friendly records)
+5. FastAPI Service Layer (REST endpoints)
+6. Frontend Dashboard (Streamlit/React)
+7. DSL Query Layer (human-friendly security queries)
+8. Deployment Layer (Docker/AWS-oriented runtime)
+
+## 7. Stakeholders
+- Security Analysts / SOC Operators
+- DevOps and Platform Engineers
+- Network Administrators
+- Academic and research contributors
+
+## 8. Technology Stack
+- **Language:** Python
+- **Backend:** FastAPI
+- **Frontend:** Streamlit / React
+- **ML:** scikit-learn, XGBoost, NumPy, Pandas
+- **Network Capture:** Scapy
+- **Database:** SQLite/PostgreSQL via SQLAlchemy + Alembic
+- **Deployment:** Docker, cloud-ready workflows
+
+## 9. Deliverables
+1. Running backend API with documented endpoints.
+2. Monitoring dashboard with threat and traffic views.
+3. Integrated ML model artifacts for inference.
+4. DSL parser/executor for security queries.
+5. Database migration support and operational scripts.
+6. Complete proposal + SRS documentation set.
+
+## 10. Risks and Mitigation
+- **Permission limits for packet capture:** provide fallback behavior and setup guidance.
+- **Model performance variance:** calibrate thresholds and retrain periodically.
+- **High traffic load:** optimize processing and support distributed workflows.
+- **Deployment inconsistencies:** use containerized environments and migration controls.
+
+## 11. Success Criteria
+- End-to-end stack runs with documented setup.
+- Traffic ingestion and predictions operate reliably.
+- API and dashboard provide consistent operational visibility.
+- DSL queries return correct, timely results.
+- Documentation is complete enough for onboarding and maintenance.

docs/SRS.md

@@ -0,0 +1,208 @@
+# Software Requirements Specification (SRS)
+## Project: SentinelX
+
+## 1. Introduction
+
+### 1.1 Purpose
+This document defines the complete software requirements for SentinelX, an intelligent AI-driven network defense platform. It establishes functional and non-functional requirements, interfaces, constraints, and acceptance criteria to guide development, testing, deployment, and maintenance.
+
+### 1.2 Scope
+SentinelX provides real-time network traffic monitoring, ML-based intrusion detection, API access, dashboard visualization, and DSL-based querying. The system is intended for operational monitoring, threat detection, analytics, and educational/research use.
+
+### 1.3 Definitions and Acronyms
+- **SRS:** Software Requirements Specification
+- **ML:** Machine Learning
+- **API:** Application Programming Interface
+- **DSL:** Domain-Specific Language
+- **SOC:** Security Operations Center
+- **NFR:** Non-Functional Requirement
+
+### 1.4 References
+- Repository README
+- DSL Guide
+- Internal architecture and learning flow documentation under `docs/Learning`
+
+## 2. Overall Description
+
+### 2.1 Product Perspective
+SentinelX is a modular platform composed of:
+- packet capture and feature extraction,
+- ML inference and validation,
+- persistent data storage,
+- FastAPI backend,
+- Streamlit/React frontend,
+- and DSL query interface.
+
+### 2.2 Product Functions
+- Capture network packets from available interfaces.
+- Transform traffic into model-ready features.
+- Predict threat class/confidence using trained ML models.
+- Expose prediction and statistics through REST APIs.
+- Display live and historical insights in dashboard views.
+- Execute DSL commands for threat/traffic/stats retrieval.
+
+### 2.3 User Classes
+1. **Security Analyst:** monitors threats and reviews predictions.
+2. **Administrator/DevOps Engineer:** deploys and maintains services.
+3. **Research/Student User:** evaluates model behavior and workflows.
+
+### 2.4 Operating Environment
+- Python 3.8+
+- Backend via FastAPI/uvicorn
+- Frontend via Streamlit (and/or React)
+- SQLite or PostgreSQL database
+- Linux/macOS/Windows (with interface/permission differences for sniffing)
+
+### 2.5 Constraints
+- Live packet capture may require elevated privileges.
+- Model quality depends on training data coverage and freshness.
+- Performance depends on host resources and traffic volume.
+- Some environments may restrict raw socket/network interface access.
+
+### 2.6 Assumptions and Dependencies
+- Required Python packages are installed.
+- Model artifacts exist and are accessible.
+- Database is initialized/migrated.
+- Network interface is accessible for live monitoring mode.
+
+## 3. External Interface Requirements
+
+### 3.1 User Interfaces
+- Web dashboard shall show system health, recent predictions, and traffic statistics.
+- Dashboard shall provide controls for viewing trends and querying DSL commands.
+
+### 3.2 Hardware Interfaces
+- Access to network interface card(s) for packet sniffing when enabled.
+
+### 3.3 Software Interfaces
+- Database interface via SQLAlchemy-compatible backend.
+- API endpoints over HTTP/JSON.
+- Model files loaded from local filesystem.
+
+### 3.4 Communication Interfaces
+- REST communication over HTTP.
+- Optional cloud/container network interfaces for deployment.
+
+## 4. Functional Requirements
+
+### 4.1 System Initialization
+- **FR-1:** System shall expose a health endpoint to report service availability.
+- **FR-2:** System shall initialize data/storage components on startup.
+- **FR-3:** System shall start even if live sniffing is unavailable, with degraded capture mode.
+
+### 4.2 Packet Capture and Feature Processing
+- **FR-4:** System shall capture network traffic packets from configured interface(s).
+- **FR-5:** System shall aggregate packet data into structured features for inference.
+- **FR-6:** System shall validate extracted features before model prediction.
+
+### 4.3 ML Inference and Validation
+- **FR-7:** System shall load trained model and preprocessing artifacts.
+- **FR-8:** System shall generate predictions for incoming feature windows.
+- **FR-9:** System shall compute prediction confidence/probability outputs.
+- **FR-10:** System shall support post-processing thresholds for alert quality control.
+
+### 4.4 API Services
+- **FR-11:** System shall provide GET and POST prediction endpoints.
+- **FR-12:** System shall provide traffic statistics and recent prediction endpoints.
+- **FR-13:** API responses shall be JSON formatted with relevant fields and status codes.
+
+### 4.5 Data Persistence
+- **FR-14:** System shall store prediction records with timestamp and metadata.
+- **FR-15:** System shall support schema migration through Alembic.
+- **FR-16:** System shall allow retrieval of historical prediction and statistics data.
+
+### 4.6 Dashboard and Visualization
+- **FR-17:** Dashboard shall display backend connectivity/health status.
+- **FR-18:** Dashboard shall show recent threats and traffic summaries.
+- **FR-19:** Dashboard shall visualize trends (e.g., attacks over time, class distribution).
+
+### 4.7 DSL Query Layer
+- **FR-20:** System shall accept supported DSL commands for threat/traffic/stat queries.
+- **FR-21:** DSL engine shall prefer DB-backed retrieval and fallback to API when needed.
+- **FR-22:** System shall return human-readable results and error messages for invalid queries.
+
+### 4.8 Security and Access
+- **FR-23:** System shall validate and sanitize API/DSL inputs.
+- **FR-24:** System shall avoid exposing sensitive runtime details in user-facing errors.
+- **FR-25:** System shall support secure deployment configuration through environment variables.
+
+### 4.9 Logging and Monitoring
+- **FR-26:** System shall log startup, prediction, and failure events.
+- **FR-27:** System shall provide enough diagnostics to troubleshoot capture/model/API issues.
+
+## 5. Non-Functional Requirements
+
+### 5.1 Performance
+- **NFR-1:** API health responses should be near real-time under normal load.
+- **NFR-2:** Prediction endpoints should respond within acceptable operational latency for monitoring workflows.
+- **NFR-3:** System should process packet windows continuously without frequent service interruption.
+
+### 5.2 Reliability and Availability
+- **NFR-4:** System should remain operational when packet capture is unavailable, with partial functionality.
+- **NFR-5:** Service should recover gracefully from transient data/model loading errors where possible.
+
+### 5.3 Scalability
+- **NFR-6:** Architecture should support horizontal or distributed processing enhancements.
+- **NFR-7:** Database and API layers should support growth in stored prediction volume.
+
+### 5.4 Security
+- **NFR-8:** Inputs shall be validated to reduce injection and malformed-request risks.
+- **NFR-9:** Secrets/configuration values shall be managed via environment/config files, not hardcoded.
+
+### 5.5 Maintainability
+- **NFR-10:** Codebase shall remain modular across capture, ML, API, dashboard, and DSL layers.
+- **NFR-11:** Database schema evolution shall be managed through versioned migrations.
+- **NFR-12:** Documentation shall be kept synchronized with endpoint and workflow changes.
+
+### 5.6 Portability
+- **NFR-13:** Core services shall run in standard Python environments and containerized setups.
+
+## 6. Data Requirements
+- Prediction entity shall include at minimum: timestamp, predicted class, confidence/probability, and relevant feature/traffic metadata.
+- Traffic statistics entity shall include aggregated packet/flow metrics over defined intervals.
+- Historical retention strategy shall support dashboard and DSL query use cases.
+
+## 7. Use Cases
+
+### UC-1: View System Health
+- **Actor:** Analyst/Admin
+- **Precondition:** Backend running
+- **Flow:** User requests health status via API or dashboard.
+- **Postcondition:** Current service state is displayed.
+
+### UC-2: Analyze Traffic and Receive Prediction
+- **Actor:** Analyst
+- **Precondition:** Feature pipeline and model loaded
+- **Flow:** Traffic is captured -> features built -> model predicts -> result stored/displayed.
+- **Postcondition:** Threat classification and confidence become available.
+
+### UC-3: Query Recent Threats Using DSL
+- **Actor:** Analyst/Research User
+- **Precondition:** DSL component available
+- **Flow:** User submits DSL query -> parser validates -> data fetched -> formatted result returned.
+- **Postcondition:** User receives query output for decision-making.
+
+## 8. Acceptance Criteria
+1. API endpoints (`/health`, `/predict`, `/traffic_stats`, `/recent_predictions`) are operational and documented.
+2. Dashboard displays backend status and recent analytical views.
+3. ML inference executes with loaded artifacts and returns predictions.
+4. Database records are created and retrievable.
+5. DSL commands for key query categories execute successfully.
+6. System supports startup in restricted sniffing environments without total failure.
+
+## 9. Verification and Validation Strategy
+- Unit-level validation for feature and inference logic where available.
+- Integration validation across capture -> inference -> storage -> API -> dashboard.
+- Data validation checks for malformed/edge-case feature windows.
+- Manual/automated smoke checks for deployment and runtime health.
+
+## 10. Future Enhancements (Non-Blocking)
+- Advanced alert correlation and incident scoring.
+- Role-based access controls and authentication hardening.
+- Extended distributed analytics integration.
+- Automated model retraining pipelines and drift monitoring.
+
+---
+**Document Version:** 1.0  
+**Prepared For:** SentinelX Project  
+**Status:** Baseline Complete SRS