⚡ py-taskpool

An asynchronous, dual-engine task execution framework for Python 3 featuring automatic retry backoff, fault isolation, and CI/CD testing.

---

📌 Overview

py-taskpool abstracts Python's standard concurrency primitives into a reliable, high-level task pool. It allows developers to dispatch mixed workloads—combining I/O-intensive network operations with CPU-bound data crunching—while guaranteeing structured result collection and automated failure recovery.

✨ Key Features

• 🔄 Dual Engine Architecture: Seamlessly toggles between ThreadPoolExecutor for network/disk I/O and ProcessPoolExecutor to bypass the GIL for multi-core computation.
• 🛡️ Automated Retry Logic: Built-in exponential/linear backoff wrapper (submit_with_retry) to handle flaky tasks automatically.
• 🔒 Isolated Fault Handling: Aggregates output state into structured result payloads without throwing unhandled exceptions across worker threads.
• 🧪 Production Ready: Fully covered by pytest unit tests and integrated into a GitHub Actions CI pipeline.
• 📦 Zero External Dependencies: Runs entirely on Python standard library modules (concurrent.futures, time).

---

🏗️ Architecture & Engine Comparison

Feature	mode="thread"	mode="process"
Underlying Class	ThreadPoolExecutor	ProcessPoolExecutor
Best Used For	Network calls, file I/O, database queries	Math operations, data processing, image encoding
GIL Behavior	Shared GIL across threads	Separate GIL per process (True Parallelism)
Memory Overhead	Low (shared memory space)	Medium (isolated memory spaces)

---

🚀 Quick Start

Installation

Clone the repository and run locally:

git clone [https://github.com/sharfuddin18/py-taskpool.git](https://github.com/sharfuddin18/py-taskpool.git)
cd py-taskpool