🚀 A language-agnostic API & AI/ML E2E testing framework
·2 min read·
MLOpsE2ETestingPlaywrightAutomationAPIDevOpsAIEngineeringCICDSoftwareEngineeringTechLeadershipTestingQualityEngineering
Over the past few weeks, I've built a language-agnostic API & AI/ML E2E testing framework that plugs seamlessly into any Python, .NET, or AI/ML project.
The idea is simple but powerful:
- 👉 Drop a JSON use-case
- 👉 Run multi-step tests automatically
- 👉 Get full observability + SLA enforcement
Just define your use cases and run. Works across APIs, ML workflows, backend — and extensible to frontend.
💭 One thing I learned while building this: Success isn't just about passing tests — it's about making testing accessible. The goal was to enable QA, PMs, and Data Scientists to validate complex AI systems and ML pipelines without needing deep coding expertise.
✨ Why This Matters — Problems It Solves
- 🌐 Fragmented E2E testing → Different frameworks per project, duplicated effort
- ⚡ Slow validation → Manual testing delays releases and ML iterations
- 🔍 Lack of observability → Hard to debug across APIs + ML workflows
- 🔄 Hard to scale → Every new project = new test setup
🛠 Stack & Technologies
- 💻 Playwright (
@playwright/test) - ⚡ TypeScript / Node.js
- 🧩 JSON-driven DSL (
e2e-core) - 🔑 Auth: Bearer · API Key · Basic · mTLS
- 📐 Validation:
ajv(JSON Schema) ·zod - 📊 Observability: Structured logs · Traces · SLA enforcement
- 🔗 CI/CD: GitHub Actions + environment-based config
- 🤖 MLOps: Supports ML pipelines, batch jobs, model endpoints
💎 Why Not Just pytest?
| pytest | This framework | |
|---|---|---|
| Test authoring | Python code | JSON use-case files |
| Execution | Sequential by default | Parallel, isolated |
| Stack coverage | Python | Python, .NET, ML, APIs |
| Observability | Plugin-dependent | Built-in logs + traces + SLA |
| CI/CD | Manual setup | Plug-and-play |
- ⚡ JSON-driven → No need to write test code for flows
- 🚀 Parallel execution → Fast, isolated runs
- 🌐 Cross-stack → One framework for Python, .NET, ML
- 🔍 Built-in observability → Logs, traces, SLA
- 🔄 CI/CD ready → Plug-and-play into pipelines
💡 What This Unlocks
- ⏱ Faster validation of APIs + ML workflows
- 🛡 Reduced manual testing effort
- 🔄 One consistent framework across teams and projects
This isn't just testing. It's a plug-and-play reliability engine for modern API and AI/ML systems.