Skip to content

tasks_testing.md

Latest commit

 

History

History
100 lines (84 loc) · 8.44 KB

tasks_testing.md

File metadata and controls

100 lines (84 loc) · 8.44 KB
Raw

EADA Pro – Comprehensive Validation & Verification Plan

0. Testing Charter

  • Objective: Guarantee that every deliverable in tasks.md functions as intended, meets hackathon judging criteria, and upholds accessibility, privacy, and performance promises.
  • Scope: Covers perception, intelligence, adaptation, accessibility, security, analytics, deployment, business collateral, and demo readiness.
  • Test Philosophy: Mix of automated regression where feasible, manual exploratory passes for UX/accessibility, and scenario-driven validation tied to personas.

1. Governance & Tooling

  • Test Owners: Assign per squad (ML Engineer for model metrics, Full-Stack for API/UI, Tech Lead for integration, Product/Privacy for compliance).
  • Infrastructure: pytest, unittest, Playwright, locust, bandit, safety, axe-core, NVDA, Wireshark, mitmproxy, Postman collections, GitHub Actions runners.
  • Artifacts: Maintain tests/ directory, tests/README.md, tests/data/ for fixtures, tests/reports/ for exports, and Notion QA board for tracking.
  • Entry/Exit Criteria: No critical bugs open, ≥95% high-priority tests passing, known issues documented with mitigations.

2. Mission Snapshot Validation

  • Feature Parity Checklist: Confirm adaptive brightness/volume, ergonomic alerts, accessibility controls, analytics dashboard, secure identity layer all demo-ready.
  • Live Demo Drill: Rehearse full mission narrative end-to-end twice; capture video proof.
  • Collateral Review: Product/Privacy strategist signs off on business, ethical, and privacy claims; cross-reference with implementation evidence.

3. Schedule Readiness Tests

  • Timeboxing Simulation: Dry run key sprints in 2-hour windows to ensure tasks fit schedule.
  • Contingency Drills: Simulate 30-minute network outage, hardware swap, or team member absence and log recovery playbook.
  • Logistics Checklist: Validate power backups, offline installers, and container images before arrival.

4. Architecture & Module Testing

4.1 Perception Layer

  • Unit Tests: Validate MediaPipe pipeline outputs (landmark counts, coordinate ranges) with synthetic fixtures.
  • Integration Tests: Run live camera feeds in controlled lighting conditions (bright office, dim room, backlit) and confirm detection rates ≥95%.
  • Regression Artifacts: Save baseline landmark JSON snapshots for diffing after model tweaks.

4.2 Intelligence Layer

  • Distance Estimation: Compare predicted distance vs. ground truth markers at 0.5 m increments (0.25–4 m); tolerance ±10 cm.
  • Posture Analytics: Validate neck/shoulder angles using motion capture reference or recorded calibration poses.
  • Blink Detector: Run eye-blink dataset (BlinkingEyes2011 or custom) to ensure ≥97% accuracy; test false positives with head turns.
  • Eye-Strain Predictor: Cross-validation on synthetic dataset; target F1 score ≥0.85.
  • Recommendation Engine: Scenario-based tests verifying rule + ML arbitration, cooldown enforcement, explanation strings present.

4.3 Adaptation Layer

  • Brightness/Volume: Mock sbc and pycaw drivers to assert correct scalar values; run manual validation on Windows host with lux meter and decibel app.
  • VLC Control: Automation script verifying pause/resume triggers on face absence/presence within 3 s.
  • Notification UX: Playwright tests checking overlay rendering, iconography, keyboard focus order.

4.4 Accessibility & Personas

  • Profile Storage: Cryptographic tests ensuring AES-256-GCM encryption/decryption with tamper detection.
  • Voice Commands: Evaluate whisper pipeline accuracy across accents; target ≥90% intent recognition.
  • Gestures: Record gesture video set; verify classification latency <200 ms and precision ≥0.9.
  • Haptic Feedback: Simulated BLE device acknowledgments; ensure no dropped alerts.
  • Persona Scenarios: Scripted walkthrough for each persona (Accessibility Analyst, Hybrid Worker, Safety Officer, Visually Impaired) with success metrics.

5. Privacy & Security Validation

  • Blockchain Ledger: Hardhat tests for smart contract functions, ensuring permission checks and event emissions.
  • ACA-Py Credentials: End-to-end issuance, storage, presentation, and revocation flows using test DIDs.
  • DIDComm Channels: Validate encryption, replay protection, and key rotation (ECDH handshake, AES-256-GCM sessions).
  • Data Lifecycle: Memory profiling to confirm no frame buffers persist beyond processing window; run objgraph or tracemalloc for leak detection.
  • Threat Modeling: STRIDE exercise; document mitigations and residual risks.
  • Pen Tests: mitmproxy and Wireshark to detect plaintext leaks; run bandit, safety, pip-audit for dependencies; OWASP ZAP on REST endpoints.

6. API & Dashboard Testing

  • Contract Tests: Schemathesis against OpenAPI spec to fuzz edge cases.
  • Functional UI Tests: Playwright flows for live metrics, history filters, alerts acknowledgment, profile edits.
  • WebSocket Reliability: Chaos tests dropping packets, delaying frames to ensure UI falls back gracefully.
  • Accessibility Audits: axe-core + manual NVDA/VoiceOver walkthrough validating WCAG AA/AAA compliance.
  • Offline Mode: Simulate network loss; confirm IndexedDB caches historical data and resyncs on reconnect.
  • Export Validation: Assert CSV/PDF outputs contain accurate metrics, timestamps, and branding.

7. Performance & Scalability

  • Inference Benchmarks: Measure FPS on reference hardware (with/without GPU) using cProfile; ensure ≥25 FPS.
  • Audio Latency: Record end-to-end command recognition time (<150 ms) using timestamped logs.
  • API Load: Locust scenario with 50 concurrent connections hitting /metrics/live and /alerts; monitor CPU/memory.
  • Cross-Device Sync: Multi-device simulation sending simultaneous diffs; assert eventual consistency and conflict resolution.
  • Battery/Resource Monitoring: Windows Performance Recorder or Intel VTune to profile CPU/GPU/power impact.

8. Data & Model Governance Tests

  • Dataset Audits: Verify licensing metadata, anonymization, and diversity coverage per demographic.
  • Model Registry: Unit test for registry CRUD operations; ensure rollback capability.
  • Bias Audits: Evaluate false positive/negative rates across demographic slices; document fairness metrics.
  • Explainability Outputs: Automated SHAP report generation for key recommendations; confirm interpretability captions appear in UI.

9. Deployment & Packaging Verification

  • Setup Scripts: Run setup_env, start_services, run_tests on clean VM; ensure zero manual steps required.
  • Docker Images: Build Besu and ACA-Py containers; perform smoke tests on local network.
  • Installer Footprint: Measure download size; keep under hackathon bandwidth constraints.
  • Fallback Switches: Toggle blockchain off and confirm application degrades gracefully with mock services.
  • Disaster Recovery: Backup/restore config files, credentials, and analytics database snapshots.

10. Business & Presentation Validation

  • ROI Model Testing: Spreadsheet formulas with unit tests (e.g., using pytestsheets or manual cross-check) to ensure calculations consistent.
  • Pricing Scenarios: Validate tier pricing vs. cost assumptions; run sensitivity analysis.
  • Pitch Deck QA: Review slides for technical accuracy, ensure diagrams match architecture.
  • Demo Script Dry Runs: Time rehearsals, capture logs/screenshots for judges, prepare fallback video.
  • Judging Criteria Matrix: Map features/tests to hackathon scoring rubric; ensure coverage.

11. Reporting & Continuous Improvement

  • Test Case Matrix: Maintain spreadsheet linking each task in tasks.md to specific test IDs.
  • Issue Tracking: Use GitHub Projects or Notion board with severity labels, SLA for fixes.
  • Daily QA Stand-up: Quick syncs to review failures and assign owners.
  • Evidence Repository: Store screenshots, recordings, logs, and reports for submission.
  • Retrospective: Post-event session capturing lessons, automation gaps, and roadmap for long-term QA.

This validation plan complements tasks.md, ensuring every build activity has a matching verification path and documented acceptance criteria.