HazCom Plan Builder — Testing Strategy

Overview

End-to-end testing strategy covering the full software development lifecycle: what to do during development, what to run in the test environment, and what gates must pass before production. Includes a Jenkins pipeline design to automate everything.

This codebase is vibe-coded — built rapidly with AI. The priority is catching broken functionality, not chasing code coverage numbers.

Tech Stack

Service	Stack	Test Framework
`tellus-ehs-hazcom-service`	FastAPI + SQLAlchemy + PostgreSQL	pytest 7.4.3
`tellus-ehs-hazcom-ui`	React + TypeScript + Vite	Vitest 4.0.12 + Playwright
`tellus-ehs-background-service`	Python + SQS + Claude AI	pytest 7.4.0+

Environments

Environment	URL	Database	Purpose
Local	`localhost:5174` / `localhost:8000`	Dev DB (DigitalOcean)	Developer machine
Test / Staging	`test.tellus-ehs.com` (example)	Separate test DB	QA, integration testing, demo
Production	`app.tellus-ehs.com` (example)	Production DB	Live users

Phase 1: During Development (Developer's Machine)

This is a vibe-coded codebase — every feature touches multiple layers (API, DB, UI, SQS) and any change can silently break something elsewhere. Run the full Playwright regression suite after every feature, not just before push.

1.1 Write Tests Alongside Features

When you build a new feature or fix a bug, write the corresponding test at the same time. Don't defer it.

What You Changed	Test to Write	Tool
New API endpoint	pytest integration test (call endpoint, assert response + DB state)	pytest
Changed API response shape	Update existing pytest test + frontend API test	pytest + Vitest
New UI component	Playwright spec if it's a user flow; Vitest if it's isolated rendering	Playwright / Vitest
Modified approval workflow	Playwright flow test (submit → approve → publish)	Playwright
Changed questionnaire logic	Playwright questionnaire spec + pytest completion % test	Both
Bug fix	Regression test that reproduces the bug, then verify it passes	Whichever layer
Database migration	Test migration up + down on local DB	Alembic

1.2 Development Workflow: Quick Checks While Coding

Run these as you're actively working, to catch obvious issues fast:

# Lint + type check (catch syntax errors immediately)
cd tellus-ehs-hazcom-service && black --check app/ && mypy app/
cd tellus-ehs-hazcom-ui && npm run lint

# Backend unit/smoke tests (< 30 seconds)
cd tellus-ehs-hazcom-service && pytest tests/flows/test_smoke.py -x -v

# Frontend unit tests (< 30 seconds)
cd tellus-ehs-hazcom-ui && npm run test:run

1.3 After Every Feature: Full Playwright Regression

This is the most important step. After completing a feature (before moving to the next task), run the entire Playwright suite to catch regressions:

cd tellus-ehs-hazcom-ui

# Run FULL Playwright regression suite (~5-8 minutes)
npx playwright test

# View the HTML report to inspect any failures
npx playwright show-report

Why the full suite, not just smoke? In a vibe-coded app, changes are interconnected. Fixing approval logic can break versioning. Changing the questionnaire can break split view. The full suite catches cross-feature regressions that smoke tests miss.

When to run it:

Trigger	What to Run
Finished implementing a feature	`npx playwright test` (full suite)
Fixed a bug	`npx playwright test` (full suite)
Changed database schema	`npx playwright test` (full suite) + migration up/down test
Quick syntax fix / typo	Lint + smoke only (use judgement)

Use --headed mode for debugging failures:

# See the browser while tests run (helpful for debugging)
npx playwright test --headed

# Step through a specific failing test
npx playwright test --debug e2e/flows/basic-plan-lifecycle.spec.ts

# Run just one spec to investigate a failure
npx playwright test e2e/flows/questionnaire.spec.ts

1.4 Before Every Push: Gate Check

Everything must pass before pushing code. This is the final gate — not the first time you run tests.

# 1. Lint + type check
cd tellus-ehs-hazcom-service && black --check app/ && mypy app/
cd tellus-ehs-hazcom-ui && npm run lint

# 2. Backend tests
cd tellus-ehs-hazcom-service && pytest tests/ -x -v

# 3. Frontend unit tests
cd tellus-ehs-hazcom-ui && npm run test:run

# 4. Full Playwright regression (should already be green from 1.3)
cd tellus-ehs-hazcom-ui && npx playwright test

Rule: If any of these fail, do NOT push. Fix it first.

Tip: If you already ran the full Playwright suite in step 1.3 and haven't changed code since, you can skip re-running it here. The point is that the full suite has been verified at least once after your changes.

1.5 Developer Testing Checklist

Before marking a task as done:

Feature works manually in the browser (quick visual check)
Lint passes (black --check, npm run lint)
Full Playwright regression suite passes (npx playwright test)
Backend tests pass (pytest tests/ -v)
If you changed an API: response shape matches the Pydantic schema
If you changed the DB: migration runs cleanly up and down
No console.error in browser dev tools during manual test
New test written for the feature/fix you just completed

Phase 2: Jenkins Pipeline (On Every Push / PR)

Automated CI that runs on every push to a feature branch and every PR to main.

2.1 Pipeline Design

┌─────────────────────────────────────────────────────────────┐
│                    JENKINS PIPELINE                         │
│                                                             │
│  ┌──────────┐   ┌──────────────┐   ┌──────────────────┐   │
│  │  Stage 1  │──▶│   Stage 2    │──▶│     Stage 3      │   │
│  │  Lint &   │   │  Unit Tests  │   │  Integration &   │   │
│  │  Build    │   │  (parallel)  │   │  Playwright E2E  │   │
│  └──────────┘   └──────────────┘   └──────────────────┘   │
│       │               │                    │               │
│   All pass?       All pass?           All pass?            │
│       │               │                    │               │
│       ▼               ▼                    ▼               │
│  ┌──────────────────────────────────────────────────────┐  │
│  │                  Stage 4: Deploy to Test              │  │
│  │         (only on merge to main / release branch)      │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

2.2 Jenkinsfile

pipeline {
    agent any

    environment {
        TEST_DB_URL = credentials('test-database-url')
        TEST_USER_EMAIL = credentials('test-user-email')
        TEST_USER_PASSWORD = credentials('test-user-password')
        TEST_SITE_ID = credentials('test-site-id')
    }

    stages {

        // ─── STAGE 1: LINT & BUILD ───────────────────────────
        stage('Lint & Build') {
            parallel {
                stage('Backend Lint') {
                    steps {
                        dir('tellus-ehs-hazcom-service') {
                            sh 'pip install -r requirements.txt'
                            sh 'black --check app/'
                            sh 'isort --check-only app/'
                            sh 'flake8 app/ --max-line-length=120'
                        }
                    }
                }
                stage('Frontend Lint & Build') {
                    steps {
                        dir('tellus-ehs-hazcom-ui') {
                            sh 'npm ci'
                            sh 'npm run lint'
                            sh 'npm run build'  // Catches TypeScript errors
                        }
                    }
                }
                stage('Background Service Lint') {
                    steps {
                        dir('tellus-ehs-background-service') {
                            sh 'pip install -r requirements.txt'
                            sh 'black --check app/'
                            sh 'flake8 app/ --max-line-length=120'
                        }
                    }
                }
            }
        }

        // ─── STAGE 2: UNIT TESTS (PARALLEL) ─────────────────
        stage('Unit Tests') {
            parallel {
                stage('Backend pytest') {
                    steps {
                        dir('tellus-ehs-hazcom-service') {
                            sh 'pytest tests/ -x -v --cov=app --cov-report=xml --junitxml=test-results.xml'
                        }
                    }
                    post {
                        always {
                            junit 'tellus-ehs-hazcom-service/test-results.xml'
                            publishCoverage adapters: [coberturaAdapter('tellus-ehs-hazcom-service/coverage.xml')]
                        }
                    }
                }
                stage('Frontend Vitest') {
                    steps {
                        dir('tellus-ehs-hazcom-ui') {
                            sh 'npm run test:run -- --reporter=junit --outputFile=test-results.xml'
                        }
                    }
                    post {
                        always {
                            junit 'tellus-ehs-hazcom-ui/test-results.xml'
                        }
                    }
                }
                stage('Background Service pytest') {
                    steps {
                        dir('tellus-ehs-background-service') {
                            sh 'pytest tests/ -x -v --cov=app --junitxml=test-results.xml'
                        }
                    }
                    post {
                        always {
                            junit 'tellus-ehs-background-service/test-results.xml'
                        }
                    }
                }
            }
        }

        // ─── STAGE 3: PLAYWRIGHT E2E ─────────────────────────
        stage('Playwright E2E') {
            steps {
                // Start backend
                dir('tellus-ehs-hazcom-service') {
                    sh 'nohup python -m app.main &'
                    sh 'sleep 5'  // Wait for backend to start
                }
                // Start frontend
                dir('tellus-ehs-hazcom-ui') {
                    sh 'nohup npm run dev:local &'
                    sh 'sleep 5'  // Wait for Vite dev server
                }
                // Run Playwright
                dir('tellus-ehs-hazcom-ui') {
                    sh 'npx playwright install chromium --with-deps'
                    sh 'npx playwright test --reporter=junit,html'
                }
            }
            post {
                always {
                    // Archive Playwright report
                    publishHTML(target: [
                        reportDir: 'tellus-ehs-hazcom-ui/playwright-report',
                        reportFiles: 'index.html',
                        reportName: 'Playwright Report'
                    ])
                    // Archive screenshots on failure
                    archiveArtifacts artifacts: 'tellus-ehs-hazcom-ui/test-results/**/*', allowEmptyArchive: true
                }
            }
        }

        // ─── STAGE 4: DEPLOY TO TEST (main branch only) ─────
        stage('Deploy to Test') {
            when {
                branch 'main'
            }
            steps {
                // Run database migrations on test environment
                dir('tellus-ehs-hazcom-service') {
                    sh 'DATABASE_URL=$TEST_DB_URL alembic upgrade head'
                }
                // Deploy backend (your deployment method)
                sh 'echo "Deploy backend to test environment"'
                // Deploy frontend
                dir('tellus-ehs-hazcom-ui') {
                    sh 'npm run build:dev'
                    sh 'echo "Deploy frontend to test environment"'
                }
            }
        }
    }

    post {
        failure {
            // Notify on failure (Slack, email, etc.)
            echo 'Pipeline failed — check test results'
        }
        success {
            echo 'All stages passed'
        }
    }
}

2.3 What Each Stage Catches

Stage	Time	What It Catches
Lint & Build	~1 min	Syntax errors, type errors, import issues, formatting violations
Unit Tests	~1 min	Broken business logic, schema validation, API contract changes, Redux state bugs
Playwright E2E	~3-5 min	Broken UI flows, missing buttons, form saves that fail, status transitions that break, modals that don't open
Deploy to Test	~2 min	Migration failures, deployment config issues

Total pipeline time: ~7-9 minutes

2.4 Pipeline Rules

All stages must pass to merge a PR
Stage 3 (Playwright) is the gate — if the UI works, ship it
Failed tests produce artifacts: screenshots, traces, JUnit reports
Coverage reports are published but not enforced as a gate (coverage numbers lie in vibe-coded apps)

Phase 3: Test Environment Verification

After code merges to main and deploys to the test environment. This is where you catch issues that don't appear locally.

3.1 Automated (Runs After Every Deploy to Test)

Jenkins triggers these automatically after Stage 4 deploys:

stage('Test Environment Verification') {
    when { branch 'main' }
    steps {
        dir('tellus-ehs-hazcom-ui') {
            // Run Playwright smoke suite against test environment
            sh '''
                TEST_BASE_URL=https://test.tellus-ehs.com \
                npx playwright test e2e/flows/smoke.spec.ts --reporter=html
            '''
        }
    }
}

What this catches: Deployment broke something, environment config differs from local, database migration didn't apply correctly, CORS issues, SSL issues.

3.2 Manual QA Checklist (After Feature Deploys)

Run by QA or the developer on the test environment — not localhost.

Basic Plan Flow

Premium Plan Flow (if AI features changed)

Create a premium plan
"Preparing questions..." overlay appears
Overlay dismisses — questionnaire shows static + dynamic questions
AI-suggested answers appear with accept/dismiss
Accept some, manually fill others
Generate Plan — AI generates content (takes 30-60s)
Content quality is reasonable (not garbage/empty)

Rejection Flow

Submit a plan → reject with reason → plan returns to draft
Rejection notes visible to the creator
Creator can edit and resubmit

Permission Checks

Non-draft plan: questionnaire is read-only
Non-draft plan: no "Generate Plan" or "Submit" buttons
Non-approver: no "Approve" or "Reject" buttons on pending plan
Archived plan: only "Create New Version" visible

Audit Logs

Plan-specific logs show all actions
Company-wide logs show actions across plans
Filtering by action/category works
CSV export downloads correctly

Cross-Browser (Quick Check)

Chrome: basic plan lifecycle works
Safari: page loads, questionnaire works
Mobile (responsive): page is usable on phone-sized screen

3.3 Data Integrity Checks

Run on the test database after deployment:

-- Verify no orphaned sections (sections without a plan)
SELECT s.section_id FROM plan_hazcom_plan_sections s
LEFT JOIN plan_hazcom_plans p ON s.plan_id = p.plan_id
WHERE p.plan_id IS NULL;

-- Verify only one active plan per site
SELECT site_id, COUNT(*) as active_count
FROM plan_hazcom_plans
WHERE version_status = 'active'
GROUP BY site_id
HAVING COUNT(*) > 1;

-- Verify version chain integrity
SELECT p.plan_id, p.version_number, p.previous_version_id
FROM plan_hazcom_plans p
WHERE p.previous_version_id IS NOT NULL
AND NOT EXISTS (
    SELECT 1 FROM plan_hazcom_plans prev
    WHERE prev.plan_id = p.previous_version_id
);

-- Verify audit logs exist for all plans
SELECT p.plan_id, p.plan_name
FROM plan_hazcom_plans p
LEFT JOIN plan_hazcom_audit_logs a ON p.plan_id = a.plan_id AND a.action = 'created'
WHERE a.log_id IS NULL;

3.4 Performance Checks (If Applicable)

Plan list page loads in < 2 seconds with 50+ plans
Questionnaire loads in < 1 second
Split view editor doesn't lag on typing (debounce works)
PDF export completes in < 10 seconds

Phase 4: Pre-Production Checklist

What must pass before deploying to production. This is the final gate.

4.1 Automated Gate (Jenkins)

stage('Pre-Production Gate') {
    when { branch 'release/*' }
    steps {
        // Full Playwright suite against test environment
        dir('tellus-ehs-hazcom-ui') {
            sh '''
                TEST_BASE_URL=https://test.tellus-ehs.com \
                npx playwright test --reporter=html
            '''
        }
        // Full backend test suite
        dir('tellus-ehs-hazcom-service') {
            sh 'pytest tests/ -v --tb=short'
        }
    }
}

4.2 Pre-Production Checklist

Code & Tests

All Jenkins pipeline stages green (lint, unit, integration, Playwright)
Full Playwright suite passes against test environment (not just smoke)
No open P0/P1 bugs related to this release
Code reviewed and approved by at least one other developer

Database

Migration runs cleanly on test DB: alembic upgrade head
Migration rollback tested: alembic downgrade -1 + alembic upgrade head
No destructive migrations (dropping columns with data) without a data migration plan
If new tables/columns: verify they exist after migration with a query

Configuration

Environment variables set in production (check for any new ones)
SQS queue exists and background service can consume from it
CORS origins include the production frontend URL
Supabase keys are for the production project (not dev)

Rollback Plan

Previous working version tagged in git
Database migration can be rolled back: alembic downgrade -1
If migration is not reversible: documented manual rollback steps
Deployment can be reverted within 5 minutes

AI-Specific (If AI features changed)

Claude API key is set in production environment
SQS background service is running and processing messages
Test a premium plan creation in test env — AI pipeline completes
Content quality spot-check: generated content makes sense

4.3 Deploy to Production

Tag the release:        git tag v1.x.x && git push --tags
Run migrations:         alembic upgrade head (on production DB)
Deploy backend:         (your deployment process)
Deploy frontend:        npm run build:prod → deploy to CDN/hosting
Post-deploy smoke test: (see below)

4.4 Post-Deploy Smoke Test (Production)

Run immediately after deploying to production. Takes < 5 minutes manually.

Login works — can sign in with a real account
Plan list loads — shows existing plans without errors
Create a draft plan — new plan appears in the list
Fill one section and save — answer persists on reload
Open split view — if plan has content, editor and preview render
Audit logs load — company audit log page shows entries
Delete the test plan — clean up test data

If any of these fail: Roll back immediately. Do not investigate on production.

Optional — run Playwright smoke against production (read-only tests only):

TEST_BASE_URL=https://app.tellus-ehs.com \
npx playwright test e2e/flows/smoke.spec.ts

Phase 5: Ongoing / Regression

What to do on an ongoing basis to keep quality up.

5.1 Scheduled Playwright Runs

Configure Jenkins to run the full Playwright suite nightly against the test environment:

// Jenkinsfile (or separate job)
triggers {
    cron('H 2 * * *')  // Run at 2 AM daily
}

Why: Catches regressions from indirect changes (dependency updates, environment drift, API changes from other services).

5.2 After Every Hotfix

A hotfix goes through the same pipeline, but faster:

Write a regression test that reproduces the bug
Fix the bug
Run: lint → pytest smoke → Playwright smoke → deploy to test
Quick manual verification on test
Deploy to production + post-deploy smoke

5.3 Before Major Releases

For big feature releases, add:

Full manual QA checklist (Phase 3.2) on test environment
Cross-browser testing (Chrome, Safari, Firefox)
Mobile responsive check
Performance spot-check with realistic data volume
Security review if auth/permissions changed

Summary: What Runs Where

┌─────────────────────────────────────────────────────────────────────┐
│                                                                     │
│  DEVELOPER MACHINE (during development)                              │
│  ├── While coding: lint + type check            (~30s)              │
│  ├── While coding: pytest smoke + Vitest        (~30s)              │
│  ├── After each feature: FULL Playwright suite  (~5-8 min)  ← KEY  │
│  └── Before push: all of the above              (gate check)        │
│                                                                     │
│  JENKINS PIPELINE (on every push / PR)                              │
│  ├── Stage 1: Lint & Build (parallel)           (~1 min)            │
│  ├── Stage 2: Unit Tests (parallel)             (~1 min)            │
│  ├── Stage 3: Playwright E2E                    (~3-5 min)          │
│  └── Stage 4: Deploy to Test (main only)        (~2 min)            │
│                                                                     │
│  TEST ENVIRONMENT (after deploy to test)                            │
│  ├── Automated: Playwright smoke against test   (~2 min)            │
│  ├── Manual: QA checklist                       (~30 min)           │
│  ├── Manual: Cross-browser spot check           (~15 min)           │
│  └── SQL: Data integrity queries                (~5 min)            │
│                                                                     │
│  PRE-PRODUCTION GATE (before deploy to prod)                        │
│  ├── Full Playwright suite against test env     (~5-8 min)          │
│  ├── Full pytest suite                          (~2 min)            │
│  ├── Migration dry run verified                 (~5 min)            │
│  └── Rollback plan documented                   (checklist)         │
│                                                                     │
│  PRODUCTION (after deploy)                                          │
│  ├── Post-deploy smoke test                     (~5 min manual)     │
│  └── Monitor error logs for 30 minutes          (passive)           │
│                                                                     │
│  ONGOING                                                            │
│  ├── Nightly: Full Playwright suite on test     (scheduled)         │
│  └── Hotfixes: regression test → fix → pipeline (same day)         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Test File Locations

Backend (`tellus-ehs-hazcom-service`)

tests/
├── conftest.py                    # DB session, auth mock, SQS mock, seed data
├── flows/
│   ├── test_smoke.py              # 5 fast API smoke tests
│   ├── test_basic_lifecycle.py    # Full CRUD + approval flow
│   ├── test_permissions.py        # Draft-only editing enforcement
│   ├── test_versioning.py         # Version chain + archive
│   ├── test_audit_trail.py        # Audit log creation + filtering
│   └── test_edge_cases.py         # Invalid transitions, pagination, 404s
├── unit/
│   ├── test_plan_service.py       # Service layer logic
│   ├── test_audit_service.py      # Audit logging
│   └── test_schemas.py            # Pydantic validation
└── helpers.py                     # create_plan(), fill_sections(), etc.

Frontend (`tellus-ehs-hazcom-ui`)

e2e/                               # Playwright E2E tests
├── fixtures/
│   ├── auth.ts                    # Login fixture
│   └── plan-helpers.ts            # API shortcuts for setup
├── pages/
│   └── plan-editor.page.ts        # Page Object Model
├── flows/
│   ├── smoke.spec.ts              # 5 smoke tests (< 2 min)
│   ├── basic-plan-lifecycle.spec.ts
│   ├── editing-permissions.spec.ts
│   ├── rejection-flow.spec.ts
│   ├── split-view-editing.spec.ts
│   ├── versioning.spec.ts
│   └── questionnaire.spec.ts
src/
├── test/setup.ts                  # Vitest setup (exists)
├── store/slices/*.test.ts         # Redux slice tests (exist)
├── utils/storage.test.ts          # Utility tests (exists)
└── services/api/
    └── hazcom-plan.api.test.ts    # API function unit tests

Background Service (`tellus-ehs-background-service`)

tests/
├── conftest.py                    # DB session, Claude AI mock
├── handlers/
│   ├── test_ai_generate.py        # Content generation handler
│   ├── test_ai_prefill.py         # Answer suggestion handler
│   └── test_pro_plan_setup.py     # Multi-phase PRO pipeline
└── test_sqs_routing.py            # Message routing to correct handler

Quick Reference: Commands

# ─── DEVELOPER (run after each feature) ────────────────
cd tellus-ehs-hazcom-service && pytest tests/ -x -v
cd tellus-ehs-hazcom-ui && npm run test:run
cd tellus-ehs-hazcom-ui && npx playwright test          # Full regression!

# ─── FULL TEST SUITE ───────────────────────────────────
cd tellus-ehs-hazcom-service && pytest tests/ -v --cov=app
cd tellus-ehs-hazcom-ui && npm run test:coverage
cd tellus-ehs-hazcom-ui && npx playwright test

# ─── PLAYWRIGHT (various modes) ────────────────────────
npx playwright test                                 # Headless
npx playwright test --headed                        # See the browser
npx playwright test --ui                            # Interactive debug UI
npx playwright test --debug e2e/flows/smoke.spec.ts # Step through
npx playwright show-report                          # View HTML report

# ─── AGAINST TEST ENVIRONMENT ──────────────────────────
TEST_BASE_URL=https://test.tellus-ehs.com npx playwright test e2e/flows/smoke.spec.ts

# ─── DATABASE MIGRATION TEST ──────────────────────────
cd tellus-ehs-hazcom-service
alembic upgrade head && alembic downgrade -1 && alembic upgrade head

Flow / Automation Tests (Playwright) — Implementation-ready Playwright test suites
Implementation Plan — What was built
Database Schema — Tables and relationships
Product Overview — Feature specifications

Overview​

Tech Stack​

Environments​

Phase 1: During Development (Developer's Machine)​

1.1 Write Tests Alongside Features​

1.2 Development Workflow: Quick Checks While Coding​

1.3 After Every Feature: Full Playwright Regression​

1.4 Before Every Push: Gate Check​

1.5 Developer Testing Checklist​

Phase 2: Jenkins Pipeline (On Every Push / PR)​

2.1 Pipeline Design​

2.2 Jenkinsfile​

2.3 What Each Stage Catches​

2.4 Pipeline Rules​

Phase 3: Test Environment Verification​

3.1 Automated (Runs After Every Deploy to Test)​

3.2 Manual QA Checklist (After Feature Deploys)​

Basic Plan Flow​

Premium Plan Flow (if AI features changed)​

Rejection Flow​

Permission Checks​

Audit Logs​

Cross-Browser (Quick Check)​

3.3 Data Integrity Checks​

3.4 Performance Checks (If Applicable)​

Phase 4: Pre-Production Checklist​

4.1 Automated Gate (Jenkins)​

4.2 Pre-Production Checklist​

Code & Tests​

Database​

Configuration​

Rollback Plan​

AI-Specific (If AI features changed)​

4.3 Deploy to Production​

4.4 Post-Deploy Smoke Test (Production)​

Phase 5: Ongoing / Regression​

5.1 Scheduled Playwright Runs​

5.2 After Every Hotfix​

5.3 Before Major Releases​

Summary: What Runs Where​

Test File Locations​

Backend (tellus-ehs-hazcom-service)​

Frontend (tellus-ehs-hazcom-ui)​

Background Service (tellus-ehs-background-service)​

Quick Reference: Commands​

Related Documentation​