Testing Strategy

Purpose: This document defines our testing philosophy, conventions, and tooling. Read this once to understand the "how" and "why" of testing at show.fm.

Testing Philosophy

The Testing Trophy

We follow a modified testing trophy approach, emphasizing integration tests as the highest-value investment:

        ___
       /   \      E2E Tests (few)
      /     \     - Critical user journeys only
     /_______\    - Expensive to run & maintain
     |       |
     |       |    Integration Tests (many) ← SWEET SPOT
     |_______|    - Business flows across services
        | |       - Mocked external services, real internal logic
        | |
        | |       Unit Tests (many)
        | |       - Pure functions, utilities
        | |       - Fast, isolated
        ===
     Static       TypeScript catches type errors at compile time

Core Principles

Test behavior, not implementation: Assert on outcomes (what data was passed, what response came back), not just that a function was called.
Mock at boundaries: Mock external services (Supabase, Stripe, R2, Resend, Google APIs) but test real internal logic.
Business logic coverage matters more than line coverage: 100% line coverage with no flow testing is worse than 60% coverage with comprehensive flow tests.
Tests are documentation: A test file should read like a specification of what the code does.

When to Write What

Situation	Test Type	Location	Example
Pure function, no side effects	Unit	`__tests__/` adjacent to source	`parseMagicTags()`, `escapeSearchQuery()`
Single service integration	Unit + Mocks	`__tests__/` adjacent to source	Stripe checkout, R2 upload
Cross-service business flow	Integration	`__tests__/` in entry point dir	Booking → Calendar → Email
Database queries/mutations	Unit + Mock Supabase	`__tests__/` adjacent to source	FTS search, CRUD operations
Shared middleware behavior	Integration	`src/api/middleware/__tests__/`	Rate limiting, auth enforcement contracts
Critical user journey	E2E (Playwright)	`tests/` root directory	Complete booking flow
Email/notification content	Unit	`templates/__tests__/`	Email HTML generation

Test Stack

yaml

unit_and_integration:
  runner: [email protected]
  config: vitest.config.ts
  environment: jsdom (default), node (workers)
  coverage: v8 provider
  pattern: src/**/__tests__/*.test.ts

e2e:
  runner: playwright
  config: playwright.config.ts
  pattern: tests/**/*.spec.ts

worker_tests:
  runner: vitest
  config: workers/[worker]/vitest.config.ts
  pattern: workers/**/__tests__/*.test.ts

Running Tests

bash

# All unit/integration tests
pnpm test

# Watch mode during development
pnpm test:watch

# Specific module
pnpm test src/lib/automation

# With coverage report
pnpm test:coverage

# E2E tests
pnpm test:e2e

# Worker tests
cd workers/automation-executor && pnpm test

File Structure Conventions

src/
├── api/routes/
│   └── bookings/
│       ├── index.ts                    # Route implementation
│       └── __tests__/
│           ├── index.test.ts           # Unit tests for routes
│           └── automation-integration.test.ts  # Integration tests
├── api/middleware/
│   ├── rate-limit.ts
│   └── __tests__/
│       └── rate-limit.test.ts          # Middleware contract tests
├── api/routes/guests/
│   ├── index.ts
│   └── __tests__/
│       └── index.test.ts               # Route-level limiter wiring
├── lib/
│   └── automation/
│       ├── events.ts                   # Module implementation
│       ├── scheduler.ts
│       └── __tests__/
│           ├── events.test.ts          # Unit tests
│           └── scheduler.test.ts
tests/
└── booking-flow.spec.ts                # E2E tests (Playwright)

Naming Conventions

Type	Pattern	Example
Unit test	`{module}.test.ts`	`events.test.ts`
Integration test	`{feature}-integration.test.ts`	`automation-integration.test.ts`
E2E test	`{flow}.spec.ts`	`booking-flow.spec.ts`

Mocking Patterns

Supabase (Database)

typescript

import { createMockSupabaseClient } from '$test';

// Use the chainable mock factory
const mockSupabase = createMockSupabaseClient({
	automation_rules: {
		select: [{ id: 'rule-123', is_enabled: true }]
	}
});

vi.mock('@supabase/supabase-js', () => ({
	createClient: () => mockSupabase
}));

Stripe (Payments)

typescript

vi.mock('stripe', () => ({
	default: class MockStripe {
		checkout = {
			sessions: { create: vi.fn().mockResolvedValue({ id: 'cs_test', url: '...' }) }
		};
		webhooks = {
			constructEvent: vi.fn().mockReturnValue(mockEvent)
		};
	}
}));

R2 (Storage)

typescript

import { createMockR2Bucket } from '$test';

const mockBucket = createMockR2Bucket();
// Supports: put, get, delete, list, head with in-memory storage

Google APIs (Calendar)

typescript

vi.mock('$lib/google-calendar/auth', () => ({
	refreshAccessToken: vi.fn().mockResolvedValue({ access_token: 'ya29.mock' }),
	exchangeCodeForTokens: vi.fn().mockResolvedValue({ access_token: '...', refresh_token: '...' })
}));

Resend (Email)

typescript

vi.mock('resend', () => ({
	Resend: class MockResend {
		emails = { send: vi.fn().mockResolvedValue({ id: 'email-123' }) };
	}
}));

Behavior Assertions (CRITICAL)

Bad - Only verifies the method was called:

typescript

expect(mockSupabaseUpdate).toHaveBeenCalled(); // Weak

Good - Verifies WHAT was passed:

typescript

expect(mockSupabaseUpdate).toHaveBeenCalledWith({
	subscription_tier: 'professional',
	stripe_customer_id: 'cus_123',
	subscription_status: 'active'
}); // Strong - catches bugs where wrong data is written

Assertion Checklist

For each test, verify the actual data:

Operation	What to Assert
DB write	Exact payload passed to insert/update
API call	Request body, headers, URL
Queue message	Message structure and contents
Email send	Recipient, subject, body content
Response	Status code AND response body

Security Testing (Required for API/DB code)

Every API route test file should include:

typescript

describe('Security', () => {
	it('should reject unauthenticated requests', async () => {
		const res = await route.request(reqWithoutAuth);
		expect(res.status).toBe(401);
	});

	it('should enforce RLS - user cannot access other podcasts', async () => {
		// Test cross-tenant access attempt
	});

	it('should validate and sanitize inputs', async () => {
		// Test with XSS vectors, SQL injection patterns
	});
});

Rate-Limit Coverage (Required for Public Endpoints)

For endpoints using rateLimit(...), add both test layers:

Middleware contract test (src/api/middleware/__tests__/rate-limit.test.ts)
- asserts 429 behavior, Retry-After, X-RateLimit-*, identifier modes, and fail-open behavior
Route wiring test (adjacent route __tests__/index.test.ts)
- asserts the route is actually protected and short-circuits before handler side effects

bash

pnpm test src/api/middleware/__tests__/rate-limit.test.ts
pnpm test src/api/routes/guests/__tests__/index.test.ts

Accessibility Testing (Required for UI Components)

typescript

import { axe, toHaveNoViolations } from 'jest-axe';

expect.extend(toHaveNoViolations);

it('should have no accessibility violations', async () => {
	const { container } = render(Component);
	expect(await axe(container)).toHaveNoViolations();
});

Coverage Thresholds

Coverage thresholds are enforced in vitest.config.ts. After adding tests:

Run pnpm test:coverage
If coverage improved, update thresholds (always round DOWN)
Never decrease thresholds without discussion

typescript

// vitest.config.ts
coverage: {
  thresholds: {
    statements: 35,
    branches: 30,
    functions: 25,
    lines: 35
  }
}

What We Don't Test

shadcn UI components (src/lib/components/ui/) - Third-party, well-tested
Type definitions (src/lib/types/) - No runtime behavior
Generated code - Database types, etc.
Test utilities (src/test/) - Testing the tests

Complex Flow Testing (Scout → Generate Workflow)

For horizontal flows that span multiple services, we use a two-stage pipeline. The scout stage assesses complexity to determine the appropriate workflow.

Stage 1: Scout the Flow

bash

/scout-test-flow [flow-name]

Scout traces the flow and scores its complexity based on 6 factors:

Factor	Simple (0 pts)	Complex (1 pt)
File count	≤3 files	>3 files
Hop count	≤2 hops	>2 hops
Worker boundary	Same runtime	Crosses to worker
External services	1 service	2+ services
Time-based scheduling	No delays	Has scheduled jobs
Multiple triggers	Single trigger	Multiple trigger types

Total score determines output:

Score 0-1 (Simple): Scout outputs a direct command—no spec file created
Score 2+ (Complex): Scout creates a spec file at docs/testing/flows/[flow-name]-spec.md

Stage 2: Generate Tests

The generate command has two modes depending on scout output:

Discovery Mode (Simple Flows)

When scout provides a direct command:

bash

/generate-tests [flow-name] path/to/file1.ts path/to/file2.ts

Generate explores the provided files and discovers the test scenarios.

Spec-Driven Mode (Complex Flows)

When scout created a spec file:

bash

/generate-tests --spec=docs/testing/flows/[flow-name]-spec.md

Generate reads only the files listed in the spec and uses the defined scenarios as the test plan. After generating tests, the spec file is updated with the generated test file locations and scenario coverage.

Why Two Stages?

Complex flows can involve 10+ files across multiple services. The spec file acts as a compressed "blueprint" that preserves context across conversations and tracks which scenarios have tests.

Fresh Conversation

Always run /generate-tests in a fresh conversation to avoid context buildup from the scout phase.

Stage 3: Sync Domain Map

After tests are generated, update the Domain Map to track coverage:

bash

/sync-domain-map --auto

This command:

Detects recently created/modified test files
Updates flow statuses (🔴 → 🟢)
Adds new files to the Test File Index
Recalculates Quick Stats

You can also target specific files or domains:

bash

/sync-domain-map src/lib/automation/__tests__/executor.test.ts
/sync-domain-map --domain automation

Stage 4: Update Internal Docs

Finally, update any related feature documentation:

bash

/write-internal-docs testing

Adding New Tests Checklist

[ ] Test file in correct location (__tests__/ adjacent to source)
[ ] Named correctly (*.test.ts for unit, *-integration.test.ts for integration)
[ ] External services mocked (Supabase, Stripe, R2, Resend, Google)
[ ] Behavior assertions verify actual data, not just method calls
[ ] Security tests included for API routes
[ ] Coverage thresholds updated if improved
[ ] package.json script added if new module: "test:modulename": "vitest run src/lib/modulename"

Quick Reference

Command	Purpose
`pnpm test`	Run all tests
`pnpm test:watch`	Watch mode
`pnpm test:coverage`	Coverage report
`pnpm test:e2e`	Playwright E2E
`pnpm test src/lib/X`	Test specific module

Domain Map - Business logic test coverage tracking
Mock Factories - Complete mock library documentation
Testing Patterns - Svelte 5, API, and Worker testing patterns
Testing Infrastructure - Stack overview and setup

Testing Strategy ​

Testing Philosophy ​

The Testing Trophy ​

Core Principles ​

When to Write What ​

Test Stack ​

Running Tests ​

File Structure Conventions ​

Naming Conventions ​

Mocking Patterns ​

Supabase (Database) ​

Stripe (Payments) ​

R2 (Storage) ​

Google APIs (Calendar) ​

Resend (Email) ​

Behavior Assertions (CRITICAL) ​

Assertion Checklist ​

Security Testing (Required for API/DB code) ​

Rate-Limit Coverage (Required for Public Endpoints) ​

Accessibility Testing (Required for UI Components) ​

Coverage Thresholds ​

What We Don't Test ​

Complex Flow Testing (Scout → Generate Workflow) ​

Stage 1: Scout the Flow ​

Stage 2: Generate Tests ​

Discovery Mode (Simple Flows) ​

Spec-Driven Mode (Complex Flows) ​

Stage 3: Sync Domain Map ​

Stage 4: Update Internal Docs ​

Adding New Tests Checklist ​

Quick Reference ​

Related Documentation ​

Testing Strategy

Testing Philosophy

The Testing Trophy

Core Principles

When to Write What

Test Stack

Running Tests

File Structure Conventions

Naming Conventions

Mocking Patterns

Supabase (Database)

Stripe (Payments)

R2 (Storage)

Google APIs (Calendar)

Resend (Email)

Behavior Assertions (CRITICAL)

Assertion Checklist

Security Testing (Required for API/DB code)

Rate-Limit Coverage (Required for Public Endpoints)

Accessibility Testing (Required for UI Components)

Coverage Thresholds

What We Don't Test

Complex Flow Testing (Scout → Generate Workflow)

Stage 1: Scout the Flow

Stage 2: Generate Tests

Discovery Mode (Simple Flows)

Spec-Driven Mode (Complex Flows)

Stage 3: Sync Domain Map

Stage 4: Update Internal Docs

Adding New Tests Checklist

Quick Reference

Related Documentation