# mockupAWS v0.5.0 Testing Strategy

## Overview

This document outlines the comprehensive testing strategy for mockupAWS v0.5.0, focusing on the new authentication, API keys, and advanced filtering features.

**Test Period:** 2026-04-07 onwards  
**Target Version:** v0.5.0  
**QA Engineer:** @qa-engineer  

---

## Test Objectives

1. **Authentication System** - Verify JWT-based authentication flow works correctly
2. **API Key Management** - Test API key creation, revocation, and access control
3. **Advanced Filters** - Validate filtering functionality on scenarios list
4. **E2E Regression** - Ensure v0.4.0 features work with new auth requirements

---

## Test Suite Overview

| Test Suite | File | Test Count | Priority |
|------------|------|------------|----------|
| QA-AUTH-019 | `auth.spec.ts` | 18+ | P0 (Critical) |
| QA-APIKEY-020 | `apikeys.spec.ts` | 20+ | P0 (Critical) |
| QA-FILTER-021 | `scenarios.spec.ts` | 24+ | P1 (High) |
| QA-E2E-022 | `regression-v050.spec.ts` | 15+ | P1 (High) |

---

## QA-AUTH-019: Authentication Tests

**File:** `frontend/e2e/auth.spec.ts`

### Test Categories

#### 1. Registration Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REG-001 | Register new user successfully | Redirect to dashboard, token stored |
| REG-002 | Duplicate email registration | Error message displayed |
| REG-003 | Password mismatch | Validation error shown |
| REG-004 | Invalid email format | Validation error shown |
| REG-005 | Weak password | Validation error shown |
| REG-006 | Missing required fields | Validation errors displayed |
| REG-007 | Navigate to login from register | Login page displayed |

#### 2. Login Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| LOG-001 | Login with valid credentials | Redirect to dashboard |
| LOG-002 | Login with invalid credentials | Error message shown |
| LOG-003 | Login with non-existent user | Error message shown |
| LOG-004 | Invalid email format | Validation error shown |
| LOG-005 | Navigate to register from login | Register page displayed |
| LOG-006 | Navigate to forgot password | Password reset page displayed |

#### 3. Protected Routes Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| PROT-001 | Access /scenarios without auth | Redirect to login |
| PROT-002 | Access /profile without auth | Redirect to login |
| PROT-003 | Access /settings without auth | Redirect to login |
| PROT-004 | Access /settings/api-keys without auth | Redirect to login |
| PROT-005 | Access /scenarios with auth | Page displayed |
| PROT-006 | Auth persistence after refresh | Still authenticated |

#### 4. Logout Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| OUT-001 | Logout redirects to login | Login page displayed |
| OUT-002 | Clear tokens on logout | localStorage cleared |
| OUT-003 | Access protected route after logout | Redirect to login |

#### 5. Token Management Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| TOK-001 | Token refresh mechanism | New tokens issued |
| TOK-002 | Store tokens in localStorage | Tokens persisted |

---

## QA-APIKEY-020: API Keys Tests

**File:** `frontend/e2e/apikeys.spec.ts`

### Test Categories

#### 1. Create API Key (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CREATE-001 | Navigate to API Keys page | Settings page loaded |
| CREATE-002 | Create new API key | Modal with full key displayed |
| CREATE-003 | Copy API key to clipboard | Success message shown |
| CREATE-004 | Key appears in list after creation | Key visible in table |
| CREATE-005 | Validate required fields | Error message shown |

#### 2. Revoke API Key (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REVOKE-001 | Revoke API key | Key removed from list |
| REVOKE-002 | Confirm before revoke | Confirmation dialog shown |

#### 3. API Access with Key (API)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| ACCESS-001 | Access API with valid key | 200 OK |
| ACCESS-002 | Access /auth/me with key | User info returned |
| ACCESS-003 | Access with revoked key | 401 Unauthorized |
| ACCESS-004 | Access with invalid key format | 401 Unauthorized |
| ACCESS-005 | Access with non-existent key | 401 Unauthorized |
| ACCESS-006 | Access without key header | 401 Unauthorized |
| ACCESS-007 | Respect API key scopes | Operations allowed per scope |
| ACCESS-008 | Track last used timestamp | Timestamp updated |

#### 4. API Key Management (API)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| MGMT-001 | List all API keys | Keys returned without full key |
| MGMT-002 | Key prefix in list | Prefix visible, full key hidden |
| MGMT-003 | Create key with expiration | Expiration date set |
| MGMT-004 | Rotate API key | New key issued, old revoked |

#### 5. API Key List View (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| LIST-001 | Display keys table | All columns visible |
| LIST-002 | Empty state | Message shown when no keys |
| LIST-003 | Display key prefix | Prefix visible in table |

---

## QA-FILTER-021: Filters Tests

**File:** `frontend/e2e/scenarios.spec.ts`

### Test Categories

#### 1. Region Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REGION-001 | Apply us-east-1 filter | Only us-east-1 scenarios shown |
| REGION-002 | Apply eu-west-1 filter | Only eu-west-1 scenarios shown |
| REGION-003 | No region filter | All scenarios shown |

#### 2. Cost Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COST-001 | Apply min cost filter | Scenarios above min shown |
| COST-002 | Apply max cost filter | Scenarios below max shown |
| COST-003 | Apply cost range | Scenarios within range shown |

#### 3. Status Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| STATUS-001 | Filter by draft status | Only draft scenarios shown |
| STATUS-002 | Filter by running status | Only running scenarios shown |

#### 4. Combined Filters
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COMBINE-001 | Combine region + status | Both filters applied |
| COMBINE-002 | URL sync with filters | Query params updated |
| COMBINE-003 | Parse filters from URL | Filters applied on load |
| COMBINE-004 | Multiple regions in URL | All regions filtered |

#### 5. Clear Filters
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CLEAR-001 | Clear all filters | Full list restored |
| CLEAR-002 | Clear individual filter | Specific filter removed |
| CLEAR-003 | Clear on refresh | Filters reset |

#### 6. Search by Name
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| SEARCH-001 | Search by exact name | Matching scenario shown |
| SEARCH-002 | Partial name match | Partial matches shown |
| SEARCH-003 | Non-matching search | Empty results or message |
| SEARCH-004 | Combine search + filters | Both applied |
| SEARCH-005 | Clear search | All results shown |

#### 7. Date Range Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| DATE-001 | Filter by from date | Scenarios after date shown |
| DATE-002 | Filter by date range | Scenarios within range shown |

---

## QA-E2E-022: E2E Regression Tests

**File:** `frontend/e2e/regression-v050.spec.ts`

### Test Categories

#### 1. Scenario CRUD with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CRUD-001 | Display scenarios list | Table with headers visible |
| CRUD-002 | Navigate to scenario detail | Detail page loaded |
| CRUD-003 | Display scenario metrics | All metrics visible |
| CRUD-004 | 404 for non-existent scenario | Error message shown |

#### 2. Log Ingestion with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| INGEST-001 | Start scenario and ingest logs | Logs accepted, metrics updated |
| INGEST-002 | Persist metrics after refresh | Metrics remain visible |

#### 3. Reports with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REPORT-001 | Generate PDF report | Report created successfully |
| REPORT-002 | Generate CSV report | Report created successfully |

#### 4. Navigation with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| NAV-001 | Navigate to dashboard | Dashboard loaded |
| NAV-002 | Navigate via sidebar | Routes work correctly |
| NAV-003 | 404 for invalid routes | Error page shown |
| NAV-004 | Maintain auth on navigation | User stays authenticated |

#### 5. Comparison with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COMPARE-001 | Compare 2 scenarios | Comparison data returned |
| COMPARE-002 | Compare 3 scenarios | Comparison data returned |

#### 6. API Authentication Errors
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| AUTHERR-001 | Access API without token | 401 returned |
| AUTHERR-002 | Access with invalid token | 401 returned |
| AUTHERR-003 | Access with malformed header | 401 returned |

---

## Test Execution Plan

### Phase 1: Prerequisites Check
- [ ] Backend auth endpoints implemented (BE-AUTH-003)
- [ ] Frontend auth pages implemented (FE-AUTH-009, FE-AUTH-010)
- [ ] API Keys endpoints implemented (BE-APIKEY-005)
- [ ] API Keys UI implemented (FE-APIKEY-011)
- [ ] Filters UI implemented (FE-FILTER-012)

### Phase 2: Authentication Tests
1. Execute `auth.spec.ts` tests
2. Verify all registration scenarios
3. Verify all login scenarios
4. Verify protected routes behavior
5. Verify logout flow

### Phase 3: API Keys Tests
1. Execute `apikeys.spec.ts` tests
2. Verify key creation flow
3. Verify key revocation
4. Verify API access with keys
5. Verify key rotation

### Phase 4: Filters Tests
1. Execute `scenarios.spec.ts` tests
2. Verify region filters
3. Verify cost filters
4. Verify status filters
5. Verify combined filters
6. Verify search functionality

### Phase 5: Regression Tests
1. Execute `regression-v050.spec.ts` tests
2. Verify v0.4.0 features with auth
3. Check pass rate on Chromium

---

## Test Environment

### Requirements
- **Backend:** Running on http://localhost:8000
- **Frontend:** Running on http://localhost:5173
- **Database:** Migrated with v0.5.0 schema
- **Browsers:** Chromium (primary), Firefox, WebKit

### Configuration
```bash
# Run specific test suite
npx playwright test auth.spec.ts
npx playwright test apikeys.spec.ts
npx playwright test scenarios.spec.ts
npx playwright test regression-v050.spec.ts

# Run all v0.5.0 tests
npx playwright test auth.spec.ts apikeys.spec.ts scenarios.spec.ts regression-v050.spec.ts

# Run with HTML report
npx playwright test --reporter=html
```

---

## Expected Results

### Pass Rate Targets
- **Chromium:** >80%
- **Firefox:** >70%
- **WebKit:** >70%

### Critical Path (Must Pass)
1. User registration
2. User login
3. Protected route access control
4. API key creation
5. API key access authorization
6. Scenario list filtering

---

## Helper Utilities

### auth-helpers.ts
Provides authentication utilities:
- `registerUser()` - Register via API
- `loginUser()` - Login via API
- `loginUserViaUI()` - Login via UI
- `registerUserViaUI()` - Register via UI
- `logoutUser()` - Logout via UI
- `createAuthHeader()` - Create Bearer header
- `createApiKeyHeader()` - Create API key header
- `generateTestEmail()` - Generate test email
- `generateTestUser()` - Generate test user data

### test-helpers.ts
Updated with auth support:
- `createScenarioViaAPI()` - Now accepts accessToken
- `deleteScenarioViaAPI()` - Now accepts accessToken
- `startScenarioViaAPI()` - Now accepts accessToken
- `stopScenarioViaAPI()` - Now accepts accessToken
- `sendTestLogs()` - Now accepts accessToken

---

## Known Limitations

1. **API Availability:** Tests will skip if backend endpoints return 404
2. **Timing:** Some tests include wait times for async operations
3. **Cleanup:** Test data cleanup may fail silently
4. **Visual Tests:** Visual regression tests not included in v0.5.0

---

## Success Criteria

- [ ] All P0 tests passing on Chromium
- [ ] >80% overall pass rate on Chromium
- [ ] No critical authentication vulnerabilities
- [ ] API keys work correctly for programmatic access
- [ ] Filters update list in real-time
- [ ] URL sync works correctly
- [ ] v0.4.0 features still functional with auth

---

## Reporting

### Test Results Format
```
Test Suite: QA-AUTH-019
Total Tests: 18
Passed: 16 (89%)
Failed: 1
Skipped: 1

Test Suite: QA-APIKEY-020
Total Tests: 20
Passed: 18 (90%)
Failed: 1
Skipped: 1

Test Suite: QA-FILTER-021
Total Tests: 24
Passed: 20 (83%)
Failed: 2
Skipped: 2

Test Suite: QA-E2E-022
Total Tests: 15
Passed: 13 (87%)
Failed: 1
Skipped: 1

Overall Pass Rate: 85%
```

---

## Appendix: Test Data

### Test Users
- Email pattern: `user.{timestamp}@test.mockupaws.com`
- Password: `TestPassword123!`
- Full Name: `Test User {timestamp}`

### Test Scenarios
- Name pattern: `E2E Test {timestamp}`
- Regions: us-east-1, eu-west-1, ap-southeast-1, us-west-2, eu-central-1
- Status: draft, running, completed

### Test API Keys
- Name pattern: `Test API Key {purpose}`
- Scopes: read:scenarios, write:scenarios, read:reports
- Format: `mk_` + 32 random characters

---

*Document Version: 1.0*  
*Last Updated: 2026-04-07*  
*Prepared by: @qa-engineer*