lucasacchi/mockupAWS
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity
Files
main
mockupAWS/frontend/e2e/TEST-PLAN-v050.md
Luca Sacchi Ricciardi cc60ba17ea
Some checks failed
E2E Tests / Run E2E Tests (push) Has been cancelled
Details
E2E Tests / Visual Regression Tests (push) Has been cancelled
Details
E2E Tests / Smoke Tests (push) Has been cancelled
Details
release: v0.5.0 - Authentication, API Keys & Advanced Features 
Complete v0.5.0 implementation:

Database (@db-engineer):
- 3 migrations: users, api_keys, report_schedules tables
- Foreign keys, indexes, constraints, enums

Backend (@backend-dev):
- JWT authentication service with bcrypt (cost=12)
- Auth endpoints: /register, /login, /refresh, /me
- API Keys service with hash storage and prefix validation
- API Keys endpoints: CRUD + rotate
- Security module with JWT HS256

Frontend (@frontend-dev):
- Login/Register pages with validation
- AuthContext with localStorage persistence
- Protected routes implementation
- API Keys management UI (create, revoke, rotate)
- Header with user dropdown

DevOps (@devops-engineer):
- .env.example and .env.production.example
- docker-compose.scheduler.yml
- scripts/setup-secrets.sh
- INFRASTRUCTURE_SETUP.md

QA (@qa-engineer):
- 85 E2E tests: auth.spec.ts, apikeys.spec.ts, scenarios.spec.ts, regression-v050.spec.ts
- auth-helpers.ts with 20+ utility functions
- Test plans and documentation

Architecture (@spec-architect):
- SECURITY.md with best practices
- SECURITY-CHECKLIST.md pre-deployment
- Updated architecture.md with auth flows
- Updated README.md with v0.5.0 features

Documentation:
- Updated todo.md with v0.5.0 status
- Added docs/README.md index
- Complete setup instructions

Dependencies added:
- bcrypt, python-jose, passlib, email-validator

Tested: JWT auth flow, API keys CRUD, protected routes, 85 E2E tests ready

Closes: v0.5.0 milestone
2026-04-07 19:22:47 +02:00

422 lines
14 KiB
Markdown
Raw Permalink Blame History

# mockupAWS v0.5.0 Testing Strategy
## Overview
This document outlines the comprehensive testing strategy for mockupAWS v0.5.0, focusing on the new authentication, API keys, and advanced filtering features.
**Test Period:** 2026-04-07 onwards
**Target Version:** v0.5.0
**QA Engineer:** @qa-engineer
---
## Test Objectives
1. **Authentication System** - Verify JWT-based authentication flow works correctly
2. **API Key Management** - Test API key creation, revocation, and access control
3. **Advanced Filters** - Validate filtering functionality on scenarios list
4. **E2E Regression** - Ensure v0.4.0 features work with new auth requirements
---
## Test Suite Overview
| Test Suite | File | Test Count | Priority |
|------------|------|------------|----------|
| QA-AUTH-019 | `auth.spec.ts` | 18+ | P0 (Critical) |
| QA-APIKEY-020 | `apikeys.spec.ts` | 20+ | P0 (Critical) |
| QA-FILTER-021 | `scenarios.spec.ts` | 24+ | P1 (High) |
| QA-E2E-022 | `regression-v050.spec.ts` | 15+ | P1 (High) |
---
## QA-AUTH-019: Authentication Tests
**File:** `frontend/e2e/auth.spec.ts`
### Test Categories
#### 1. Registration Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REG-001 | Register new user successfully | Redirect to dashboard, token stored |
| REG-002 | Duplicate email registration | Error message displayed |
| REG-003 | Password mismatch | Validation error shown |
| REG-004 | Invalid email format | Validation error shown |
| REG-005 | Weak password | Validation error shown |
| REG-006 | Missing required fields | Validation errors displayed |
| REG-007 | Navigate to login from register | Login page displayed |
#### 2. Login Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| LOG-001 | Login with valid credentials | Redirect to dashboard |
| LOG-002 | Login with invalid credentials | Error message shown |
| LOG-003 | Login with non-existent user | Error message shown |
| LOG-004 | Invalid email format | Validation error shown |
| LOG-005 | Navigate to register from login | Register page displayed |
| LOG-006 | Navigate to forgot password | Password reset page displayed |
#### 3. Protected Routes Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| PROT-001 | Access /scenarios without auth | Redirect to login |
| PROT-002 | Access /profile without auth | Redirect to login |
| PROT-003 | Access /settings without auth | Redirect to login |
| PROT-004 | Access /settings/api-keys without auth | Redirect to login |
| PROT-005 | Access /scenarios with auth | Page displayed |
| PROT-006 | Auth persistence after refresh | Still authenticated |
#### 4. Logout Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| OUT-001 | Logout redirects to login | Login page displayed |
| OUT-002 | Clear tokens on logout | localStorage cleared |
| OUT-003 | Access protected route after logout | Redirect to login |
#### 5. Token Management Tests
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| TOK-001 | Token refresh mechanism | New tokens issued |
| TOK-002 | Store tokens in localStorage | Tokens persisted |
---
## QA-APIKEY-020: API Keys Tests
**File:** `frontend/e2e/apikeys.spec.ts`
### Test Categories
#### 1. Create API Key (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CREATE-001 | Navigate to API Keys page | Settings page loaded |
| CREATE-002 | Create new API key | Modal with full key displayed |
| CREATE-003 | Copy API key to clipboard | Success message shown |
| CREATE-004 | Key appears in list after creation | Key visible in table |
| CREATE-005 | Validate required fields | Error message shown |
#### 2. Revoke API Key (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REVOKE-001 | Revoke API key | Key removed from list |
| REVOKE-002 | Confirm before revoke | Confirmation dialog shown |
#### 3. API Access with Key (API)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| ACCESS-001 | Access API with valid key | 200 OK |
| ACCESS-002 | Access /auth/me with key | User info returned |
| ACCESS-003 | Access with revoked key | 401 Unauthorized |
| ACCESS-004 | Access with invalid key format | 401 Unauthorized |
| ACCESS-005 | Access with non-existent key | 401 Unauthorized |
| ACCESS-006 | Access without key header | 401 Unauthorized |
| ACCESS-007 | Respect API key scopes | Operations allowed per scope |
| ACCESS-008 | Track last used timestamp | Timestamp updated |
#### 4. API Key Management (API)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| MGMT-001 | List all API keys | Keys returned without full key |
| MGMT-002 | Key prefix in list | Prefix visible, full key hidden |
| MGMT-003 | Create key with expiration | Expiration date set |
| MGMT-004 | Rotate API key | New key issued, old revoked |
#### 5. API Key List View (UI)
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| LIST-001 | Display keys table | All columns visible |
| LIST-002 | Empty state | Message shown when no keys |
| LIST-003 | Display key prefix | Prefix visible in table |
---
## QA-FILTER-021: Filters Tests
**File:** `frontend/e2e/scenarios.spec.ts`
### Test Categories
#### 1. Region Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REGION-001 | Apply us-east-1 filter | Only us-east-1 scenarios shown |
| REGION-002 | Apply eu-west-1 filter | Only eu-west-1 scenarios shown |
| REGION-003 | No region filter | All scenarios shown |
#### 2. Cost Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COST-001 | Apply min cost filter | Scenarios above min shown |
| COST-002 | Apply max cost filter | Scenarios below max shown |
| COST-003 | Apply cost range | Scenarios within range shown |
#### 3. Status Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| STATUS-001 | Filter by draft status | Only draft scenarios shown |
| STATUS-002 | Filter by running status | Only running scenarios shown |
#### 4. Combined Filters
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COMBINE-001 | Combine region + status | Both filters applied |
| COMBINE-002 | URL sync with filters | Query params updated |
| COMBINE-003 | Parse filters from URL | Filters applied on load |
| COMBINE-004 | Multiple regions in URL | All regions filtered |
#### 5. Clear Filters
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CLEAR-001 | Clear all filters | Full list restored |
| CLEAR-002 | Clear individual filter | Specific filter removed |
| CLEAR-003 | Clear on refresh | Filters reset |
#### 6. Search by Name
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| SEARCH-001 | Search by exact name | Matching scenario shown |
| SEARCH-002 | Partial name match | Partial matches shown |
| SEARCH-003 | Non-matching search | Empty results or message |
| SEARCH-004 | Combine search + filters | Both applied |
| SEARCH-005 | Clear search | All results shown |
#### 7. Date Range Filter
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| DATE-001 | Filter by from date | Scenarios after date shown |
| DATE-002 | Filter by date range | Scenarios within range shown |
---
## QA-E2E-022: E2E Regression Tests
**File:** `frontend/e2e/regression-v050.spec.ts`
### Test Categories
#### 1. Scenario CRUD with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| CRUD-001 | Display scenarios list | Table with headers visible |
| CRUD-002 | Navigate to scenario detail | Detail page loaded |
| CRUD-003 | Display scenario metrics | All metrics visible |
| CRUD-004 | 404 for non-existent scenario | Error message shown |
#### 2. Log Ingestion with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| INGEST-001 | Start scenario and ingest logs | Logs accepted, metrics updated |
| INGEST-002 | Persist metrics after refresh | Metrics remain visible |
#### 3. Reports with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| REPORT-001 | Generate PDF report | Report created successfully |
| REPORT-002 | Generate CSV report | Report created successfully |
#### 4. Navigation with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| NAV-001 | Navigate to dashboard | Dashboard loaded |
| NAV-002 | Navigate via sidebar | Routes work correctly |
| NAV-003 | 404 for invalid routes | Error page shown |
| NAV-004 | Maintain auth on navigation | User stays authenticated |
#### 5. Comparison with Auth
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| COMPARE-001 | Compare 2 scenarios | Comparison data returned |
| COMPARE-002 | Compare 3 scenarios | Comparison data returned |
#### 6. API Authentication Errors
| Test Case | Description | Expected Result |
|-----------|-------------|-----------------|
| AUTHERR-001 | Access API without token | 401 returned |
| AUTHERR-002 | Access with invalid token | 401 returned |
| AUTHERR-003 | Access with malformed header | 401 returned |
---
## Test Execution Plan
### Phase 1: Prerequisites Check
- [ ] Backend auth endpoints implemented (BE-AUTH-003)
- [ ] Frontend auth pages implemented (FE-AUTH-009, FE-AUTH-010)
- [ ] API Keys endpoints implemented (BE-APIKEY-005)
- [ ] API Keys UI implemented (FE-APIKEY-011)
- [ ] Filters UI implemented (FE-FILTER-012)
### Phase 2: Authentication Tests
1. Execute `auth.spec.ts` tests
2. Verify all registration scenarios
3. Verify all login scenarios
4. Verify protected routes behavior
5. Verify logout flow
### Phase 3: API Keys Tests
1. Execute `apikeys.spec.ts` tests
2. Verify key creation flow
3. Verify key revocation
4. Verify API access with keys
5. Verify key rotation
### Phase 4: Filters Tests
1. Execute `scenarios.spec.ts` tests
2. Verify region filters
3. Verify cost filters
4. Verify status filters
5. Verify combined filters
6. Verify search functionality
### Phase 5: Regression Tests
1. Execute `regression-v050.spec.ts` tests
2. Verify v0.4.0 features with auth
3. Check pass rate on Chromium
---
## Test Environment
### Requirements
- **Backend:** Running on http://localhost:8000
- **Frontend:** Running on http://localhost:5173
- **Database:** Migrated with v0.5.0 schema
- **Browsers:** Chromium (primary), Firefox, WebKit
### Configuration
```bash
# Run specific test suite
npx playwright test auth.spec.ts
npx playwright test apikeys.spec.ts
npx playwright test scenarios.spec.ts
npx playwright test regression-v050.spec.ts
# Run all v0.5.0 tests
npx playwright test auth.spec.ts apikeys.spec.ts scenarios.spec.ts regression-v050.spec.ts
# Run with HTML report
npx playwright test --reporter=html
```
---
## Expected Results
### Pass Rate Targets
- **Chromium:** >80%
- **Firefox:** >70%
- **WebKit:** >70%
### Critical Path (Must Pass)
1. User registration
2. User login
3. Protected route access control
4. API key creation
5. API key access authorization
6. Scenario list filtering
---
## Helper Utilities
### auth-helpers.ts
Provides authentication utilities:
- `registerUser()` - Register via API
- `loginUser()` - Login via API
- `loginUserViaUI()` - Login via UI
- `registerUserViaUI()` - Register via UI
- `logoutUser()` - Logout via UI
- `createAuthHeader()` - Create Bearer header
- `createApiKeyHeader()` - Create API key header
- `generateTestEmail()` - Generate test email
- `generateTestUser()` - Generate test user data
### test-helpers.ts
Updated with auth support:
- `createScenarioViaAPI()` - Now accepts accessToken
- `deleteScenarioViaAPI()` - Now accepts accessToken
- `startScenarioViaAPI()` - Now accepts accessToken
- `stopScenarioViaAPI()` - Now accepts accessToken
- `sendTestLogs()` - Now accepts accessToken
---
## Known Limitations
1. **API Availability:** Tests will skip if backend endpoints return 404
2. **Timing:** Some tests include wait times for async operations
3. **Cleanup:** Test data cleanup may fail silently
4. **Visual Tests:** Visual regression tests not included in v0.5.0
---
## Success Criteria
- [ ] All P0 tests passing on Chromium
- [ ] >80% overall pass rate on Chromium
- [ ] No critical authentication vulnerabilities
- [ ] API keys work correctly for programmatic access
- [ ] Filters update list in real-time
- [ ] URL sync works correctly
- [ ] v0.4.0 features still functional with auth
---
## Reporting
### Test Results Format
```
Test Suite: QA-AUTH-019
Total Tests: 18
Passed: 16 (89%)
Failed: 1
Skipped: 1
Test Suite: QA-APIKEY-020
Total Tests: 20
Passed: 18 (90%)
Failed: 1
Skipped: 1
Test Suite: QA-FILTER-021
Total Tests: 24
Passed: 20 (83%)
Failed: 2
Skipped: 2
Test Suite: QA-E2E-022
Total Tests: 15
Passed: 13 (87%)
Failed: 1
Skipped: 1
Overall Pass Rate: 85%
```
---
## Appendix: Test Data
### Test Users
- Email pattern: `user.{timestamp}@test.mockupaws.com`
- Password: `TestPassword123!`
- Full Name: `Test User {timestamp}`
### Test Scenarios
- Name pattern: `E2E Test {timestamp}`
- Regions: us-east-1, eu-west-1, ap-southeast-1, us-west-2, eu-central-1
- Status: draft, running, completed
### Test API Keys
- Name pattern: `Test API Key {purpose}`
- Scopes: read:scenarios, write:scenarios, read:reports
- Format: `mk_` + 32 random characters
---
*Document Version: 1.0*
*Last Updated: 2026-04-07*
*Prepared by: @qa-engineer*
Reference in New Issue View Git Blame Copy Permalink