mockupAWS/docs/DATA-ARCHIVING.md

Data Archiving Strategy

mockupAWS v1.0.0 - Data Lifecycle Management

---

Table of Contents

1. Overview
2. Archive Policies
3. Implementation
4. Archive Job
5. Querying Archived Data
6. Monitoring
7. Storage Estimation

---

Overview

As mockupAWS accumulates data over time, we implement an automated archiving strategy to:

• Reduce storage costs by moving old data to archive tables
• Improve query performance on active data
• Maintain data accessibility through unified views
• Comply with data retention policies

Archive Strategy Overview

┌─────────────────────────────────────────────────────────────────┐
│                     Data Lifecycle                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Active Data (Hot)    │    Archive Data (Cold)                  │
│  ─────────────────    │    ──────────────────                   │
│  • Fast queries       │    • Partitioned by month               │
│  • Full indexing      │    • Compressed                         │
│  • Real-time writes   │    • S3 for large files                 │
│                                                                 │
│  scenario_logs        │    → scenario_logs_archive              │
│  (> 1 year old)       │    (> 1 year, partitioned)              │
│                                                                 │
│  scenario_metrics     │    → scenario_metrics_archive           │
│  (> 2 years old)      │    (> 2 years, aggregated)              │
│                                                                 │
│  reports              │    → reports_archive                    │
│  (> 6 months old)     │    (> 6 months, S3 storage)             │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

---

Archive Policies

Policy Configuration

Table	Archive After	Aggregation	Compression	S3 Storage
scenario_logs	365 days	No	No	No
scenario_metrics	730 days	Daily	No	No
reports	180 days	No	Yes	Yes

Detailed Policies

1. Scenario Logs Archive (> 1 year)

Criteria:

• Records older than 365 days
• Move to scenario_logs_archive table
• Partitioned by month for efficient querying

Retention:

• Archive table: 7 years
• After 7 years: Delete or move to long-term storage

2. Scenario Metrics Archive (> 2 years)

Criteria:

• Records older than 730 days
• Aggregate to daily values before archiving
• Store aggregated data in scenario_metrics_archive

Aggregation:

• Group by: scenario_id, metric_type, metric_name, day
• Aggregate: AVG(value), COUNT(samples)

Retention:

• Archive table: 5 years
• Aggregated data only (original samples deleted)

3. Reports Archive (> 6 months)

Criteria:

• Reports older than 180 days
• Compress PDF/CSV files
• Upload to S3
• Keep metadata in reports_archive table

Retention:

• S3 storage: 3 years with lifecycle to Glacier
• Metadata: 5 years

---

Implementation

Database Schema

Archive Tables

-- Scenario logs archive (partitioned by month)
CREATE TABLE scenario_logs_archive (
    id UUID PRIMARY KEY,
    scenario_id UUID NOT NULL,
    received_at TIMESTAMPTZ NOT NULL,
    message_hash VARCHAR(64) NOT NULL,
    message_preview VARCHAR(500),
    source VARCHAR(100) NOT NULL,
    size_bytes INTEGER NOT NULL,
    has_pii BOOLEAN NOT NULL,
    token_count INTEGER NOT NULL,
    sqs_blocks INTEGER NOT NULL,
    archived_at TIMESTAMPTZ DEFAULT NOW(),
    archive_batch_id UUID
) PARTITION BY RANGE (DATE_TRUNC('month', received_at));

-- Scenario metrics archive (with aggregation support)
CREATE TABLE scenario_metrics_archive (
    id UUID PRIMARY KEY,
    scenario_id UUID NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    metric_type VARCHAR(50) NOT NULL,
    metric_name VARCHAR(100) NOT NULL,
    value DECIMAL(15,6) NOT NULL,
    unit VARCHAR(20) NOT NULL,
    extra_data JSONB DEFAULT '{}',
    archived_at TIMESTAMPTZ DEFAULT NOW(),
    archive_batch_id UUID,
    is_aggregated BOOLEAN DEFAULT FALSE,
    aggregation_period VARCHAR(20),
    sample_count INTEGER
) PARTITION BY RANGE (DATE_TRUNC('month', timestamp));

-- Reports archive (S3 references)
CREATE TABLE reports_archive (
    id UUID PRIMARY KEY,
    scenario_id UUID NOT NULL,
    format VARCHAR(10) NOT NULL,
    file_path VARCHAR(500) NOT NULL,
    file_size_bytes INTEGER,
    generated_by VARCHAR(100),
    extra_data JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL,
    archived_at TIMESTAMPTZ DEFAULT NOW(),
    s3_location VARCHAR(500),
    deleted_locally BOOLEAN DEFAULT FALSE,
    archive_batch_id UUID
);

Unified Views (Query Transparency)

-- View combining live and archived logs
CREATE VIEW v_scenario_logs_all AS
SELECT 
    id, scenario_id, received_at, message_hash, message_preview,
    source, size_bytes, has_pii, token_count, sqs_blocks,
    NULL::timestamptz as archived_at,
    false as is_archived
FROM scenario_logs
UNION ALL
SELECT 
    id, scenario_id, received_at, message_hash, message_preview,
    source, size_bytes, has_pii, token_count, sqs_blocks,
    archived_at,
    true as is_archived
FROM scenario_logs_archive;

-- View combining live and archived metrics
CREATE VIEW v_scenario_metrics_all AS
SELECT 
    id, scenario_id, timestamp, metric_type, metric_name,
    value, unit, extra_data,
    NULL::timestamptz as archived_at,
    false as is_aggregated,
    false as is_archived
FROM scenario_metrics
UNION ALL
SELECT 
    id, scenario_id, timestamp, metric_type, metric_name,
    value, unit, extra_data,
    archived_at,
    is_aggregated,
    true as is_archived
FROM scenario_metrics_archive;

Archive Job Tracking

-- Archive jobs table
CREATE TABLE archive_jobs (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    job_type VARCHAR(50) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'pending',
    started_at TIMESTAMPTZ,
    completed_at TIMESTAMPTZ,
    records_processed INTEGER DEFAULT 0,
    records_archived INTEGER DEFAULT 0,
    records_deleted INTEGER DEFAULT 0,
    bytes_archived BIGINT DEFAULT 0,
    error_message TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Archive statistics view
CREATE VIEW v_archive_statistics AS
SELECT 
    'logs' as archive_type,
    COUNT(*) as total_records,
    MIN(received_at) as oldest_record,
    MAX(received_at) as newest_record,
    SUM(size_bytes) as total_bytes
FROM scenario_logs_archive
UNION ALL
SELECT 
    'metrics' as archive_type,
    COUNT(*) as total_records,
    MIN(timestamp) as oldest_record,
    MAX(timestamp) as newest_record,
    0 as total_bytes
FROM scenario_metrics_archive
UNION ALL
SELECT 
    'reports' as archive_type,
    COUNT(*) as total_records,
    MIN(created_at) as oldest_record,
    MAX(created_at) as newest_record,
    SUM(file_size_bytes) as total_bytes
FROM reports_archive;

---

Archive Job

Running the Archive Job

# Preview what would be archived (dry run)
python scripts/archive_job.py --dry-run --all

# Archive all eligible data
python scripts/archive_job.py --all

# Archive specific types only
python scripts/archive_job.py --logs
python scripts/archive_job.py --metrics
python scripts/archive_job.py --reports

# Combine options
python scripts/archive_job.py --logs --metrics --dry-run

Cron Configuration

# Run archive job nightly at 3:00 AM
0 3 * * * /opt/mockupaws/.venv/bin/python /opt/mockupaws/scripts/archive_job.py --all >> /var/log/mockupaws/archive.log 2>&1

Environment Variables

# Required
export DATABASE_URL="postgresql+asyncpg://user:pass@host:5432/mockupaws"

# For reports S3 archiving
export REPORTS_ARCHIVE_BUCKET="mockupaws-reports-archive"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

---

Querying Archived Data

Transparent Access

Use the unified views for automatic access to both live and archived data:

-- Query all logs (live + archived)
SELECT * FROM v_scenario_logs_all 
WHERE scenario_id = 'uuid-here'
ORDER BY received_at DESC
LIMIT 1000;

-- Query all metrics (live + archived)
SELECT * FROM v_scenario_metrics_all 
WHERE scenario_id = 'uuid-here'
  AND timestamp > NOW() - INTERVAL '2 years'
ORDER BY timestamp;

Optimized Queries

-- Query only live data (faster)
SELECT * FROM scenario_logs 
WHERE scenario_id = 'uuid-here'
ORDER BY received_at DESC;

-- Query only archived data
SELECT * FROM scenario_logs_archive 
WHERE scenario_id = 'uuid-here'
  AND received_at < NOW() - INTERVAL '1 year'
ORDER BY received_at DESC;

-- Query specific month partition (most efficient)
SELECT * FROM scenario_logs_archive 
WHERE received_at >= '2025-01-01' 
  AND received_at < '2025-02-01'
  AND scenario_id = 'uuid-here';

Application Code Example

from sqlalchemy import select
from src.models.scenario_log import ScenarioLog

async def get_logs(db: AsyncSession, scenario_id: UUID, include_archived: bool = False):
    """Get scenario logs with optional archive inclusion."""
    
    if include_archived:
        # Use unified view for complete history
        result = await db.execute(
            text("""
                SELECT * FROM v_scenario_logs_all 
                WHERE scenario_id = :sid
                ORDER BY received_at DESC
            """),
            {"sid": scenario_id}
        )
    else:
        # Query only live data (faster)
        result = await db.execute(
            select(ScenarioLog)
            .where(ScenarioLog.scenario_id == scenario_id)
            .order_by(ScenarioLog.received_at.desc())
        )
    
    return result.scalars().all()

---

Monitoring

Archive Job Status

-- Check recent archive jobs
SELECT 
    job_type,
    status,
    started_at,
    completed_at,
    records_archived,
    records_deleted,
    pg_size_pretty(bytes_archived) as space_saved
FROM archive_jobs
ORDER BY started_at DESC
LIMIT 10;

-- Check for failed jobs
SELECT * FROM archive_jobs 
WHERE status = 'failed'
ORDER BY started_at DESC;

Archive Statistics

-- View archive statistics
SELECT * FROM v_archive_statistics;

-- Archive growth over time
SELECT 
    DATE_TRUNC('month', archived_at) as archive_month,
    archive_type,
    COUNT(*) as records_archived,
    pg_size_pretty(SUM(total_bytes)) as bytes_archived
FROM v_archive_statistics
GROUP BY DATE_TRUNC('month', archived_at), archive_type
ORDER BY archive_month DESC;

Alerts

# archive-alerts.yml
groups:
  - name: archive_alerts
    rules:
      - alert: ArchiveJobFailed
        expr: increase(archive_job_failures_total[1h]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Data archive job failed"
          
      - alert: ArchiveJobNotRunning
        expr: time() - max(archive_job_last_success_timestamp) > 90000
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Archive job has not run in 25 hours"
          
      - alert: ArchiveStorageGrowing
        expr: rate(archive_bytes_total[1d]) > 1073741824  # 1GB/day
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Archive storage growing rapidly"

---

Storage Estimation

Projected Storage Savings

Assuming typical usage patterns:

Data Type	Daily Volume	Annual Volume	After Archive	Savings
Logs	1M records/day	365M records	365M in archive	0 in main
Metrics	500K records/day	182M records	60M aggregated	66% reduction
Reports	100/day (50MB each)	1.8TB	1.8TB in S3	100% local reduction

Cost Analysis (Monthly)

Storage Type	Before Archive	After Archive	Monthly Savings
PostgreSQL (hot)	$200	$50	$150
PostgreSQL (archive)	$0	$30	-$30
S3 Standard	$0	$20	-$20
S3 Glacier	$0	$5	-$5
Total	$200	$105	$95

Estimates based on AWS us-east-1 pricing, actual costs may vary.

---

Maintenance

Monthly Tasks

1. Review archive statistics

   SELECT * FROM v_archive_statistics;
   

2. Check for old archive partitions

   SELECT 
       schemaname, 
       tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
   FROM pg_tables
   WHERE tablename LIKE 'scenario_logs_archive_%'
   ORDER BY tablename;
   

3. Clean up old S3 files (after retention period)

   aws s3 rm s3://mockupaws-reports-archive/archived-reports/ \
     --recursive \
     --exclude '*' \
     --include '*2023*'
   

Quarterly Tasks

1. Archive job performance review

   • Check execution times
   • Optimize batch sizes if needed

2. Storage cost review

   • Verify S3 lifecycle policies
   • Consider Glacier transition for old archives

3. Data retention compliance

   • Verify deletion of data past retention period
   • Update policies as needed

---

Troubleshooting

Archive Job Fails

# Check logs
tail -f storage/logs/archive_*.log

# Run with verbose output
python scripts/archive_job.py --all --verbose

# Check database connectivity
psql $DATABASE_URL -c "SELECT COUNT(*) FROM archive_jobs;"

S3 Upload Fails

# Verify AWS credentials
aws sts get-caller-identity

# Test S3 access
aws s3 ls s3://mockupaws-reports-archive/

# Check bucket policy
aws s3api get-bucket-policy --bucket mockupaws-reports-archive

Query Performance Issues

-- Check if indexes exist on archive tables
SELECT indexname, indexdef 
FROM pg_indexes 
WHERE tablename LIKE '%_archive%';

-- Analyze archive tables
ANALYZE scenario_logs_archive;
ANALYZE scenario_metrics_archive;

-- Check partition pruning
EXPLAIN ANALYZE 
SELECT * FROM scenario_logs_archive 
WHERE received_at >= '2025-01-01' 
  AND received_at < '2025-02-01';

---

References

• PostgreSQL Table Partitioning
• AWS S3 Lifecycle Policies
• Database Migration
• Archive Job Script

---

Document Version: 1.0.0 Last Updated: 2026-04-07