feat(ingestion): implement log monitoring script with webhook integration

- Add logwhisperer.sh script for tailing and monitoring system logs
- Implement pattern matching for critical errors (FATAL, ERROR, OOM, segfault)
- Add JSON payload generation with severity levels
- Implement rate limiting and offset tracking per log source
- Add install.sh with interactive configuration and systemd support
- Create comprehensive test suite with pytest
- Add technical specification documentation
- Update CHANGELOG.md following Common Changelog standard

All 12 tests passing. Follows Metodo Sacchi (Safety first, little often, double check).
This commit is contained in:
Luca Sacchi Ricciardi
2026-04-02 16:09:00 +02:00
parent 34dbba1201
commit 69f475ec78
6 changed files with 1148 additions and 0 deletions

51
CHANGELOG.md Normal file
View File

@@ -0,0 +1,51 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Common Changelog](https://common-changelog.org/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.1.0] - 2026-04-02
### Added
- feat: Implement log ingestion script (`logwhisperer.sh`) for monitoring system logs
- Monitor multiple log sources: syslog, nginx, postgresql
- Pattern matching for critical errors (FATAL, ERROR, OOM, segfault, disk full)
- JSON payload generation with severity levels (low, medium, critical)
- Rate limiting to prevent alert flooding (30s per source/pattern)
- Offset tracking for each log file to avoid reprocessing
- HTTP POST dispatch to configurable webhook with retry logic
- Dry-run mode for testing pattern matching without sending webhooks
- Configuration file support (`/etc/logwhisperer/config.env`)
- Command-line flags: `--help`, `--validate`, `--config`, `--dry-run`, `--test-line`
- feat: Create installation script (`install.sh`)
- Interactive configuration wizard
- UUID v4 generation for CLIENT_ID
- Systemd service creation (when run as root)
- Support for both system-wide and user-local installation
- Prerequisite checking (bash, curl)
- Connectivity test to webhook URL
- test: Add comprehensive test suite (`tests/test_logwhisperer.py`)
- Script existence and executable validation
- Configuration validation tests
- Pattern matching tests (FATAL, OOM, ERROR patterns)
- JSON payload structure validation
- Severity mapping verification
- docs: Create technical specification for Feature 1 (Log Ingestion)
- Architecture diagram and component description
- Requirements (functional and non-functional)
- Safety guidelines (Metodo Sacchi)
- Acceptance criteria
### Security
- Configuration files created with restrictive permissions (600)
- No hardcoded credentials in scripts
- HTTPS validation for webhook URLs (warning for non-HTTPS)
- Read-only access to log files (no modifications)