Skip to content

Phase 3: Infrastructure Separation - Summary

Completed Work

Phase 3A: Monitoring Stack Separation ✅

Date: April 9, 2026
Duration: ~3 hours
Status: Complete and operational

What Was Done

  • Created independent monitoring stack at /srv/infra/
  • Migrated 9 monitoring services from logkeep-* to monitoring-* namespace
  • Moved 10.6GB of historical data from Docker volumes to /srv/infra/data/monitoring/
  • Created dedicated monitoring-network isolated from application networks
  • Updated all service configurations for new container names
  • Connected Prometheus to both monitoring and application networks for scraping

Services Migrated

  1. monitoring-prometheus - 30-day metric retention, 1.6GB data
  2. monitoring-grafana - Dashboards and visualization, 42MB data
  3. monitoring-loki - Log aggregation, 9GB data
  4. monitoring-alertmanager - Email alert routing, 21 alert rules
  5. monitoring-promtail - Log shipping to Loki
  6. monitoring-node-exporter - System metrics
  7. monitoring-cadvisor - Container metrics
  8. monitoring-blackbox-exporter - SSL certificate monitoring

Results

  • All services healthy and operational
  • 10/13 Prometheus targets up (3 expected down: headscale, logkeep-blue, logkeep-green)
  • Grafana dashboards fully functional with historical data
  • Alert rules loaded and Alertmanager configured
  • Monitoring continues even when application containers stop (verified)
  • Old logkeep-* monitoring containers removed

Files Created

  • /srv/infra/docker-compose.monitoring.yml - Monitoring stack definition
  • /srv/infra/configs/ - All monitoring configurations centralized
  • /srv/infra/data/monitoring/ - Persistent data storage

Phase 3B: Backup System Documentation ✅

Date: April 9, 2026
Status: Documented, implementation deferred

What Was Done

  • Analyzed existing backup system (/srv/backups/backup-databases.sh)
  • Documented current state: 3 databases, daily backups, 7-day retention, remote sync to pyrite
  • Proposed comprehensive improvements (monitoring, testing, rotation)
  • Deferred implementation as current system is working reliably

Documentation Created

  • /srv/infra/docs/backup-improvements.md - Detailed improvement plan

Phase 3C: Nginx Configuration Centralization ✅

Date: April 9, 2026
Status: Complete

What Was Done

  • Copied all nginx site configs to /srv/infra/configs/nginx/sites-available/
  • Created comprehensive nginx management documentation
  • Documented SSL certificate renewal process
  • Prepared for version control integration

Configs Centralized

  1. bench.perdrizet.org - Bench application (→ 127.0.0.1:8010)
  2. blue.conf - LogKeep blue slot
  3. green.conf - LogKeep green slot
  4. staging.conf - LogKeep staging
  5. headscale - Headscale VPN admin
  6. gpt.conf - GPT/LLM service
  7. matrix.perdrizet.org - Matrix server (historical)
  8. prism.perdrizet.org - Prism service (historical)

Documentation Created

  • /srv/infra/configs/nginx/README.md - Complete nginx management guide
  • /srv/infra/docs/infrastructure-layout.md - Full VPS layout documentation

Infrastructure State After Phase 3

Directory Structure

/srv/infra/                          # Central infrastructure
├── configs/                         # 132KB - All configurations
│   ├── prometheus.yml
│   ├── alert-rules.yml
│   ├── grafana-datasources.yml
│   ├── grafana-dashboards/
│   ├── loki-config.yml
│   ├── promtail-config.yml
│   ├── alertmanager.yml
│   ├── blackbox.yml
│   └── nginx/                       # NEW
│       ├── README.md
│       └── sites-available/         # 8 site configs
├── data/                            # 12GB - Persistent data
│   └── monitoring/
│       ├── prometheus/  (1.6GB)
│       ├── grafana/     (42MB)
│       ├── loki/        (9GB)
│       └── alertmanager/ (4KB)
├── docs/                            # 16KB - Documentation
│   ├── backup-improvements.md
│   └── infrastructure-layout.md
├── logs/                            # (empty, ready for use)
├── scripts/                         # (ready for management scripts)
└── docker-compose.monitoring.yml    # 8KB - Monitoring stack

Services Status

  • Monitoring: 8 containers, all healthy, fully independent
  • Applications: 9 containers (logkeep, bench), operational
  • Nginx: Systemd service, configs centralized, SSL managed by certbot
  • Backups: Daily at 2 AM, working reliably, syncing to pyrite

Networks

  • monitoring-network - Independent monitoring stack
  • logkeep_logkeep-network - LogKeep applications
  • docker_default - Bench applications
  • Prometheus bridges networks for metric scraping

Next Steps (Phase 3D - Future)

Not Yet Started

  1. Shared Services Organization
  2. Consider separating PostgreSQL to /srv/infra/docker-compose.databases.yml
  3. Redis organization for shared caching

  4. Evaluate consolidating postgres containers

  5. Application Organization

  6. Move /opt/logkeep/ to /srv/apps/logkeep/
  7. Move /opt/bench/ to /srv/apps/bench/

  8. Separate application docker-compose files

  9. Network Simplification

  10. Reduce number of Docker networks
  11. Create unified app network strategy

  12. Document network architecture

  13. Nginx Optimization

  14. Review and optimize site configs
  15. Add security headers uniformly
  16. Implement rate limiting
  17. Consider HTTP/3 support

Metrics & Achievements

Data Preserved

  • 10.6GB of monitoring data migrated successfully
  • 30 days of Prometheus metrics retained
  • 0 data loss during migration
  • All Grafana dashboards functional with history

Reliability Improvements

  • Monitoring no longer coupled to application services
  • Independent failure domains (monitoring vs apps)
  • Centralized configuration management
  • All infrastructure configs now in /srv/infra/

Documentation Created

  • 3 comprehensive markdown documents
  • Complete nginx management guide
  • Full infrastructure layout documented
  • Backup improvement roadmap

Time Investment

  • Phase 3A (Monitoring): ~3 hours
  • Phase 3B (Backups): ~30 minutes (documentation only)
  • Phase 3C (Nginx): ~1 hour
  • Total: ~4.5 hours

Lessons Learned

  1. Container Naming - Consistent prefixes (monitoring-*) make management easier
  2. Volume Migration - Using temporary Alpine containers for volume copy is fast and reliable
  3. Network Strategy - Independent networks + selective bridging works well
  4. Configuration Centralization - Having all configs in one place simplifies backup and version control
  5. Incremental Migration - Migrating piece by piece reduces risk and allows testing
  1. /srv/infra/docs/infrastructure-layout.md - Overview of entire VPS
  2. /srv/infra/configs/nginx/README.md - Nginx management
  3. /srv/infra/docs/backup-improvements.md - Backup system enhancements
  4. This file (phase3-summary.md) - What was accomplished

Last Updated: April 9, 2026
Status: Phase 3A-C Complete, Phase 3D Pending