Automation Framework

The automation framework built around standardized shell scripting patterns, declarative service management, and integrated monitoring. Provides consistent operations across all nodes while maintaining flexibility for node-specific requirements.

Framework Architecture

Bootstrap System

The automation framework centers around a universal bootstrap system that provides standardized dependencies and environment setup for all scripts:

1
2
# Universal bootstrap pattern used by all scripts
source "$(git -C "$(dirname "$0")" rev-parse --show-toplevel)/.scripts/bootstrap.sh"

Bootstrap Components:

Environment Setup: Automatic repository root detection and navigation
Library Loading: Standardized loading of all utility libraries
Error Handling: Consistent error trapping and notification patterns
Logging Framework: Unified logging with script identification and timestamps

Library Structure

The framework consists of modular libraries providing specialized functionality:

.scripts/
├── bootstrap.sh           # Universal dependency sourcing
├── lib/
│   ├── common.sh         # Core utilities and logging
│   ├── init.sh           # Declarative service initialization
│   ├── services.sh       # System service management
│   ├── templates.sh      # Configuration template rendering
│   └── validation.sh     # Input validation and system checks
└── ops/
    ├── check-package-updates    # Security update monitoring
    ├── envsubst-crontab        # Cron job template management
    ├── envsubst-directory      # Bulk template rendering
    ├── envsubst-ufw           # Firewall rule automation
    ├── lint-docker-compose    # Docker Compose validation
    ├── setup-tiers            # Data tier infrastructure setup
    ├── sync-remote            # Git repository synchronization
    └── sync-tiers             # Data backup synchronization

Core Library Functions

Common Utilities (`common.sh`)

Provides fundamental utilities used across all automation scripts:

Logging Framework

1
2
3
4
log_info "Starting operation..."
log_warn "Non-critical issue detected"
log_error "Critical error occurred"
log_success "Operation completed successfully"

Features:

Automatic script name detection for log prefixes
Consistent timestamp formatting
Error output to stderr
Integration with monitoring systems

Repository Navigation

1
2
navigate_to_repo_root        # Changes to repository root
REPO_ROOT                   # Global variable set by navigation

Capabilities:

Automatic git repository detection
Consistent working directory establishment
Error handling for non-git environments

Argument Validation

1
require_arg "NODE" "$NODE" "Usage: $0 <NODE> <TELEGRAM_URL>"

Validation Features:

Required argument checking with descriptive errors
Usage hint integration
Standardized exit codes for different error types

Template Rendering

1
render_template "input.template" "output.conf"

Template System:

Environment variable substitution via envsubst
Atomic file operations with temporary files
Error handling and rollback on failures

Telegram Integration

1
2
3
4
5
setup_error_trap "Operation Name" "$TELEGRAM_BOT_URL" "$NODE"
telegram_success "Operation Name" "$TELEGRAM_BOT_URL" "Success message" "$NODE"
telegram_error "Operation Name" "$TELEGRAM_BOT_URL" "Error details" "$NODE"
telegram_info "Operation Name" "$TELEGRAM_BOT_URL" "Information" "$NODE"
telegram_warn "Operation Name" "$TELEGRAM_BOT_URL" "Warning message" "$NODE"

Notification Features:

Automatic error trapping with notifications
Rich message formatting with Markdown support
Node identification for multi-host environments
Timestamp inclusion for audit trails

Service Management (`services.sh`)

Provides standardized systemd service management functions:

Service Status Operations

1
2
3
service_exists "docker"              # Check service existence
service_is_active "nginx"           # Check if service is running
service_is_enabled "ssh"            # Check if service is enabled

Service Control Operations

1
2
3
4
start_and_enable_service "docker"   # Start and enable service
stop_and_disable_service "apache2"  # Stop and disable service
restart_service "nginx"             # Restart service
reload_service "systemd-resolved"   # Reload configuration

Service Monitoring

1
2
get_service_status "docker"         # Detailed status information
wait_for_service "mysql" 60 2       # Wait for service with timeout

Validation Framework (`validation.sh`)

Comprehensive input and system validation:

File System Validation

1
2
validate_file_exists "/etc/passwd" "System password file"
validate_directory_exists "/var/log" "System log directory"

System Validation

1
2
3
4
5
validate_command_exists "docker" "Container runtime"
validate_env_vars "NODE" "TELEGRAM_BOT_URL" "DOMAIN"
validate_user "root"
validate_sudo
validate_architecture "x86_64"

Network and Resource Validation

1
2
3
4
validate_network_connectivity "google.com" 443 5
validate_disk_space "/var" 1024    # 1GB minimum
validate_number_range "$PORT" 1024 65535 "Port number"
validate_pattern "$EMAIL" "^[^@]+@[^@]+\.[^@]+$" "Email address"

Declarative Initialization (`init.sh`)

Provides a declarative approach to service initialization:

Service Initialization Pattern

1
init_service "docker" "docker.io docker-compose" custom_docker_setup

Initialization Steps:

Package Installation: Automatic installation of required packages
Environment Setup: Node and tier environment variable configuration
Custom Setup: Execution of service-specific configuration functions
Service Management: Automatic service start and enable
Health Validation: Comprehensive health checks with failure handling

Base System Setup

1
init_base_system    # Standardized base system initialization

Base System Components:

Debian/Ubuntu system validation
Package list updates
Essential package installation (curl, wget, git, rsyslog)
System logging service configuration

Template Management (`templates.sh`)

Advanced template rendering capabilities:

Single File Rendering

1
render_template "nginx.conf.template" "nginx.conf"

Batch Directory Rendering

1
render_template_directory "templates/" "output/"

Variable-Specific Rendering

1
render_template_with_vars "config.template" "config.conf" "variables.env"

Template Features:

Atomic file operations with temporary files
Environment variable validation
Batch processing with directory preservation
External variable file support

Operational Scripts

Infrastructure Setup

Tier Setup (`setup-tiers`)

1
.scripts/ops/setup-tiers <node> <tier1_path> <tier2_path> <tier3_path>

Operations Performed:

Creates complete @tier symbolic link structure
Establishes project-level tier directories
Sets up host-level .host/@tier links
Creates shared directory hierarchy with cross-links
Links .env files to @tier1 for configuration management

Interactive Safety Features:

Confirmation prompts for existing symlink removal
Validation of target directory existence
Safe handling of non-symlink paths
Comprehensive logging of all operations

Data Synchronization (`sync-tiers`)

1
.scripts/ops/sync-tiers <NODE> <TELEGRAM_BOT_URL>

Synchronization Process:

Validation: Confirms node directory and compose file existence
Tier 1 Sync: Executes rclone1 service for critical data
Tier 2 Sync: Executes rclone2 service for application data
Error Handling: Automatic error trapping with Telegram notifications
Success Reporting: Completion notifications with operation details

Configuration Management

Template Directory Processing (`envsubst-directory`)

1
TEMPLATE_VAR=value .scripts/ops/envsubst-directory ./templates ./output

Processing Features:

Recursive template file discovery
Directory structure preservation
Existing file backup to /tmp
Environment variable substitution
Atomic file operations

Cron Job Management (`envsubst-crontab`)

1
NODE=icarus TIER1=/mnt/tier1 .scripts/ops/envsubst-crontab ./cron.template

Crontab Features:

Environment variable substitution in cron templates
Current crontab backup before installation
Validation of rendered crontab before installation
Automatic rollback on installation failures
Preview of rendered crontab before installation

Firewall Rule Management (`envsubst-ufw`)

1
.scripts/ops/envsubst-ufw ./firewall-rules ./host.rules

Firewall Management:

Docker service firewall rules via ufw-docker
Host-level UFW rules for system protection
Rule file processing with comment and blank line handling
Idempotent rule application
Error handling with rule validation

Development and Maintenance

Docker Compose Linting (`lint-docker-compose`)

1
2
.scripts/ops/lint-docker-compose            # Lint all compose files
.scripts/ops/lint-docker-compose --hook     # Pre-commit hook mode

Linting Features:

Automatic fixing of common Docker Compose issues
Pre-commit hook integration with staged file processing
Exclusion of data directories from linting (data, state, vault)
Automatic re-staging of fixed files in hook mode
Integration with development workflows

Package Update Monitoring (`check-package-updates`)

1
.scripts/ops/check-package-updates <NODE> <TELEGRAM_BOT_URL>

Update Monitoring:

System package list updates
Available upgrade detection and reporting
Telegram notifications with package details
Security update prioritization
Formatted update notifications with package counts

Repository Synchronization (`sync-remote`)

1
.scripts/ops/sync-remote <NODE> <TELEGRAM_BOT_URL>

Git Operations:

Automatic staging of node directory changes
Branch creation and management (sync/<NODE> pattern)
Rebase integration with origin/trunk
Conflict detection and handling
Force-push with lease protection
Bot identity configuration for automated commits

Error Handling and Monitoring

Universal Error Trapping

All scripts implement standardized error handling:

1
setup_error_trap "Operation Description" "$TELEGRAM_BOT_URL" "$NODE"

Error Handling Features:

Automatic error detection via set -euo pipefail
Context-aware error messages with line numbers and commands
Immediate Telegram notifications on script failures
Graceful cleanup of temporary resources
Integration with monitoring and alerting systems

Monitoring Integration

The framework integrates comprehensive monitoring:

Health Check Integration

Service startup validation
Database integrity checks
Network connectivity verification
Resource availability monitoring

Alert Classification

Success: Operation completion notifications
Info: Informational updates and status reports
Warning: Non-critical issues requiring attention
Error: Critical failures requiring immediate intervention

Logging and Audit Trail

Structured Logging

1
2
3
[script_name] [INFO] 2024-01-20 15:30:45 UTC Operation started
[script_name] [WARN] 2024-01-20 15:30:50 UTC Configuration issue detected
[script_name] [SUCCESS] 2024-01-20 15:31:00 UTC Operation completed

Audit Features

Consistent log formatting across all scripts
Timestamp integration with timezone information
Operation correlation through script name identification
Integration with centralized logging systems

Framework Best Practices

Script Development Standards

Universal Bootstrap: All scripts must source the bootstrap system
Error Handling: Implement error trapping for all critical operations
Argument Validation: Validate all required arguments before execution
Logging Integration: Use standardized logging functions throughout
Documentation: Include usage instructions and parameter descriptions

Security Considerations

Privilege Validation: Verify required permissions before execution
Input Sanitization: Validate and sanitize all external inputs
Secret Handling: Use secure methods for credential management
Audit Logging: Maintain comprehensive logs for security auditing
Error Information: Avoid exposing sensitive data in error messages

Operational Reliability

Idempotent Operations: Ensure scripts can be safely re-executed
Atomic Changes: Use temporary files and atomic moves for critical updates
Rollback Capability: Implement rollback mechanisms for configuration changes
Health Validation: Verify system state after all operations
Resource Cleanup: Ensure proper cleanup of temporary resources

This automation framework provides a robust foundation for system management while maintaining consistency, security, and operational reliability.

Last updated on October 2, 2025

Operational Procedures