Concepts
Automation Scripts

Automation Scripts

Reference for the automation framework in .scripts/. These scripts handle setup, configuration, and maintenance across all nodes.

Bootstrap Framework

bootstrap.sh

Purpose: Universal dependency sourcing and environment setup
Location: .scripts/bootstrap.sh

Usage Pattern:

1
2

# Standard bootstrap pattern used by all scripts
source "$(git -C "$(dirname "$0")" rev-parse --show-toplevel)/.scripts/bootstrap.sh"

Functions Provided:

  • Automatic repository root detection and navigation
  • Library loading (common.sh, validation.sh, templates.sh, services.sh, init.sh)
  • Environment variable setup (SCRIPTS_ROOT)
  • Consistent error handling preparation

Integration: All automation scripts must source the bootstrap system as their first operation to ensure consistent environment and library availability.

Core Library Functions

common.sh

Purpose: Core utilities, logging, and fundamental operations
Location: .scripts/lib/common.sh

Logging Functions 

1
2
3
4

log_info "Operation started"           # Informational messages
log_warn "Configuration issue"         # Non-critical warnings  
log_error "Critical failure"           # Error messages (to stderr)
log_success "Operation completed"      # Success confirmations

Repository Navigation 

1
2
3

navigate_to_repo_root                  # Navigate to git repository root
navigate_to_repo_root "nodename"       # Navigate with node context
# Sets global variable: REPO_ROOT

Argument Validation 

1
2

require_arg "PARAMETER" "$VALUE" "Usage: script <param>"
# Exits with code 64 if parameter is empty

Template Rendering 

1
2
3

render_template "input.template" "output.conf"
# Uses envsubst for environment variable substitution
# Atomic file operations with temporary files

Telegram Integration 

1
2
3
4
5
6
7
8

# Error trap setup
setup_error_trap "Operation Name" "$TELEGRAM_BOT_URL" "$NODE"

# Manual notifications
telegram_success "Title" "$BOT_URL" "Message" "$NODE"
telegram_error "Title" "$BOT_URL" "Error details" "$NODE" 
telegram_info "Title" "$BOT_URL" "Information" "$NODE"
telegram_warn "Title" "$BOT_URL" "Warning" "$NODE"

Utility Functions 

1
2

get_timestamp                          # Returns standardized timestamp
ensure_debian_ubuntu                   # Validates OS compatibility

services.sh

Purpose: Systemd service management utilities
Location: .scripts/lib/services.sh

Service Status Functions 

1
2
3

service_exists "docker"                # Check if service unit exists
service_is_active "nginx"             # Check if service is running
service_is_enabled "ssh"              # Check if service is enabled

Service Control Functions 

1
2
3
4

start_and_enable_service "docker"     # Start and enable service
stop_and_disable_service "apache2"    # Stop and disable service  
restart_service "nginx"               # Restart service
reload_service "systemd-resolved"     # Reload configuration

Service Monitoring Functions 

1
2

get_service_status "docker"           # Detailed status information
wait_for_service "mysql" 60 2         # Wait with timeout and interval

validation.sh

Purpose: Input validation and system verification
Location: .scripts/lib/validation.sh

File System Validation 

1
2

validate_file_exists "/path/file" "Description"
validate_directory_exists "/path/dir" "Description"

System Validation 

1
2
3
4
5

validate_command_exists "docker" "Container runtime"
validate_env_vars "VAR1" "VAR2" "VAR3"     # Multiple variables
validate_user "root"                        # Current user check
validate_sudo                              # Sudo privileges check
validate_architecture "x86_64"             # System architecture

Network and Resource Validation 

1
2
3
4

validate_network_connectivity "host" port timeout
validate_disk_space "/path" required_mb "Description"
validate_number_range "$VALUE" min max "Description"
validate_pattern "$VALUE" "regex" "Description"

templates.sh

Purpose: Configuration template rendering
Location: .scripts/lib/templates.sh

Template Rendering Functions 

1
2
3

render_template "input.template" "output.conf"
render_template_directory "templates/" "output/"
render_template_with_vars "template" "output" "vars.env"

Features:

  • Environment variable substitution via envsubst
  • Atomic file operations with .tmp files
  • Batch directory processing with structure preservation
  • External variable file support

init.sh

Purpose: Declarative service initialization framework
Location: .scripts/lib/init.sh

Core Initialization Functions 

1
2
3
4
5

init_service "service_name" "packages" "custom_function"
# Performs: package installation, environment setup, custom config, 
#          service management, health validation

init_base_system                       # Base system initialization

Helper Functions 

1
2
3
4

install_packages package1 package2    # Install required packages
ensure_service_running "service"      # Start and enable service
setup_node_host_environment          # Set NODE, TIER1/2/3, SCRIPTS vars
health_check_service "service"       # Validate service health

Operational Scripts

Infrastructure Management

setup-tiers

Purpose: Complete data tier infrastructure setup
Location: .scripts/ops/setup-tiers

Usage:

1
2
3
4

.scripts/ops/setup-tiers <node> <tier1_path> <tier2_path> <tier3_path>

# Example:
.scripts/ops/setup-tiers icarus /mnt/ssd/tier1 /mnt/hdd/tier2 /mnt/bulk/tier3

Operations Performed:

  1. Directory Creation: Ensures all tier base directories exist
  2. Project Discovery: Automatically discovers all projects in node directory
  3. Symbolic Link Creation: Creates @tier1, @tier2, @tier3 links for each project
  4. Environment File Linking: Links .env files to @tier1 for configuration
  5. Host Link Creation: Creates .host/@tier links for system-level access
  6. Shared Structure Setup: Establishes shared directory hierarchy with cross-links

Safety Features:

  • Interactive confirmation for existing symlink replacement
  • Validation of target directory existence and permissions
  • Comprehensive logging of all operations
  • Safe handling of non-symlink existing paths

sync-tiers

Purpose: Automated data tier synchronization and backup
Location: .scripts/ops/sync-tiers

Usage:

1
2
3
4

.scripts/ops/sync-tiers <NODE> <TELEGRAM_BOT_URL>

# Example:
.scripts/ops/sync-tiers icarus https://api.telegram.org/bot<token>/sendMessage

Process Flow:

  1. Validation Phase: Confirms node directory and docker-compose.yml existence
  2. Tier 1 Sync: Executes rclone1 Docker service for critical configuration data
  3. Tier 2 Sync: Executes rclone2 Docker service for application data
  4. Error Handling: Automatic error trapping with detailed Telegram notifications
  5. Success Reporting: Completion confirmation with operation summary

Docker Compose Integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Required services in node/.host/docker-compose.yml
services:
  rclone1:
    image: rclone/rclone:latest
    volumes:
      - ./@tier1:/data/tier1
      - ./rclone/config:/home/rclone/.config/rclone:ro
    command: sync /data/tier1 remote:backup/tier1
    
  rclone2:
    image: rclone/rclone:latest
    volumes:
      - ./@tier2:/data/tier2  
      - ./rclone/config:/home/rclone/.config/rclone:ro
    command: sync /data/tier2 remote:backup/tier2

Configuration Management

envsubst-directory

Purpose: Batch template rendering for entire directories
Location: .scripts/ops/envsubst-directory

Usage:

1
2
3
4

[ENV_VAR=value...] .scripts/ops/envsubst-directory <source_dir> <dest_dir>

# Example:
SERVICE_NAME=nginx PORT=80 .scripts/ops/envsubst-directory ./templates ./config

Features:

  • Recursive Processing: Discovers all files in source directory recursively
  • Structure Preservation: Maintains relative directory structure in destination
  • Backup Creation: Backs up existing destination files to /tmp with timestamps
  • Environment Substitution: Uses envsubst for variable replacement
  • Atomic Operations: Creates temporary files before final placement

File Processing:

  • Processes all files (not just .template extensions)
  • Handles filenames with spaces and special characters safely
  • Creates destination subdirectories as needed
  • Provides detailed logging for each file processed

envsubst-crontab

Purpose: Crontab installation from templates with environment variables
Location: .scripts/ops/envsubst-crontab

Usage:

1
2
3
4

[ENV_VARS...] .scripts/ops/envsubst-crontab <template_file>

# Example:
NODE=icarus SCRIPTS=/opt/scripts .scripts/ops/envsubst-crontab ./cron.template

Standard Environment Variables:

1
2
3
4
5
6
7

NODE             # Node name (e.g., "icarus")
TIER1            # Path to tier1 directory
TIER2            # Path to tier2 directory  
TIER3            # Path to tier3 directory
SCRIPTS          # Path to scripts directory
HOST_IP          # Host IP address
DOMAIN           # Base domain name

Safety Features:

  • Backup Creation: Current crontab backed up before installation
  • Validation: Rendered crontab validated before installation
  • Preview: Shows rendered crontab content before installation
  • Rollback: Automatic rollback on installation failure
  • Empty Check: Prevents installation of empty crontabs

envsubst-ufw

Purpose: UFW firewall rule application with template support
Location: .scripts/ops/envsubst-ufw

Usage:

1
2
3
4

.scripts/ops/envsubst-ufw <rules_dir> [host_rules_file]

# Example:
.scripts/ops/envsubst-ufw ./node/.host/ufw/rules ./node/.host/ufw/host.rules

Rule Processing:

  1. Docker Service Rules: Processes *.rules files using ufw-docker command
  2. Host Rules: Processes optional host-level UFW rules
  3. Idempotent Application: Rules can be safely re-applied
  4. Comment Handling: Ignores blank lines and # comments
  5. Error Handling: Stops on first rule failure with detailed error reporting

Rule File Format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Docker service rules (processed by ufw-docker)
# File: rules/service.rules
allow service-name 80/tcp
allow service-name 443/tcp
deny service-name 23/tcp

# Host rules (processed by standard ufw)  
# File: host.rules
allow from 192.168.1.0/24 to any port 22
deny from any to any port 23
limit ssh

Development and Maintenance

lint-docker-compose

Purpose: Docker Compose file linting and formatting
Location: .scripts/ops/lint-docker-compose

Usage:

1
2

.scripts/ops/lint-docker-compose          # Lint all compose files
.scripts/ops/lint-docker-compose --hook   # Pre-commit hook mode

Features:

  • Automatic Fixing: Uses dclint with --fix option for automatic corrections
  • Pre-commit Integration: --hook mode processes only staged files
  • Exclusions: Excludes data, state, vault directories from linting
  • Re-staging: In hook mode, automatically re-stages fixed files
  • Recursive Processing: Processes all docker-compose*.yml files in repository

Pre-commit Hook Integration:

1
2
3

# .git/hooks/pre-commit (example)
#!/bin/bash
.scripts/ops/lint-docker-compose --hook

check-package-updates

Purpose: System package update monitoring with notifications
Location: .scripts/ops/check-package-updates

Usage:

1
2
3
4

.scripts/ops/check-package-updates <NODE> <TELEGRAM_BOT_URL>

# Example:
.scripts/ops/check-package-updates icarus https://api.telegram.org/bot<token>/sendMessage

Process:

  1. Package List Update: Runs apt-get update to refresh package information
  2. Update Detection: Uses apt-get --just-print upgrade to identify available updates
  3. Formatting: Formats package list for Telegram notification with counts
  4. Notification: Sends Telegram alert with update details and package names
  5. Error Handling: Automatic error trapping with failure notifications

Output Format:

5 updates available: `package1`, `package2`, `package3`, `package4`, `package5`

sync-remote

Purpose: Git repository synchronization for node changes
Location: .scripts/ops/sync-remote

Usage:

1
2
3
4

.scripts/ops/sync-remote <NODE> <TELEGRAM_BOT_URL>

# Example:
.scripts/ops/sync-remote icarus https://api.telegram.org/bot<token>/sendMessage

Git Operations:

  1. Staging: Automatically stages all changes in node directory
  2. Branch Management: Creates or switches to sync/<NODE> branch
  3. Bot Identity: Configures commits with “Adam Jensen [bot]” identity
  4. Commit Creation: Creates commits with timestamp and node identification
  5. Rebase Integration: Rebases onto origin/trunk with conflict detection
  6. Force Push: Uses --force-with-lease for safe remote updates

Branch Strategy:

  • Branch Pattern: sync/<NODE> (e.g., sync/icarus, sync/daedalus)
  • Base Branch: origin/trunk
  • Commit Message: [NODE] Auto-commit: TIMESTAMP
  • Conflict Handling: Automatic rebase abort and notification on conflicts

Host-Specific Scripts

Node Initialization Scripts

Each host maintains numbered initialization scripts in .host/.scripts/:

UFW Firewall Initialization

Pattern: <NODE>/.host/.scripts/10-ufw

Common Implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/usr/bin/env bash
source "$(git -C "$(dirname "$0")" rev-parse --show-toplevel)/.scripts/bootstrap.sh"
SCRIPT_NAME="init/<NODE>/ufw"

navigate_to_repo_root

setup_ufw_config() {
    local ufw_dir="$REPO_ROOT/$NODE/.host/ufw"
    if [[ -d "$ufw_dir" ]]; then
        log_info "Applying UFW configuration..."
        "$SCRIPTS_ROOT/ops/envsubst-ufw" "$ufw_dir"
    fi
}

init_service "ufw" "ufw" "setup_ufw_config"

Cron Job Initialization

Pattern: <NODE>/.host/.scripts/20-cron or <NODE>/.host/.scripts/30-cron

Common Implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/usr/bin/env bash
source "$(git -C "$(dirname "$0")" rev-parse --show-toplevel)/.scripts/bootstrap.sh"
SCRIPT_NAME="init/<NODE>/cron"

navigate_to_repo_root

setup_cron_config() {
    local crontab_file="$REPO_ROOT/$NODE/.host/cron/crontab"
    if [[ -f "$crontab_file" ]]; then
        log_info "Applying cron configuration..."
        "$SCRIPTS_ROOT/ops/envsubst-crontab" "$crontab_file"
    fi
}

init_service "cron" "cron" "setup_cron_config"

API Documentation

Automation Script API

Standard Exit Codes 

1
2
3
4
5

0    # Success
1    # General error  
64   # Command line usage error (missing arguments)
65   # Data format error (invalid input)
66   # Input file not found

Environment Variable Conventions 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Required by most scripts
NODE                    # Current node name
TELEGRAM_BOT_URL       # Telegram notification endpoint
REPO_ROOT              # Repository root path (set by bootstrap)
SCRIPTS_ROOT           # Scripts directory path (set by bootstrap)

# Tier paths (set by init framework)
TIER1                  # Path to tier 1 storage
TIER2                  # Path to tier 2 storage  
TIER3                  # Path to tier 3 storage

Logging API

All scripts support consistent logging through the common library:

1
2
3
4
5

# Log levels with automatic timestamp and script identification
log_info "message"     # [script] [INFO] timestamp message
log_warn "message"     # [script] [WARN] timestamp message  
log_error "message"    # [script] [ERROR] timestamp message (to stderr)
log_success "message"  # [script] [SUCCESS] timestamp message

Error Handling API

Standardized error handling with automatic notifications:

1
2
3
4
5

# Setup automatic error trapping
setup_error_trap "Operation Description" "$TELEGRAM_BOT_URL" "$NODE"

# Manual error reporting
telegram_error "Title" "$TELEGRAM_BOT_URL" "Error details" "$NODE"

Docker Compose Integration API

Service Discovery Pattern

Services are discovered through Docker labels:

1
2
3
4

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.service-name.rule=Host(`service.domain.com`)"
  - "traefik.http.services.service-name.loadbalancer.server.port=8080"

Health Check Pattern

Services implement standardized health checks:

1
2
3
4
5
6

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 60s

Network Assignment Pattern

Services connect to standardized network types:

1
2
3
4
5
6
7
8
9

networks:
  enclave:              # Internal service communication
    ipv4_address: 10.x.y.z
  direct:               # External connectivity
    ipv4_address: 172.20.x.y  
  root-proxy:           # Traefik integration
    external: true
  socket:               # Docker API access
    internal: true

Usage Examples

Complete Service Setup 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# 1. Repository setup
git clone <repository>
cd infrastructure
./init.sh

# 2. Tier structure initialization
.scripts/ops/setup-tiers icarus /mnt/ssd/tier1 /mnt/hdd/tier2 /mnt/bulk/tier3

# 3. Service initialization
cd icarus/.host
../../.scripts/ops/envsubst-ufw ./ufw/rules ./ufw/host.rules
../../.scripts/ops/envsubst-crontab ./cron/crontab

# 4. Service deployment
cd ../arr
docker compose up -d

# 5. Backup setup
cd ../.host
../../.scripts/ops/sync-tiers icarus $TELEGRAM_BOT_URL

Maintenance Workflow 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Update check
.scripts/ops/check-package-updates icarus $TELEGRAM_BOT_URL

# Configuration update
vim icarus/arr/docker-compose.yml
.scripts/ops/lint-docker-compose

# Sync changes
.scripts/ops/sync-remote icarus $TELEGRAM_BOT_URL

# Backup execution
.scripts/ops/sync-tiers icarus $TELEGRAM_BOT_URL

Development Workflow 

1
2
3
4
5
6
7
8
9

# Pre-commit setup
.scripts/ops/lint-docker-compose --hook  # Add to git hooks

# Template development
export SERVICE_NAME=newservice PORT=8080
.scripts/ops/envsubst-directory ./templates ./config

# Testing
docker compose -f ./config/docker-compose.yml config

This reference covers the key scripts for managing and maintaining the automation framework.

Last updated on
Configurations