Data Tiers

How data is organized using @tier1, @tier2, and @tier3. Every service uses the same data organization patterns for consistent backup strategies and storage management.

How Tiers Work

The tier system uses symbolic links so services can use consistent @tier1, @tier2, and @tier3 paths while the actual storage can be anywhere you want.

Tier Classification

Directory Structure

Project Level

project/
├── @tier1 -> /path/to/tier1/storage/project/
├── @tier2 -> /path/to/tier2/storage/project/
├── @tier3 -> /path/to/tier3/storage/project/
└── .env -> @tier1/.env

Host Level

node/.host/
├── @tier1 -> /path/to/tier1/storage/
├── @tier2 -> /path/to/tier2/storage/
└── @tier3 -> /path/to/tier3/storage/

What Goes Where

@tier1 - Config and Secrets

Small, critical stuff that rarely changes:

1
2
3
4

volumes:
  - ./@tier1/traefik/cf-dns-api-token:/run/secrets/traefik-cf-dns-api-token:ro
  - ./@tier1/nginx/data:/etc/nginx:ro
  - ./@tier1/pihole/data:/etc/pihole:ro

Storage needs:

• Fast, reliable storage (SSD recommended)
• Encrypted if possible
• Multiple backup copies
• Version control friendly

@tier2 - User Data and Databases

Important stuff that changes regularly:

1
2
3
4

volumes:
  - ./@tier2/server/data:/usr/src/app/upload
  - ./@tier2/postgres/data:/var/lib/postgresql/data
  - ./@tier2/mariadb/data:/var/lib/mysql

Storage needs:

• Good performance for databases
• Regular automated backups
• RAID or redundancy recommended
• Sized for data growth

@tier3 - Cache and Temp Files

Stuff you can recreate if lost:

1
2
3
4

volumes:
  - ./@tier3/jellyfin/cache:/cache
  - ./@tier3/logs:/var/log
  - ./@tier3/shared/media:/media:ro

Storage needs:

• Large capacity
• Fast for better performance
• Can use cheaper storage
• Automated cleanup helpful

Shared Data

Services can share data through the shared directory:

@tier3/shared/
├── project1/
│   ├── service1/downloads/
│   └── service2/media/
└── project2/
    └── service1/files/

Example - Media Pipeline:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# qBittorrent downloads here
qbittorrent:
  volumes:
    - ./@tier3/shared/qbt/downloads:/downloads

# Jellyfin reads from here
jellyfin:
  volumes:
    - ./@tier3/shared/qbt/downloads/media:/media:ro
    - ./@tier3/shared/jef/media:/library:ro

# Sonarr/Radarr manage both
sonarr:
  volumes:
    - ./@tier3/shared:/shared  # Single mount for hard links

Setup and Management

Initial Setup

1
2

# Set up tiers for a node
.scripts/ops/setup-tiers node /ssd/tier1 /hdd/tier2 /bulk/tier3

This creates:

• All necessary directories
• Symbolic links for each project
• Host-level tier links
• Shared directory structure

Backup Automation

1
2

# Sync tiers to cloud storage
.scripts/ops/sync-tiers node $TELEGRAM_BOT_URL

What gets backed up:

• @tier1: Critical configs (rclone with encryption)
• @tier2: User data (rclone with verification)
• @tier3: Usually not backed up (too large, recreatable)

Docker Integration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Host backup service
services:
  rclone-tier1:
    image: rclone/rclone:latest
    volumes:
      - ./@tier1:/data/tier1:ro
      - ./rclone/config:/config/rclone:ro
    command: sync /data/tier1 remote:backup/tier1 --config /config/rclone
    
  rclone-tier2:
    image: rclone/rclone:latest
    volumes:
      - ./@tier2:/data/tier2:ro
      - ./rclone/config:/config/rclone:ro
    command: sync /data/tier2 remote:backup/tier2 --config /config/rclone

Storage Examples

Development Setup

1
2

# Everything on one disk
setup-tiers dev /home/user/tiers/tier1 /home/user/tiers/tier2 /home/user/tiers/tier3

Production Setup

1
2

# Different storage types
setup-tiers prod /ssd/tier1 /raid/tier2 /bulk/tier3

Home Server Setup

1
2

# Mixed storage
setup-tiers home /nvme/tier1 /ssd/tier2 /hdd/tier3

Recovery Procedures

@tier1 Recovery (Critical)

1. Restore from latest backup
2. Verify file permissions
3. Test service startup
4. Check certificate dates

@tier2 Recovery (Data)

1. Restore from daily backup
2. Run database integrity checks
3. Verify application data
4. Test user access

@tier3 Recovery (Cache)

1. Clear cache directories
2. Let services rebuild cache
3. Monitor performance
4. Optionally restore from backup for speed

Best Practices

Choosing Tiers

Put in @tier1:

• Configuration files
• SSL certificates
• Database schemas
• API keys and passwords
• Service discovery configs

Put in @tier2:

• User uploads
• Database data files
• Application state
• User settings
• Important business data

Put in @tier3:

• Application logs
• Transcoded media
• Search indexes
• Temporary processing files
• Downloaded content

Security

• @tier1: Encrypt, restrict access, version control
• @tier2: Regular backups, access controls, integrity checks
• @tier3: Cleanup policies, capacity monitoring

This tier system keeps your data organized and makes backup/recovery straightforward across all nodes.