Recipes
Tier Recovery

Tier Recovery

How to recover your data from remote backups when things go wrong.

What You Need

  • Access to your node
  • Working rclone config (in @tier1/.host/rclone/config)
  • Know your remote names (check your rclone config)

Find Your Remote Names 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

cd node/.host

# List all your configured remotes
rclone --config @tier1/.host/rclone/config listremotes

# Example output might be:
# yandex_daedalus_tier1:
# yandex_daedalus_tier2:
# s3_backup_tier1:
# s3_backup_tier2:

Quick Recovery (Everything) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Navigate to your node
cd /opt/infrastructure/node  # replace 'node' with actual node name

# Stop all services
find . -name "docker-compose.yml" -exec docker compose -f {} down \;

# Go to host directory
cd .host

# Restore using your actual remote names
rclone --config @tier1/.host/rclone/config sync your_tier1_remote: ./@tier1
rclone --config @tier1/.host/rclone/config sync your_tier2_remote: ./@tier2

# Fix ownership
sudo chown -R root:root ./@tier1 ./@tier2 ./@tier3

# Start services
cd ..
find . -name "docker-compose.yml" -exec docker compose -f {} up -d \;

Tier 1 Only (Configs and Secrets) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

cd node/.host

# Stop services
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} down \;

# Restore tier1
cd .host
rclone --config @tier1/.host/rclone/config sync your_tier1_remote: ./@tier1

# Fix ownership
sudo chown -R root:root ./@tier1

# Start services
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} up -d \;

Tier 2 Only (Application Data) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

cd node/.host

# Stop services (important for databases)
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} down \;

# Restore tier2
cd .host
rclone --config @tier1/.host/rclone/config sync your_tier2_remote: ./@tier2

# Fix ownership
sudo chown -R root:root ./@tier2

# Start services
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} up -d \;

Using Docker Services

If you want to use the existing backup containers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

cd node/.host

# Stop services first
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} down \;

cd .host

# Edit docker-compose.yml temporarily:
# Change the command from:
#   sync /data/tier1 remote_name: -P
# To:
#   sync remote_name: /data/tier1 -P

# Run the services
docker compose run --rm rclone1  # restores tier1
docker compose run --rm rclone2  # restores tier2

# Fix ownership
sudo chown -R root:root ./@tier1 ./@tier2

# Revert the docker-compose.yml changes
git checkout docker-compose.yml

Check What’s Available 

1
2
3
4
5
6
7
8
9

cd node/.host

# See what backups you have
rclone --config @tier1/.host/rclone/config ls your_tier1_remote:
rclone --config @tier1/.host/rclone/config ls your_tier2_remote:

# Check sizes
rclone --config @tier1/.host/rclone/config size your_tier1_remote:
rclone --config @tier1/.host/rclone/config size your_tier2_remote:

Disaster Recovery (Lost Everything)

If you lost the entire node:

  1. Follow initial setup guide
  2. Clone repository
  3. Setup tier structure
  4. Get rclone config (from another node or recreate)
  5. Restore data:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

cd node/.host

# Restore everything
rclone --config @tier1/.host/rclone/config sync your_tier1_remote: ./@tier1
rclone --config @tier1/.host/rclone/config sync your_tier2_remote: ./@tier2

# Fix ownership
sudo chown -R root:root ./@tier1 ./@tier2 ./@tier3

# Start services gradually
cd .. && find . -name "docker-compose.yml" -exec docker compose -f {} up -d \;

Common Issues

Don’t Know Your Remote Names 

1
2
3
4
5

# Check your rclone config
rclone --config @tier1/.host/rclone/config listremotes

# Or look at the config file directly
cat @tier1/.host/rclone/config

Config File Missing 

1
2
3
4
5

# Copy from another working node
scp other-node:/path/@tier1/.host/rclone/config @tier1/.host/rclone/

# Or you'll need to reconfigure rclone from scratch
rclone --config @tier1/.host/rclone/config config

Services Won’t Start 

1
2
3
4
5

# Check logs
docker compose logs service-name

# Try recreating containers
docker compose down && docker compose up -d --force-recreate

Wrong Permissions 

1
2
3

# Always fix ownership after any restore
sudo chown -R root:root ./@tier1 ./@tier2 ./@tier3
sudo chmod -R 755 ./@tier1 ./@tier2 ./@tier3

What’s in Each Tier

  • Tier 1: Configs, secrets, certificates
  • Tier 2: Databases, user data, application state
  • Tier 3: Cache, logs (usually not backed up)

Recovery Steps

  1. Stop services - Prevent corruption
  2. Check your remotes - Know what you’re restoring from
  3. Restore tier1 first - Get configs and secrets
  4. Then tier2 - Application data
  5. Fix ownership - Always!
  6. Start services - Test everything works

Remember: You should know your remote names, or check your rclone config to find them!

Last updated on
Initial Host Setup