Storage Manager

LVM-based 3-tier storage management for Doh with automatic PV discovery, guided tier assignment, and bcache acceleration.

Architecture

The Storage Manager uses LVM Physical Volumes organized into 3 volume groups to create a flexible and powerful storage architecture.

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Tier 0 Cache  │    │  Tier 1 Archive │    │  Tier 2 Backup  │
│   (vg_cache)    │    │   (vg_archive)  │    │   (vg_backup)   │
├─────────────────┤    ├─────────────────┤    ├─────────────────┤
│ • Fast SSDs     │ ──>│ • Primary Data  │ ──>│ • Large HDDs    │
│ • bcache cache  │    │ • LVM Pool      │    │ • Backup Mirror │
│ • 50-500GB      │    │ • Expandable    │    │ • ≥ Archive Size│
└─────────────────┘    └─────────────────┘    └─────────────────┘

The Intelligent Workflow System

The Storage Manager uses an intelligent workflow system that:

• Enforces Dependencies: Each command validates that prerequisite steps have been completed.
• Tracks Progress: All workflow state is stored in .doh/static/storage.pod.yaml under storage_state.
• Provides Guidance: Clear error messages guide you to the next required step.
• Shows Status: doh storage status displays complete workflow progress with timestamps.

Workflow State Tracking

The system tracks these workflow states:

Dependency Chain

The commands follow a strict dependency chain to ensure a successful setup: deps → prepare → init → setup

Requirements

System Dependencies

The storage manager requires the following system tools. These can be installed easily with the doh storage deps command.

• lvm2: Linux LVM tools for volume management
• bcache-tools: Caching layer for faster access
• rsync: Backup synchronization
• parted: Partition management tool
• util-linux: System utilities

Device Requirements

For the best results, use devices that match the role of each tier:

Getting Started: The Setup Workflow

Follow these steps to configure your storage system from start to finish.

Step 1: Install Dependencies

First, install the required system tools. The command will detect your OS and use the appropriate package manager.

doh storage deps

Step 2: Prepare LVM Physical Volumes (PVs)

Critical: You must create LVM Physical Volumes on your storage drives before they can be discovered and used by the storage manager.

Use parted or gparted to:

1. Create partitions on your drives.
2. Set the partition type to "Linux LVM" (ID 8e).
3. Run sudo pvcreate /dev/deviceXpY on the partition.

Example:

# Create an LVM PV on a new drive
sudo parted /dev/nvme0n1 mklabel gpt
sudo parted /dev/nvme0n1 mkpart primary 0% 100%
sudo parted /dev/nvme0n1 set 1 lvm on
sudo pvcreate /dev/nvme0n1p1

# Repeat for all other drives you intend to use
sudo pvcreate /dev/sda1     # Archive device
sudo pvcreate /dev/sdb1     # Another archive device
sudo pvcreate /dev/sdc1     # Backup device

For more detailed instructions, run doh storage prepare guide.

Step 3: Discover and Assign PVs to Tiers

Once you have created your LVM PVs, run the guided assignment tool. This command will:

• Detect all LVM Physical Volumes.
• Recommend tier assignments based on drive characteristics (speed, size).
• Guide you through assigning each PV to the appropriate tier (Cache, Archive, or Backup).
• Automatically create or extend the LVM Volume Groups (vg_cache, vg_archive, vg_backup).

doh storage init

Step 4: Configure Mount Points (Optional)

When to use: Anytime - before or after doh storage setup. This step is idempotent and can be run multiple times.

By default, storage mounts at /mnt/storage and /mnt/backup, which require sudo access. For user-friendly access, configure mount points in your home directory:

doh storage mount

Smart Behavior:

• Auto-Mount Detection: Automatically finds storage auto-mounted to /media/ by desktop environments
• Intelligent Relocation: Moves auto-mounted storage to your configured paths with proper permissions
• Before setup: Updates configuration only - you'll run doh storage setup next
• After setup: Detects active mounts and offers to remount at new locations automatically
• Always: Suggests mounting at ~/storage and ~/backup with proper user permissions
• Idempotent: Safe to run multiple times - won't break existing configurations

This command intelligently handles auto-mounted storage issues and can migrate your storage to new mount points even while it's actively in use.

Step 5: Implement the Storage System

After assigning your PVs to tiers (and optionally configuring mount points), implement the storage system. This creates logical volumes, sets up bcache, formats filesystems, and mounts at the configured locations.

doh storage setup

After this step, your 3-tier storage system will be fully operational.

Command Reference

System Dependencies

doh storage deps                 # Install system dependencies (lvm2, bcache-tools, etc)
doh storage deps --dry-run       # Show what would be installed

Discovery & Preparation

doh storage prepare              # Analyze drives, show tier recommendations, and guide PV creation
doh storage prepare list-all     # Show all drives including system drives
doh storage prepare show-lvm     # Show detailed LVM analysis and diagnostics
doh storage prepare guide        # Detailed LVM PV creation instructions

Configuration & Setup

doh storage init                 # Guided PV assignment to tiers
doh storage init list-only       # Show PV categorization only, without configuration
doh storage init apply           # Auto-assign PVs using recommendations
doh storage init --json          # Output PV categorization in JSON format
doh storage mount                # Configure user-friendly mount points (anytime, idempotent, smart remounting)
doh storage setup                # Implement the full storage system
doh storage setup --dry-run      # Preview setup operations without making changes
doh storage setup auto           # Auto-setup without prompts using current configuration
doh storage setup deps-only      # Only install system dependencies

Management & Status

doh storage status               # Check system and workflow status
doh storage mount                # Change mount points (works on live systems too)
doh storage extend tier1 /dev/sdd1 # Add a new PV to an existing tier
doh storage config show          # Display current configuration with workflow status and pod analysis
doh storage config edit          # Interactively edit the modernized tier-based configuration
doh storage config set --key <key> --value <value>  # Set specific configuration values
doh storage config reset         # Reset configuration to modernized defaults
doh storage repair check         # Run diagnostics to check for issues

Backup Operations

doh storage backup sync          # Synchronize primary storage to the backup tier
doh storage backup status        # Show backup status and last sync time
doh storage backup restore       # Restore from backup (interactive)
doh storage backup verify        # Verify backup integrity
doh storage backup perms         # Fix backup directory permissions (requires sudo)

Automated Backup System

The Storage Manager includes a sophisticated server-managed backup system that runs automatically:

Backup Monitoring Features

• Scheduled Backups: Automatic backups based on cron schedule (default: hourly)
• Timer State Management: Tracks last run, next scheduled time, and missed windows
• Health Monitoring: Continuous monitoring of backup health and storage usage
• Error Recovery: Automatic error state detection and recovery attempts
• Clock Drift Correction: Adjusts for system clock changes to maintain schedule accuracy
• Storage Analysis: Monitors backup storage usage and warns about capacity issues

Backup Configuration

storage:
  backup_schedule: '0 * * * *'     # Cron format (default: hourly)
  rsync_options: ['-a', '--delete'] # rsync options for synchronization
  rsync_excludes:                   # Patterns to exclude from backups
    - '/cache'                      # bcache artifacts
    - '*.tmp'                       # Temporary files
    - '/lost+found'                 # Filesystem recovery
  backup_timer_state:               # Managed automatically - do not edit
    last_run: null
    next_scheduled: null
    missed_windows: 0
    max_missed_windows: 2
    error_state: false

The backup system initializes automatically when the storage manager loads and continues running in the background, ensuring your data is regularly synchronized to the backup tier.

Configuration

The storage manager's configuration, including volume group names and device assignments, is stored in /.doh/static/storage.pod.yaml. This dedicated configuration file is automatically inherited by the main pod.yaml, providing centralized storage management. While you can view the configuration directly, it is strongly recommended to use the CLI tools to modify it.

Default volume groups:

storage:
  tier0_volume_group: 'vg_cache'     # Cache tier
  tier1_volume_group: 'vg_archive'   # Archive tier
  tier2_volume_group: 'vg_backup'    # Backup tier

To modify these or other settings, use doh storage config edit.

Examples

New System Setup

# 1. Install system dependencies
doh storage deps

# 2. Analyze drives and get PV creation guidance
doh storage prepare

# 3. Create LVM PVs on your drives (manual step)
sudo pvcreate /dev/nvme0n1p1  # Fast SSD for cache
sudo pvcreate /dev/sda1       # Primary storage
sudo pvcreate /dev/sdb1       # More primary storage
sudo pvcreate /dev/sdc1       # Large backup drive

# 4. Assign PVs to tiers
doh storage init

# 5. Configure user-friendly mount points (optional)
doh storage mount

# 6. Implement the storage system
doh storage setup

Adding Storage

# Prepare a new drive as an LVM PV
sudo pvcreate /dev/sdd1

# Extend the archive tier with the new drive
doh storage extend tier1 /dev/sdd1

Changing Mount Points

# Change mount points on a live system
doh storage mount

# The command will:
# 1. Detect current mounts and auto-mounted storage
# 2. Offer user-friendly defaults (~/storage, ~/backup)
# 3. Relocate auto-mounted storage from /media/ paths
# 4. Safely unmount, remount, and set permissions
# 5. Add fstab entries for persistence
# 6. Update storage state for web UI visibility

Dealing with Auto-Mounted Storage

If your desktop environment auto-mounts storage to /media/username/, the mount command will automatically detect and relocate it:

# Detect and fix auto-mounted storage
doh storage mount

# Example output:
# 🔍 Auto-Mount Detection Results:
#   Backup storage found: /dev/mapper/vg_backup-lv_backup
#     Currently at: /media/username/backup_volume
#     Will move to: ~/backup
#   Archive storage found: /dev/mapper/vg_archive-lv_archive  
#     Currently at: /media/username/archive_volume
#     Will move to: ~/storage

Check Status

# Overall system and workflow status
doh storage status

# Detailed LVM information
sudo vgdisplay
sudo pvdisplay

Troubleshooting

Workflow Issues

If you encounter an error about a step not being complete, check the workflow status.

# Check workflow status
doh storage status

This will show which steps are complete and which need to be run.

No PVs Found

If doh storage init finds no PVs, verify that they exist and are correctly configured.

# Check if PVs exist on the system
sudo pvdisplay

# Check for LVM signatures on a specific device
sudo blkid /dev/deviceXpY

# If missing, create the PV
sudo pvcreate /dev/deviceXpY

Permission Issues

If you see permission errors, ensure LVM tools are installed and you have sudo access.

# Check sudo access works
sudo vgs

# Fix backup directory permissions if needed
doh storage backup perms

Auto-Mount Issues

If your storage appears and disappears depending on access method, or if files seem missing:

# Check for auto-mounted storage
doh storage mount

# This will detect if your storage was auto-mounted to /media/ paths
# and relocate it to your configured mount points

Web UI Issues

If storage doesn't appear in the web interface:

# Check storage state
doh storage status

# Update storage state for web UI
doh storage mount  # This updates state automatically

Advanced Features

Health Monitoring

The storage manager includes advanced monitoring capabilities:

• SMART Health: Monitors drive health using SMART data
• I/O Statistics: Tracks read/write performance and patterns
• Usage Analysis: Monitors storage capacity and growth trends
• Tier Recommendations: Suggests when to expand tiers based on usage

Growth Management

The system provides intelligent growth recommendations:

# Check tier growth capabilities
doh storage status

# Get detailed usage recommendations
doh storage extend tier1 --dry-run  # Preview expansion

Safety

• Always backup important data before modifying drives.
• Use dry-run modes (--dry-run) where available to preview changes.
• Verify device paths (/dev/sdX) carefully before running commands to avoid data loss.
• The automated backup system provides continuous protection for your primary storage.

Architecture Details

The Storage Manager has been refactored with a centralized architecture to eliminate code duplication and improve reliability:

Centralized Utility Methods

• User Management

  • getUserInfo(): Gets current user permissions and identity
  • setDirectoryPermissions(): Sets permissions without recursive operations that can hang
  • createMountDirectory(): Creates directories with proper ownership

• Mount Operations

  • isMounted(): Checks if a path is currently mounted
  • getMountedDevice(): Gets the device mounted at a specific path
  • unmountPath(): Safely unmounts with lazy fallback for busy filesystems
  • mountDevice(): Mounts devices with automatic permission setting
  • addToFstab(): Adds persistent mount entries

• Auto-Mount Management

  • detectAutoMounts(): Discovers storage auto-mounted by desktop environments
  • relocateAutoMounts(): Moves auto-mounted storage to proper configured paths

State Management

All storage state is centrally managed and synchronized with the web UI through:

• updateStorageState(): Updates storage visibility and status for web interface
• getStorageState(): Retrieves current workflow and mount state
• Automatic state persistence in /.doh/static/storage.pod.yaml