Home Doh Ref
Guides β–Ό
Dohballs β–Ό
  • πŸ“ doh_modules
    • πŸ“¦ dataforge
    • πŸ“¦ express
    • πŸ“ sso
    • πŸ“ user

Cloud Manager

A comprehensive, real-time system for managing distributed Doh instances through a unified web dashboard. Cloud Manager enables remote administration, file synchronization, Git repository management, direct SSH terminal access, and collaborative sharing across multiple cloud-connected Doh deployments.

What is Cloud Manager?

Cloud Manager transforms individual Doh instances into a connected, event-driven ecosystem where you can:

  • Monitor multiple instances from a single, real-time dashboard.
  • Control remote instances with commands, Git operations, and full SSH terminal access.
  • Synchronize files and folders between any connected instances with smart, bidirectional sync.
  • Share access to your instances with team members and collaborators with granular, time-limited permissions.
  • Manage Git repositories across your entire cloud infrastructure from a central UI.
  • View remote logs and pod configurations directly in the dashboard.
  • Automate ongoing sync relationships with Active Mirrors.
  • Integrate with AI tooling via the Model Context Protocol (MCP).

Think of it as your mission control for distributed Doh deployments, built on a secure, modern, and event-driven architecture.

Core Concepts

Instances

Each connected Doh deployment appears as an "instance" in Cloud Manager. Instances maintain persistent connections via WebSocket and can be controlled remotely. Instances can be owned or shared, with ownership and permissions clearly displayed in the UI.

The Cloud Manager Host

A Cloud Manager is not a special type of server; it is simply a standard Doh instance with the cloud_manager module installed. This provides incredible flexibility:

  • Any Instance, Any Cloud: Any Doh instance can be turned into a Cloud Manager host.
  • Clouds of Clouds: You can run multiple, isolated Cloud Managers for different teams, projects, or environments, even on the same physical machine, because each instance is just a folder.
  • Networking: The Cloud Manager instance only needs to be network-accessible to the other instances that will connect to it. For hosts with dynamic IP addresses, a simple Dynamic DNS (DynDNS) setup is sufficient to keep it reachable.

Sharing

Any instance owner can share access with other usersβ€”either to specific instances or all their instances. Shares can have custom permissions (e.g., read-only, command execution) and optional expiration dates, with all access managed through a secure, database-driven system.

File Synchronization

Smart bidirectional sync between instances with conflict detection, progress tracking, and caching. You can perform one-time syncs or create persistent "Active Mirrors" for ongoing relationships.

Remote Management

Cloud Manager provides multiple ways to manage instances remotely:

  • Commands: Execute predefined or custom commands.
  • Git: A full-featured UI for managing repositories.
  • SSH Terminal: A fully functional, browser-based terminal for direct shell access.

MCP (Model Context Protocol)

A WebSocket-based protocol allowing developer tools to securely connect to the Cloud Manager and interact with managed instances. This enables advanced, programmatic control and automation.

The Dashboard

The Cloud Manager dashboard, accessible at /admin/cloud, is a centralized UI for all management tasks. It features a responsive, event-driven interface that updates in real-time without polling.

Main Tabs

  • Instances: The main overview of all your owned and shared instances. Monitor status, view system information, and access quick actions.
  • File Sync: A powerful interface for browsing remote filesystems and setting up one-time or recurring (Active Mirror) synchronization tasks between instances.
  • Git: Discover and manage Git repositories on any connected instance. View status, stage/commit files, manage branches, and perform remote operations like push and pull.
  • SSH: Open a full-featured, low-latency SSH terminal to any connected instance you have permission to access.
  • Logs: View real-time logs from your instances directly within the dashboard.
  • Shares: Manage who has access to your instances. Create, view, and revoke shares.
  • Pods: Inspect the pod.yaml configuration files of your connected instances.

Key UI Components

  • Instance Selection Toolbar: Once an instance is selected, a floating toolbar appears on the right, providing quick access to common actions like update, restart, and upgrade.
  • Recent Actions Panel: A consolidated, real-time log of all commands and operations performed across all instances, showing their status (pending, success, error) and output.

Getting Started: Setting Up and Connecting

Part 1: Setting Up Your Cloud Manager Host

Before you can connect instances, you need one Doh instance to act as the central manager.

Step 1: Install the Cloud Manager Module

On the Doh instance you want to designate as your Cloud Manager, run:

# This installs the manager UI and backend
doh install cloud cloud_manager

Step 2: Create a Superadmin User

A Cloud Manager needs at least one user account to own and manage instances. The easiest way to create the initial administrator is with the poduser command.

# Create a new user and grant them superadmin privileges
doh poduser

Follow the prompts to make a superadmin and set the username and password. This user will be able to log in to the Cloud Manager and anchor other instances to their account.

Step 3: Start Your Manager and Ensure It's Accessible

Run your Cloud Manager instance like any other Doh instance:

doh run

Ensure this instance is accessible over the network from any other instances you plan to connect.

Part 2: Connecting an Instance to Your Manager

Now, from any other Doh instance you want to manage, you can connect it to your new Cloud Manager.

The easiest way to connect is with a single command:

# Set the cloud endpoint and anchor the instance in one step
doh cloud anchor http://your-cloud-manager-url.com

This command will begin an interactive prompt for the username and password you created in Part 1. Your credentials are used only once to get a secure token and are never stored on the client instance.

Once anchored, the instance will appear on your Cloud Manager's dashboard.

Feature Deep Dive

SSH Terminal Access

Cloud Manager provides a fully functional, browser-based SSH terminal that is compatible with both Bun and Node.js runtimes.

  • How to Use: Navigate to the SSH tab, or click the "Terminal" button on any instance card. A new terminal window will open, giving you direct shell access to the instance.
  • Real PTY Implementation: The terminal uses the script command on Unix-like systems to allocate a proper pseudo-terminal (PTY). This ensures that the shell runs in interactive mode, with a visible prompt and full support for terminal features, which is not possible with a standard child_process.spawn().
  • Features:
    • Real-time, low-latency interaction via WebSockets.
    • Terminal resizing with proper SIGWINCH handling.
    • Full ANSI color and UTF-8 support.
    • Quick command buttons for common operations (cd ~, doh update, etc.).
    • Secure, using the existing Cloud Manager permission system (command:cloud_instance).

File Synchronization

The File Sync tab provides a modern, efficient synchronization system with server-side comparison and cross-platform compatibility.

  • Server-Side Comparison Architecture: The Cloud Manager orchestrates file comparisons using a DRY (Don't Repeat Yourself) architecture where managed_site instances collect file metadata and the cloud_manager performs intelligent comparisons.
  • One-Time Sync:
    1. Select a source instance and browse to the file/folder.
    2. Select a destination instance. Files sync to the same path.
    3. The system performs server-side analysis showing all files with their sync status (identical, different, new).
    4. Execute the sync with real-time, file-level progress feedback.
  • Active Mirrors: For ongoing synchronization, you can save a sync configuration as an "Active Mirror". The dashboard will show the mirror's status (in sync, needs sync, stale) and allow you to re-run the sync with a single click.
  • Advanced Sync Features:
    • Server-Side Comparison: Eliminates duplicate comparison logic by centralizing file analysis in the Cloud Manager
    • Cross-Platform Compatibility: Handles line ending differences between Unix/Linux (LF) and Windows (CRLF) systems automatically
    • Always-Fresh MD5 Verification: Uses real-time MD5 checksums calculated on-demand for accurate sync decisions with line ending normalization for text files
    • Single-Call Optimization: The get_file_comparison_data command collects all file metadata (size, mtime, MD5) in one efficient call
    • Proper Glob Pattern Matching: Uses standard glob patterns for file exclusion instead of simple string matching
    • Complete File Visibility: Shows all files in comparison results, including identical files with green "Synced" badges
    • Batch MD5 Optimization: Calculates MD5 checksums for multiple files simultaneously, reducing network overhead
    • Intelligent Transfer Decisions: Skips files with identical content automatically, preventing unnecessary bandwidth usage
    • Conflict Resolution: Automatically backs up destination files that are newer than the source files before overwriting them
    • Large File Support: Uses chunked streaming for efficient transfer of large files with progress tracking
    • Cross-Instance Efficiency: MCP (Model Context Protocol) commands automatically benefit from smart sync optimizations for programmatic file operations
    • Dohball Processing: Automatically processes dohball.json removals after file sync operations to ensure dohballs stay clean and to-spec by removing obsolete files

Git Repository Management

The Git tab provides a comprehensive UI for managing Git repositories on any connected instance.

  • Discovery: Automatically discovers all Git repositories within an instance's project directory.
  • Operations:
    • Status: View staged/unstaged files, commit history, and branch information.
    • File Actions: Stage, unstage, and commit files directly from the UI.
    • Branching: View, create, and switch between branches.
    • Remotes: Pull from and push to remote repositories.
    • Diff Viewer: See changes between commits.

Cloud Sharing System

The Shares tab allows you to securely share access to your instances.

  • Share Types:
    • Instance-Specific: Share access to one particular instance.
    • Global: Share access to ALL your current and future instances at once.
    • Permission-Controlled: Grant specific abilities (e.g., read:cloud_instance, command:cloud_instance). If no permissions are specified, the sharee inherits the owner's full permissions.
    • Time-Limited: Shares can be set to automatically expire.
  • How it Works: When you share an instance, the sharee will see it in their dashboard, grouped under "Shared with Me". All Cloud Manager features (Git, SSH, File Sync) seamlessly respect sharing permissions.

Architecture & Security

Cloud Manager is built on a modern, secure, and event-driven architecture with a DRY (Don't Repeat Yourself) design pattern.

DRY Architecture Pattern

The Cloud Manager implements a unified architecture where:

  • Cloud Manager as Orchestrator: The cloud_manager module serves as the central orchestrator for all cross-instance operations

  • Managed Site as Universal Anchor: All instances use the managed_site module as the universal anchor for consistent command handling

  • Server-Side Processing: File comparisons, sync operations, and data analysis are performed server-side to reduce network overhead

  • Single-Call Optimization: The get_file_comparison_data command collects all necessary file metadata in one efficient call

  • Centralized Logic: Duplicate comparison logic has been eliminated by centralizing file analysis in the Cloud Manager

  • Event-Driven UI: The dashboard uses a WebSocket connection to receive real-time updates from the server. There is no polling; data is pushed to the UI instantly when the state changes on the backend (e.g., an instance connects, a command completes).

  • JWT Authentication: Instance connections are secured using JSON Web Tokens (JWT). A short-lived siteAuthToken is generated for an instance to authenticate with the Cloud Manager.

  • Contextual Permissions: The system uses Doh's powerful contextual permission engine. Access is not based on simple roles, but is dynamically evaluated against the action being performed and the specific object being acted upon (e.g., Doh.permit(user, 'command:cloud_instance', instanceObject)). This allows for fine-grained control and secure sharing.

  • Database-Driven: Shares and Active Mirrors are persisted in a database, ensuring data integrity and consistency across sessions.

  • Secure Command & Git Handling: All remote operations are subject to path validation and command whitelisting to prevent unauthorized access or execution.

Recent Improvements

Cross-Platform File Synchronization

The Cloud Manager now includes enhanced cross-platform file synchronization capabilities:

  • Line Ending Normalization: Text files automatically have their line endings normalized during MD5 calculation to ensure files are correctly identified as identical across Unix/Linux (LF) and Windows (CRLF) systems
  • Binary File Preservation: Binary files maintain their exact byte-for-byte integrity without any normalization
  • Intelligent File Detection: The system automatically detects binary vs text files using the isFileBinary function
  • Enhanced Debugging: Comprehensive logging and debugging capabilities for troubleshooting sync issues

Improved File Discovery

  • Proper Glob Pattern Matching: File exclusion now uses standard glob patterns (e.g., .doh/, node_modules/) instead of simple string matching
  • Enhanced Pattern Support: Supports directory patterns, wildcard patterns, and exact matches for flexible file filtering
  • Comprehensive File Inclusion: Ensures important configuration files like .doh.yaml are properly included in sync operations

User Interface Enhancements

  • Complete File Visibility: The File Sync interface now shows all files in comparison results, including identical files with green "Synced" badges
  • Dark Mode Compatibility: Improved styling for dark mode environments with proper opacity and contrast
  • Unified Sync Controls: Consolidated duplicate sync buttons into a single, DRY system for better user experience
  • Enhanced Push/Pull Configuration: Push and pull buttons now properly add configurations to mirror areas instead of executing immediate syncs

API Reference

The Cloud Manager exposes a REST API for programmatic control.

WebSocket Events (Real-time)

Cloud Manager pushes real-time updates to dashboard clients via Socket.IO.

  • cloud:instance-update

    • Emitted when instance state/health changes.
    • Payload includes counts and an array of enriched instance objects.
    • Example payload:
      {
        "totalInstances": 3,
        "totalMcpClients": 1,
        "totalUsers": 2,
        "instances": [{
          "id": "fp-123",
          "userId": "alice",
          "info": { "fingerprint": "fp-123", "friendlyName": "prod-eu", "hostname": "prod.example.com", "port": 443, "https": true },
          "detailedInfo": null,
          "status": "connected",
          "connectedAt": "2025-01-01T12:00:00.000Z",
          "lastHeartbeat": "2025-01-01T12:01:00.000Z",
          "lastActivity": "2025-01-01T12:01:00.000Z",
          "health": { "memory": {}, "cpu": {}, "uptime": 12345 },
          "commandCount": 4,
          "online": true,
          "lastVerified": "2025-01-01T12:01:00.000Z",
          "responseTime": 85,
          "mcpControllable": true
        }],
        "timestamp": "2025-01-01T12:01:00.000Z"
      }
      
  • cloud:command-progress

    • Progress updates for long-running commands (e.g., file transfers).
    • Payload: { instanceId, commandId, progress, timestamp }.
  • cloud:sync-activity

    • Timeline items for sync operations (push/pull/cross-instance).
    • Example payload:
      {
        "id": "op-789",
        "type": "push",
        "status": "running",
        "path": "/var/www/app",
        "sourceId": "fp-src",
        "destinationId": "fp-dst",
        "sourceName": "dev",
        "destinationName": "prod",
        "message": "Syncing 42 file(s)...",
        "filesCount": 42,
        "timestamp": "2025-01-01T12:02:00.000Z"
      }
      
  • tty_data, tty_exit

    • Forwarded terminal data/lifecycle events from instances to the dashboard.

MCP Namespace (/mcp)

Dedicated Socket.IO namespace for MCP-compatible developer tools.

  • Authenticate: mcp:authenticate

    • Request: { siteAuthToken, connectionType: "mcp_client" }
    • Response: { success, connectionId, userId, connectionType }
  • Commands: mcp:command

    • Types: mcp:list_user_instances, mcp:get_logs, mcp:restart_instance, mcp:sync_files, mcp:run_command, mcp:get_status, mcp:cross_instance_sync_files, mcp:cross_instance_sync_folder, mcp:compare_files_cross_instance, mcp:execute_sync_from_comparison
    • Common shape: { targetInstance, params } (fingerprint or instanceId in targetInstance)
    • Response: { success, data | error }

Instance Management

  • GET /api/cloud/instances: List all instances accessible to the authenticated user (owned and shared).
  • POST /api/cloud/instance/:id/command: Execute a command on a specific instance.
  • GET /api/cloud/instance/:id/info: Get detailed system information from an instance.
  • POST /api/cloud/instance/:id/disconnect: Forcibly disconnect an instance.

File Operations

  • GET /api/cloud/instance/:id/browse: Browse the filesystem of an instance.
  • POST /api/cloud/sync/cross-instance: Perform a sync operation between two instances.

Sharing

  • GET /api/cloud/shares/list: List all shares created by or shared with the user.
  • POST /api/cloud/shares/save: Create or update a share.
  • POST /api/cloud/shares/delete: Delete/revoke a share.

Active Mirrors

  • GET /api/cloud/mirrors/list: List the user's active mirrors.
  • POST /api/cloud/mirrors/save: Create or update an active mirror.
  • POST /api/cloud/mirrors/delete: Delete an active mirror.

Additional REST Endpoints

  • Instances and health

    • POST /api/cloud/instances/refresh: Force verification sweep and return latest states
    • GET /api/cloud/instance/:id/status: Fresh ping/health for a single instance
    • GET /api/cloud/mcp-connections: List MCP client connections and controllable instances for the user
  • File browse and transfer

    • GET /api/cloud/instance/:id/folders: List known project folders on an instance
    • GET /api/cloud/instance/:id/browse: Server-side directory listing with metadata
    • POST /api/cloud/instance/:id/transfer/start: Start a chunked transfer session
    • GET /api/cloud/instance/:id/transfer/:transferId/status: Query transfer state
    • POST /api/cloud/instance/:id/transfer/:transferId/cancel: Cancel a transfer
  • Synchronization

    • POST /api/cloud/instance/:id/sync/enhanced: One-shot enhanced file sync for an instance
    • POST /api/cloud/instance/:id/sync/folder: Folder sync via streaming/chunked pipeline
    • POST /api/cloud/sync/compare-cross-instance: Server-side comparison between two instances
    • POST /api/cloud/sync/execute-from-comparison: Execute transfers from a prior comparison result
    • POST /api/cloud/sync/cross-instance: Streaming cross-instance sync orchestration
    • GET /api/cloud/sync/activity: Placeholder for historical activity (real-time via sockets)
  • Shares

    • GET /api/cloud/shares/list
    • POST /api/cloud/shares/save
    • POST /api/cloud/shares/delete
  • Push Mirrors (Active Mirrors)

    • GET /api/cloud/mirrors/list
    • POST /api/cloud/mirrors/save
    • POST /api/cloud/mirrors/delete
    • POST /api/cloud/mirrors/sync
    • POST /api/cloud/mirrors/check
  • Pull Mirrors

    • GET /api/cloud/pulls/list
    • POST /api/cloud/pulls/save
    • POST /api/cloud/pulls/delete
    • POST /api/cloud/pulls/sync
    • POST /api/cloud/pulls/check
  • Authentication (Anchoring)

    • POST /api/cloud/request-anchor-token: Exchange dashboard credentials for short-lived siteAuthToken
    • POST /api/cloud/request-anchor-token-as: Same, on behalf of a target user (requires permission)

Permissions & Security

  • All HTTP/Socket operations use session auth or JWT (siteAuthToken).
  • Access control is contextual with await Doh.permit(req.user, 'action:context', runtimeContext).
    • Examples: read:cloud_instance, command:cloud_instance with the target instance object context.
  • Shares extend access across users with optional expiration and scoped permissions.

Example Requests

  • List instances

    GET /api/cloud/instances
    Authorization: session cookie
    
    { "success": true, "instances": [], "stats": { "total": 0 } }
    
  • Request anchor token (for doh cloud anchor)

    POST /api/cloud/request-anchor-token
    Content-Type: application/json
    
    { "username": "admin@example.com", "password": "...", "instance_info": { "fingerprint": "fp-123" } }
    
    { "success": true, "siteAuthToken": "<jwt>", "expires": 3600 }
    
  • Cross-instance sync (server-side orchestration)

    POST /api/cloud/sync/cross-instance
    Content-Type: application/json
    
    {
      "sourceInstanceId": "fp-src",
      "destinationInstanceId": "fp-dst",
      "sourcePath": "/var/www/app",
      "targetPath": "/var/www/app",
      "options": { "exclude": ["node_modules/**", ".doh/**"] }
    }
    
    { "success": true, "operationId": "op-789" }
    

Authentication

  • POST /api/cloud/request-anchor-token: Request a site authentication token for an instance.
  • POST /api/cloud/request-anchor-token-as: Request a token on behalf of another user (requires special permissions).

Database Schema

Cloud Manager automatically creates and manages the following tables in the application's primary database.

cloud_shares

CREATE TABLE cloud_shares (
  id TEXT PRIMARY KEY,              -- Unique identifier
  sharerId TEXT NOT NULL,           -- Instance owner's user ID
  shareeId TEXT NOT NULL,           -- User ID receiving access
  instanceId TEXT,                  -- Specific instance ID or NULL for all
  permissions TEXT,                 -- JSON array of permissions or NULL for all
  description TEXT,                 -- Human-readable description
  createdTimestamp TEXT NOT NULL,   -- Creation date
  expiresTimestamp TEXT,            -- Optional expiration
  isEnabled INTEGER DEFAULT 1       -- Enable/disable flag
);

active_mirrors

CREATE TABLE active_mirrors (
  id TEXT PRIMARY KEY,
  userId TEXT NOT NULL,
  sourceInstanceId TEXT NOT NULL,
  destinationInstanceId TEXT NOT NULL,
  sourcePath TEXT NOT NULL,
  targetPath TEXT NOT NULL,
  type TEXT NOT NULL,
  createdTimestamp TEXT,
  lastCheckTimestamp TEXT,
  lastSyncTimestamp TEXT,
  needsSync INTEGER,
  isEnabled INTEGER DEFAULT 1
);
Last updated: 10/22/2025