Chat Files

Peer-to-peer file transfer with WebRTC-first delivery and socket relay fallback.

Overview

chat_files provides file sharing within chat rooms. Files transfer directly between peers via WebRTC data channels when possible, falling back to a server-mediated socket relay when P2P connections can't be established. Files are cached locally in IndexedDB, and users who have downloaded a file become seeders for other peers. The module handles thumbnails for images, inline previews for media/PDF/text, and synchronized media playback across room participants.

When server seeding is enabled (the default), the uploader can also push files to the server for persistent availability. Users claim permanent server storage within the temp-cache window (infinite by default, configurable via temp_cache_minutes) and are charged against their tier quota.

Architecture

chat_files_server (nodejs)     chat_files_client (browser)
├── File metadata DB           ├── StorageMirrorRegistry (plugin system)
├── Seeder registry            ├── TransferManager (orchestration)
├── Socket signal relay        ├── FileCache (IndexedDB)
├── Relay chunk routing        ├── ChunkAccumulator
├── Server seeding (multer)    ├── ChatFileCard (UI)
└── Tier quota enforcement     └── WebRTC DataChannel

Loaded by: chat.doh.yaml (no standalone package — part of core chat) Depends on: chat_server (rooms, messages, membership)

File Structure

Configuration

# pod.yaml override example
chat_files:
  max_file_size: 1073741824   # 1GB
  cache_max_bytes: 1000       # 1GB IndexedDB cache
  ice_servers:
    - urls: 'stun:stun.example.com:3478'
    - urls: 'turn:turn.example.com:3478'
      username: 'user'
      credential: 'pass'

Server Seeding Configuration

The chat_files.server_seeding block controls the server-side file storage system. All sub-keys are optional and have defaults.

Retention policy: Permanently stored files (those the user explicitly claimed via store-permanently) are never auto-deleted. When temp_cache_minutes is set to a number, unclaimed temp-cache files are deleted after that window. When temp_cache_minutes is false (the default), unclaimed files persist indefinitely unless the server cap forces eviction (oldest-first among unclaimed files). Users release their own permanent storage via POST /api/chat/files/remove-from-server. If the server cap is reached and all remaining files are permanently claimed, new temp uploads are blocked until space is freed.

# pod.yaml
chat_files:
  server_seeding:
    enabled: true
    storage_path: '/uploads/chat_files'
    max_total_storage_bytes: 10737418240
    temp_cache_minutes: false       # false = infinite (no expiry); set to a number for auto-cleanup
    cleanup_interval_hours: 1
    allowed_mime_types: null    # or ['image/png', 'image/jpeg', 'video/mp4']

Upload Tier Quotas

The chat_files.upload_tiers block maps permission group names to per-user permanent storage limits. The default key applies to all users not in a named group.

Enforcement happens at the POST /api/chat/files/store-permanently route. The server checks the user's total_stored_bytes (tracked in chat.file_user_status) against their tier limit before accepting a claim. If the new file would exceed the limit, the route returns HTTP 403 with a message showing current usage and the tier cap.

To configure custom tiers, add new group-name keys to upload_tiers. The tier name must match an existing permission group name:

# pod.yaml
chat_files:
  upload_tiers:
    default: 104857600       # 100MB
    doh_pro: 5368709120      # 5GB
    enterprise: 107374182400 # 100GB — requires 'enterprise' permission group

Database Schema

File metadata shape (in chat.files):

{
  id: string,            // UUID
  room_id: string,
  name: string,          // Original filename
  size: number,          // Bytes
  mime_type: string,
  chunk_size: number,
  total_chunks: number,
  thumbnail: string,     // Base64 data URL (images only, capped at 50KB) or null
  sender: string,        // Username
  created_at: number,    // Timestamp
  md5: string,           // MD5 hash (set after server upload)
  sanitized_name: string // Filesystem-safe name (set after server upload)
}

Seeder record shape (in chat.file_seeds):

{
  id: string,            // "{file_id}:{username}" or "{file_id}:__server__"
  file_id: string,
  username: string,      // "__server__" for server-side seeder
  seeded_at: number,
  // Server seeder additional fields:
  stored: boolean,
  permanent: boolean,    // true once a user claims permanent storage
  cached_at: number,
  size: number,
  md5: string,
  storage_file_id: string  // Physical file ID (may differ from file_id after dedup)
}

Storage claim shape (in chat.file_storage_claims):

{
  id: string,            // "{file_id}:{username}"
  file_id: string,
  username: string,
  claimed_at: number,
  size: number
}

File message format (in chat.messages.content):

[FILE]{"file_id":"...","name":"...","size":123,"mime_type":"...","thumbnail":"..."}

REST API Reference

GET /api/chat/files/user

// Query params
{ room_id?: string, search?: string }
// Response
{
  files: [{
    id, name, size, mime_type, thumbnail, created_at,
    room_id, room_name, room_type, sender_chat_name
  }]
}

POST /api/chat/files/upload

Must be a real HTTP request (not socket). The global Express middleware must have parsed the multipart body. The file is written to STORAGE_PATH/{file_id} and registered as a __server__ seeder.

// Multipart form fields
file_id: string   // UUID from prior chat:file:share socket event
file: File        // Binary file data

// Response
{ success: true, file_id, expires_at: number, deduped: boolean }

MD5 deduplication is applied: if the server already holds a file with the same hash, the new upload is discarded and the existing physical file is reused.

POST /api/chat/files/store-permanently

// Request
{ file_id: string }
// Response (success)
{ success: true, total_stored_bytes, tier_limit, tier_name }
// Response (quota exceeded, 403)
{ error: "Storage limit exceeded. Using 95MB of 100MB (Free tier)." }

GET /api/chat/files/user/storage

// Response
{
  total_stored_bytes: number,
  tier_name: string,          // 'Free', 'doh_pro', etc.
  tier_limit: number,
  claimed_files: [{
    file_id, name, size, room_id, created_at, claimed_at
  }]
}

Socket Events Reference

File Sharing

Side effects: creates chat.files entry, registers sender as seeder, creates [FILE] message, broadcasts chat:message_received.

Peer Discovery

chat:file:request_peers — Full peer discovery: queries all seeds for the file, fetches live room sockets, resolves which seeders are currently online, checks server availability and user claim. Used by TransferManager when initiating a download.

chat:file:server_status — Lightweight server-only check: two targeted ID lookups (FILE_SEEDS for {file_id}:__server__ and FILE_STORAGE_CLAIMS for {file_id}:{username}). No socket fetch, no peer resolution. Used by ChatFileCard._setupServerStorageUI() at card init to populate the server storage UI without triggering a full peer query per file on room load.

WebRTC Signaling

signal_type: 'offer' | 'answer' | 'ice-candidate'

Socket Relay Transfer

Relay chunks are base64-encoded. Acks sent every relay_batch_size (8) chunks.

Seeding & Sync

action: 'play' | 'pause' | 'seek'

Transfer Flow

1. Check local memory (localFiles Map)
   └─ Found → complete (instant)

2. Check IndexedDB cache (FileCache)
   └─ Found → complete, register as seeder

3. Probe room peers (chat:file:probe, 3s timeout)
   └─ Peer responds → attempt transfer

4. Query seed registry (chat:file:request_peers)
   └─ Seeders found → attempt transfer

5. Wait passively (60s) for new seeders
   └─ chat:file:seeder_available / chat:room_peer_joined
   └─ Timeout → 'unavailable'

Transfer attempt:
  a. Try WebRTC DataChannel (10s timeout)
     └─ Success → complete, cache, register seeder
     └─ Fail → try next method

  b. Try socket relay (base64 chunks, batched with ack)
     └─ Success → complete, cache, register seeder
     └─ Fail → try next seeder, then give up

Client Architecture

StorageMirrorRegistry

A plugin system for storage backends. Each plugin tracks whether a file is stored and provides context menu actions. Plugins are registered via StorageMirrorRegistry.register(plugin).

Built-in plugins:

Plugins expose isStored(fileId), queryStatus(fileId), and getContextMenuItems(fileId, card) for UI integration. ChatFileCard uses the registry to render storage indicator icons and context menu actions.

TransferManager (Singleton)

Orchestrates all file transfers. Key methods:

FileCache (IndexedDB)

DB: doh_chat_files, store: files, indices: room_id, cached_at

ChunkAccumulator

Efficiently accumulates binary chunks, merging to Blob at the blob_merge_threshold to prevent memory pressure on large files.

ChatFileCard Pattern

Sub-objects: thumbEl, info (nameEl, metaEl), progressBar (fill), actionArea (downloadBtn, toggleBtn, cancelBtn, statusEl), previewArea, resizeHandle, serverStorageArea (storeBtn, storedIndicator, removeBtn)

Supports inline preview for images, video, audio, PDF, and text files. Previews are resizable with cross-peer size synchronization.

Server storage UI lifecycle (non-browse mode, server_seeding_enabled only):

1. post_builder_phase calls _setupServerStorageUI()
2. Subscribes to chat:file:server_cached broadcasts (updates UI if server caches this file while card is live)
3. Emits chat:file:server_status once to get current server_available, user_has_claim, and expires_at — populates initial UI state

Files browser view (ChatRoomFileViewer in chat_client.js) renders cards with _browseMode: true, which skips all of the above. It determines server storage state separately: a single GET /api/chat/files/user/storage call returns the user's claimed_files array, from which it builds a _serverClaims map used for filtering and the "saved on server" indicator. Only permanently claimed files appear in this view — temp-cached-only files do not.

Per-window state: ChatRoomFileViewer accepts a _storagePrefix property (default 'chat-files') used as the localStorage key prefix for view mode, type filters, and storage filters. FilesDohtopContent sets this to 'chat-files-<windowId>' so each Files DohtopWindow instance has independent filter state.

Orphaned files: /api/chat/files/user also returns files uploaded by the current user whose room has been deleted. These are enriched with room_type: 'deleted' and room_deleted: true. The file browser groups them under a collapsed "Deleted Rooms" folder in the sidebar.

Room type enrichment: File responses include room_type (from the room object) and partner_username (for DM rooms), enabling the files sidebar to render proper room icons (format icons, DM avatars, bot gear, issue bug) via ChatRoomStore and ChatUserStore.

Testing

1. Share a file in a chat room — verify [FILE] message appears
2. Open a second browser/tab in the same room — download the file
3. Verify WebRTC transfer (check console for "WebRTC" transport label)
4. Test with WebRTC blocked (e.g., restrictive NAT) — verify socket relay fallback
5. Close all tabs, reopen — verify cached file loads from IndexedDB
6. Test media sync: play a video in one tab, verify playback syncs to other tabs
7. Test server seeding: upload a file, verify POST /api/chat/files/upload succeeds, then call store-permanently and check /api/chat/files/user/storage reflects the new total
8. Verify quota enforcement: attempt to permanently store a file that would exceed the tier limit — expect HTTP 403 with usage message