Auto-Packager: Automated Dependencies

Auto-Packager

The Auto-Packager is the intelligent backbone of the Doh framework's load system. It automatically analyzes your codebase to:

Discover and register modules, packages, and patterns
Manage dependencies between components
Generate essential manifests for the framework
Create distributable packages (Dohballs)
Handle ESM imports for Node.js and browser environments
Optimize performance through caching and parallel processing

With minimal configuration, the Auto-Packager ensures your Doh application's components are properly linked, versioned, cached, and ready for distribution.

Note: For a higher-level view of how the Auto-Packager interacts with DohPath, Manifests, and the runtime Load System, see the Doh.js Core Architecture Overview.

Key Concepts

Packages

Packages are the fundamental units of code organization in Doh. They have:

A unique name identifier
A set of load instructions (dependencies)
Optional installation requirements
Associated metadata

Packages can be defined in JavaScript files using Doh.Package() or in YAML files with .doh.yaml extension.

Modules

Modules are executable units of code in Doh, defined using Doh.Module(). A module typically:

Has a unique name
Contains a set of patterns
A set of load instructions (dependencies)
Has a callback function that executes when loaded

Patterns

Patterns define reusable components within modules. They support inheritance and can be used to build complex UI and business logic components.

Dohballs

Dohballs are versioned, self-contained distribution units that package related Doh components together. They:

Can be generated by the Auto-Packager
Use content-based versioning (via dohball.json within the source folder) to ensure consistency and trigger rebuilds
Can be hosted and shared between Doh applications
Can be baked in parallel for faster builds

Environment Branching Handling

The Auto-Packager intelligently processes environment-specific code defined through conditional decorators in load statements. Here's how it works:

1. Conditional Load Statement Processing

When the Auto-Packager encounters conditional load statements (e.g., browser?? feature_ui or nodejs?? server_feature), it:

Parses and extracts the condition and the actual dependency
Records both in the dependency graph, marking the dependency as conditional
Includes the dependency in appropriate manifests, maintaining the condition
Ensures that even conditional dependencies are properly resolved and tracked

Example:

// Auto-Packager processes this structure while preserving conditions
Doh.Package('feature', {
  load: [
    'core',                    // Always loaded
    'browser?? ui_components', // Browser-only
    'nodejs?? server_api'      // Node.js-only
  ]
});

2. Environment-Specific Dependency Graphs

The Auto-Packager builds separate dependency sub-graphs for different environments:

It identifies environment-specific branches in the dependency tree
Validates that environment-specific dependencies don't create cross-environment conflicts
Ensures that conditional dependencies only affect their intended environments

3. Bifurcated Module Pattern Support

The Auto-Packager is optimized to handle the common bifurcated module pattern:

// Main package with environment branching
Doh.Package('feature_name', {
  load: [
    'feature_name_common',         // Common functionality
    'browser?? feature_name_ui',    // Browser-specific implementation
    'nodejs?? feature_name_server'  // Node.js-specific implementation
  ]
});

The Auto-Packager ensures that:

The common module is properly loaded in all environments
Browser-specific modules are only included in browser bundles
Server-specific modules are only included in Node.js bundles
Dependency ordering is maintained correctly across environments

Using the Auto-Packager

Automatic Operation

The Auto-Packager runs automatically when Doh boots, unless run with no-pack (doh run no-pack). It:

Scans your project directories
Processes JavaScript and YAML files
Extracts Doh-specific declarations
Generates necessary manifests

Manual Triggering

You can manually trigger the Auto-Packager using the Doh CLI:

# Run the auto-packager to update manifests
doh update

# Upgrade all Dohballs from configured hosts
doh upgrade

# Bake specific or all locally hosted Dohballs
# Use no argument to bake all eligible packages
# Specify package names to bake only those
doh bake [package_name...]

# Force rebake specific or all locally hosted Dohballs
doh rebake [package_name...]

Configuration

The Auto-Packager's behavior is controlled through the pod.yaml file:

# Update behavior controls
always_reinstall_dohballs: false
always_upgrade_dohballs: false
always_restore_dohballs: false
always_update_npm_dependencies: false

# Dohball generation configuration
dohball_deployment:
  expose_packages: '*'     # Glob pattern or list of packages to expose as Dohballs ('*' for all eligible)
  expose_doh_js: false     # Whether to include doh_js in Dohballs (rarely needed)
  ignore_packages:         # List of packages to never expose as Dohballs
    - private_package
  ignore_paths:            # List of directory paths (relative to root) to never include in Dohballs
    - secret_folder
    - build_output
  rebake: false            # Force rebake on every run (useful for debugging)
  compile_manifest: true   # Generate a dohball_manifest.json for hosting baked dohballs
  worker_pool_size: 4      # Number of parallel workers for baking Dohballs

# ESM Import handling via esm.sh
esm_sh_external: []       # List of packages to exclude from esm.sh auto-bundling (e.g., ['react', 'react-dom'])
always_esbuild: false     # Whether to always run the esbuild processor after packaging

# Dohball hosts for fetching remote packages
dohball_host:
  - https://main.dohball.host
  - https://backup.dohball.host

# Packages to load when the host is booted (auto run)
host_load:
  - core_module
  - ui_module

# Files/folders to ignore during scanning
packager_ignore:
  directories:
    # DEFAULTS:
    - node_modules
    - .git
    - .doh # Auto-ignored, but good practice to list
    - dohballs # Auto-ignored
    - dist # Often contains build artifacts not part of source

File Processing

JavaScript Files

The Auto-Packager parses JavaScript files looking for Doh-specific declarations:

// Module definition
Doh.Module('myModule', ['dep1', 'dep2'], function(Pattern) {
  // Module implementation

  // Pattern definition
  Pattern('MyPattern', ['InheritsFrom'], {
    // Pattern implementation
  });
});

// Package definition (repacking an npm module as a Doh module of the same name)
Doh.Package('dep2', {
  install: ['npm:dep2'],
  load: ['dep1', 'global import * as dp2 from "dep2"'],
});


// CLI command registration
Doh.CLI('myModule', {
  'command-name': {
    file: 'path/to/script.js', // dynamically imported when command is called
    help: 'help displayed in `doh help` for this command'
  }
});

// Installation instructions
Doh.Install('myModule', {
  'npm:axios': 'latest'
});

// Pod configuration
Doh.Pod('myModule', {
  // Pod configuration outer scope
  // add keys for your module as needed
  myModule: {
    // your settings
  }
});

YAML Files

The Auto-Packager also processes .doh.yaml files for simpler package definitions:

# Simple definition with just a load dependency
simple_package: dependency_package

# Complex definition with multiple options
complex_package:
  load: 
    - import YAML from "yaml"
    - import Axios from "axios"
  install:
    'npm:yaml': '' # empty value for latest
    'npm:axios': ''

Parameter Resolution and Module Communication

The Auto-Packager plays a crucial role in enabling module communication through parameter resolution:

For each Doh.Module() declaration, it extracts the callback function's parameter names
These parameters are recorded in the module's metadata
When a module is loaded, the parameters are resolved from Doh.Globals or created if they don't exist
The Auto-Packager ensures that modules sharing the same parameter name receive the same object instance

This parameter resolution system is what enables modules to communicate without explicit import/export statements:

// The Auto-Packager sees both modules use 'SharedData' parameter
// and ensures they receive the same object instance

Doh.Module('ModuleA', function(SharedData) {
  // Initialize the shared object
  SharedData.counter = 0;
  SharedData.increment = function() {
    SharedData.counter++;
  };
});

Doh.Module('ModuleB', ['ModuleA'], function(SharedData) {
  // Use the shared object initialized by ModuleA
  SharedData.increment();
  console.log(SharedData.counter); // Outputs: 1
});

Dependency Management

The Auto-Packager builds and validates the dependency graph of your application:

Expands dependency trees to include transitive dependencies
Detects and reports cyclic dependencies
Resolves conflicts between multiple versions
Manages both local and remote dependencies

When a cyclic dependency is detected, the Auto-Packager will report it and fail, preventing potentially problematic code from running.

Environment-Specific Dependencies

The Auto-Packager handles environment-specific dependencies by:

Parsing conditional decorators in load statements (e.g., browser??, nodejs??)
Building separate dependency graphs for different environments
Ensuring that environment-specific code is only included in the appropriate bundles
Validating that environment-specific dependencies don't create conflicts across environments

Example:```javascript // The Auto-Packager will ensure ui-lib is only included in browser bundles // and fs-extra is only included in Node.js bundles Doh.Module('FeatureModule', [ 'common-lib', // Included in all environments 'browser?? ui-lib', // Only included in browser bundles 'nodejs?? server-utilities', // Only included in Node.js bundles 'Doh.pod.debug?? debug-tools' // Only included if Doh.pod.debug is true ], function() { // Module implementation });


## Manifest Generation

The Auto-Packager generates several important [manifest files](/docs/core/manifests) in two primary locations:

### Private Manifests (`/.doh/manifests/`)

These manifests contain internal build information, caches, and potentially sensitive configuration. They are generally not distributed and are excluded from being served by all Doh systems by default.

- **CLI Manifest** (`cli_manifest.json`): Maps registered [CLI](/docs/tools/cli) commands to their implementation files.
- **Pod Manifest** (`pod_manifest.json`): Aggregated [Pod](/docs/core/pods) configurations from `Doh.Pod()` declarations.
- **Node ESM Manifest** (`node_esm_manifest.json`): Import map for Node.js environments, mapping package names to their local paths in `node_modules`. (CURRENTLY UNUSED)
- **AP Cache** (`ap_cache.json`): Auto-Packager file cache. Stores file metadata (mtime, hash), parsed Doh definitions, and import information to speed up subsequent runs.
- **Autopackager Problems** (`autopackager_problems.json`): Logs errors encountered during packaging.
- **Pattern Duplicates** (`pattern_duplicates.json`): Lists patterns defined in multiple files.
- **Deprecated Features** (`deprecated_features.json`): Logs usage of deprecated Doh features.
- **Compiled Dohball Manifest** (`compiled_dohball_manifest.json`): Internal cache of manifests fetched from remote Dohball hosts.

### Public Manifests (`/doh_js/manifests/`)

These manifests describe the public structure and dependencies of your application and are often used by the framework at runtime.

- **Package Manifest** (`package_manifest.json`): The central manifest defining all discovered [packages](/docs/core/packages), their dependencies (`load`), files, and associated metadata.
- **Pattern Manifest** (`patterns_manifest.json`): Maps [pattern](/docs/patterns/patterns) names to the module they belong to.
- **Core Patterns Manifest** (`core_patterns_manifest.json`): A list of core Doh patterns provided by the framework itself.
- **Assets Manifest** (`assets_manifest.json`): Maps modules to the static assets (`DohPath.DohSlash()` calls) they reference.
- **Browser ESM Manifest** (`browser_esm_manifest.json`): Import map for browser environments, mapping package names to CDN URLs (typically `esm.sh`).
- **Dohball Manifest** (`dohball_manifest.json`): If `compile_manifest` is true, this lists the locally baked [Dohballs](/docs/core/dohballs) available for hosting, including version and update time. Used by remote Doh instances to discover available Dohballs on this host.
- **Module Dependency Graph** (`module_dep_graph.json`): Represents the dependency relationships between packages/modules.

### Dohball Tracking Manifest (`/dohballs/dohballs.json`)

- **Orphaned Dohballs** (`dohballs.json`): Located in the root of the Dohball hosting directory (`/dohballs/`). Tracks Dohball archives (`.tar.gz` files) that exist in the hosting directory but no longer correspond to an actively exposed package according to the current `pod.yaml` configuration. This helps identify and manage potentially obsolete Dohballs.

## ESM Import Management

The Auto-Packager automatically manages [ESM imports](/docs/core/load) for NPM packages used in your Doh code:

1.  Scans JavaScript files for `import` statements and `Doh.load()` calls referencing external packages.
2.  Resolves the package path and version using `node_modules`.
3.  **For Node.js:** Generates entries in `node_esm_manifest.json` mapping the package name to its local path.
4.  **For Browsers:** Generates browser-compatible URLs using available `node_modules`, otherwise `esm.sh` (e.g., `https://esm.sh/react@18`) and adds them to `browser_esm_manifest.json`. Packages listed in `pod.yaml` under `esm_sh_external` are excluded from `esm.sh` bundling.
5.  These manifests are used to create import maps, allowing the same import syntax (e.g., `import React from 'react'`) to work seamlessly in both environments.

## Environment Branching Support

The Auto-Packager has special handling for environment-specific code:

### 1. Conditional Load Statements

Load statements with environment conditions (`browser??`, `nodejs??`, etc.) are processed to ensure they only apply in the appropriate environments:

```javascript
// Auto-Packager processes these conditionals correctly
Doh.Module('CrossPlatformFeature', [
  'core-dependency',                // Always included
  'browser?? dom-utilities',         // Only included in browser bundles
  'nodejs?? server-utilities',       // Only included in Node.js bundles
  'Doh.pod.debug?? debug-tools'       // Only included if Doh.pod.debug is true
], function() {
  // Module implementation
});

2. Bifurcated Module Pattern

The Auto-Packager recognizes and optimizes for the common bifurcated module pattern:

// Main package with environment-specific branch loading
Doh.Package('feature_name', {
  load: [
    'feature_name_common',
    'browser?? feature_name_ui',
    'nodejs?? feature_name_server'
  ]
});

// Common shared functionality
Doh.Module('feature_name_common', [], function() {
  // Common implementation
});

// Browser UI implementation
Doh.Module('feature_name_ui', ['feature_name_common'], function() {
  // Browser-specific implementation
});

// Server API implementation
Doh.Module('feature_name_server', ['feature_name_common'], function() {
  // Server-specific implementation
});

The Auto-Packager ensures that:

Dependencies are correctly tracked across environment branches
Common code is properly shared between environments
Environment-specific code is only included when needed
The dependency graph remains acyclic across all environments

Dohball Creation

Dohballs are created ("baked") based on the dohball_deployment configuration in pod.yaml:

Packages eligible for exposure are identified based on expose_packages, ignore_packages, and ignore_paths.
If specific package names are provided via doh bake [names...] or doh rebake [names...], the list is filtered.
Packages are grouped by their shared parent directory path (e.g., all packages under /modules/user/ might be baked into /dohballs/modules/user.tar.gz).
The Auto-Packager checks for a dohball.json file within the source directory (e.g., /modules/user/dohball.json). The version field in this file determines the Dohball version.
The packer compares the current source content hash against the hash stored in the cache (ap_cache.json) for the last bake. A Dohball is only baked if the content has changed or if rebake is forced (doh rebake or pod.yaml).
The baking process (creating the .tar.gz archive) is parallelized using a worker pool (size configured by worker_pool_size).
The resulting archive is placed in the /dohballs/ directory, mirroring the source structure (e.g., /modules/user/ becomes /dohballs/modules/user.tar.gz).
If compile_manifest is true, /doh_js/manifests/dohball_manifest.json is updated with information about the baked Dohball (version, updated time).
The /dohballs/dohballs.json manifest is updated to track any orphaned Dohballs (existing archives not matching current exposed packages).

Performance Optimization

The Auto-Packager employs several techniques to ensure speed:

Caching

Utilizes /.doh/manifests/ap_cache.json to store file modification times, content hashes, and parsed Doh definitions (modules, packages, patterns, imports). On subsequent runs, only files that have changed since the last run are re-parsed, significantly speeding up the process.

Parallel Processing

Dohball creation (bake and rebake) uses a configurable worker pool (worker_pool_size in pod.yaml) to perform the CPU-intensive tasks of hashing, compressing, and archiving files in parallel.

Selective Scanning

The Auto-Packager can be configured to ignore specific directories to reduce unnecessary file processing.

Integration with Other Doh Systems

Pod System

The Auto-Packager integrates with the Pod system by:

Reading configuration from pod.yaml
Generating pod manifests
Processing Doh.Pod() declarations

Module System

Integration with the Module system includes:

Processing Doh.Module() definitions
Resolving module dependencies
Generating module-related manifests

Load System

The Auto-Packager uses the Load System for:

Parsing load statements and decorators
Handling conditional loading (browser??, nodejs??, etc.)
Resolving relative and absolute paths
Managing ES module imports

Pattern System

The Auto-Packager interacts with the Pattern system by:

Extracting pattern definitions
Generating pattern manifests
Tracking pattern inheritance

Real-World Examples

1. Cross-Platform Feature with Shared Code

// Define a cross-platform feature with environment branching
Doh.Package('user_management', {
  load: [
    // Common functionality
    'user_management_common',
    // Environment-specific implementations
    'browser?? user_management_ui',
    'nodejs?? user_management_server'
  ]
});

// The Auto-Packager:
// 1. Detects the environment branching pattern
// 2. Tracks dependencies for each branch
// 3. Includes common code in all environments
// 4. Bundles UI code only for browsers
// 5. Bundles server code only for Node.js

2. Configuration-Based Loading

// Define a module with configuration-based features
Doh.Module('admin_panel', [
  // Base functionality
  'admin_core',
  
  // Conditional features based on configuration
  'Doh.pod.features.analytics?? admin_analytics',
  'Doh.pod.features.permissions?? admin_permissions',
  
  // Environment-specific and config-specific
  'browser&&Doh.pod.features.dashboard?? admin_dashboard'
], function() {
  // Module implementation
});

// The Auto-Packager:
// 1. Parses the complex conditions
// 2. Tracks configuration-dependent features
// 3. Handles combined environment and config conditions

Debug Information

The Auto-Packager generates several diagnostic files within /.doh/manifests/:

autopackager_problems.json: Logs errors or inconsistencies found during scanning and packaging.
pattern_duplicates.json: Lists any patterns defined with the same name in multiple files.
deprecated_features.json: Records usage of deprecated Doh functions or parameters.
dohballs.json (in /dohballs/): Tracks orphaned Dohball archives.

Troubleshooting Common Issues

1. Cyclic Dependencies

If the Auto-Packager detects cyclic dependencies, it will report an error. To fix:

Identify the cycle in the dependency graph
Refactor one of the modules to break the cycle:
- Move shared functionality to a common dependency
- Use events or callbacks instead of direct dependencies
- Use dynamic loading with Doh.load() instead of static dependencies

2. Missing Modules

If a module is referenced but not found:

Check the module name and path
Ensure the module is properly defined with Doh.Module()
Check for typos in load statements
Run doh update to refresh manifests

3. Environment-Specific Issues

If environment-specific code is not working correctly:

Verify conditional decorators (browser??, nodejs??) are properly applied
Check that environment detection is working (IsBrowser(), IsNode())
Ensure common modules are loaded before environment-specific ones
Verify the dependency order in bifurcated module patterns

Best Practices

Organize Related Components
- Keep related packages, modules, and patterns in the same directory
- This optimizes Dohball generation and improves load performance
Use Environment Branching Consistently
- Follow the bifurcated pattern (common + environment-specific modules)
- Keep environment-specific code clearly separated with conditional decorators
Avoid Cyclic Dependencies
- Design your module hierarchy to avoid circular dependencies
- Use events or interfaces to break cycles when needed
Leverage Parameter Sharing
- Use consistent parameter names for modules that need to share state
- Document shared parameters and their expected structure
Regular Updates
- Run doh update when adding new components
- This ensures all manifests stay in sync with your code
Manifest Awareness
- Understand the manifests that describe your application
- Use them for debugging and comprehending your application structure
Maintain Clean Dependencies
- Periodically review your dependencies
- Remove unused packages to keep your application lean

Command-Line Reference

# Run the auto-packager to update manifests and perform checks
doh update

# Bake Dohballs whose content has changed since the last bake.
# Can optionally specify package names to only check those.
doh bake [package_name...]

# Force rebaking of Dohballs, regardless of content changes.
# Can optionally specify package names to only rebake those.
doh rebake [package_name...]

# Upgrade installed Dohballs from remote hosts
doh upgrade

API Reference

Core Functions

Doh.Module(name, dependencies, callback): Define a module
Doh.Package(name, config): Define a package
Pattern(name, inherits, idea): Define a pattern in a Doh module
Doh.Install(name, assets, callback): Define installation requirements
Doh.CLI(package, commands): Register CLI commands
Doh.Pod(package, config): Register pod configuration

Advanced Features

Load decorators: See Load System for advanced import syntax (global, direct, etc.)
Path Substitution: Use ^/ in paths within load arrays for references relative to the file's location (see DohPath)