The Auto-Packager is the intelligent backbone of the Doh framework's load system. It automatically analyzes your codebase to:
With minimal configuration, the Auto-Packager ensures your Doh application's components are properly linked, versioned, cached, and ready for distribution.
Note: For a higher-level view of how the Auto-Packager interacts with DohPath, Manifests, and the runtime Load System, see the Doh.js Core Architecture Overview.
Packages are the fundamental units of code organization in Doh. They have:
Packages can be defined in JavaScript files using Doh.Package()
or in YAML files with .doh.yaml
extension. Doh.Package()
is a compile-time declaration the Auto-Packager scans from .js
files to collect JSON-like metadata (name, load, install, pod, cli). It does not reference modules or attach callbacks.
Modules are executable units of code in Doh, defined using Doh.Module()
. A module is a package with a callback, created atomically under the same name in Doh.Packages
. Do not pair a separate Doh.Package(name, ...)
with a later Doh.Module(name, ...)
for the same name; overlapping declarations are treated as conflicts by the scanner. A module typically:
Patterns define reusable components within modules. They support inheritance and can be used to build complex UI and business logic components.
Dohballs are versioned, self-contained distribution units that package related Doh components together. They:
dohball.json
within the source folder) to ensure consistency and trigger rebuildsdohball_host
moduleThe Auto-Packager intelligently processes environment-specific code defined through conditional decorators in load statements. Here's how it works:
When the Auto-Packager encounters conditional load statements (e.g., browser?? feature_ui
or nodejs?? server_feature
), it:
Example:
// Auto-Packager processes this structure while preserving conditions
Doh.Package('feature', {
load: [
'core', // Always loaded
'browser?? ui_components', // Browser-only
'nodejs?? server_api' // Node.js-only
]
});
The Auto-Packager builds separate dependency sub-graphs for different environments:
The Auto-Packager is optimized to handle the common bifurcated module pattern:
// Main package with environment branching
Doh.Package('feature_name', {
load: [
'feature_name_common', // Common functionality
'browser?? feature_name_ui', // Browser-specific implementation
'nodejs?? feature_name_server' // Node.js-specific implementation
]
});
The Auto-Packager ensures that:
The Auto-Packager runs automatically when Doh boots, unless run with no-pack (doh run no-pack
). It:
You can manually trigger the Auto-Packager using the Doh CLI:
# Run the auto-packager to update manifests
doh update
# Upgrade all Dohballs from configured hosts
doh upgrade
# Bake specific or all locally hosted Dohballs
# Use no argument to bake all eligible packages
# Specify package names to bake only those
doh bake [package_name...]
# Force rebake specific or all locally hosted Dohballs
doh rebake [package_name...]
The Auto-Packager's behavior is controlled through the pod.yaml
file:
# Update behavior controls
always_reinstall_dohballs: false
always_upgrade_dohballs: false
always_restore_dohballs: false
always_update_npm_dependencies: false
# Dohball generation configuration
dohball_deployment:
expose_packages: '*' # Glob pattern or list of packages to expose as Dohballs ('*' for all eligible)
expose_doh_js: false # Whether to include doh_js in Dohballs (rarely needed)
ignore_packages: # List of packages to never expose as Dohballs
- private_package
ignore_paths: # List of directory paths (relative to root) to never include in Dohballs
- secret_folder
- build_output
rebake: false # Force rebake on every run (useful for debugging)
compile_manifest: true # Generate a dohball_manifest.json for hosting baked dohballs
worker_pool_size: 4 # Number of parallel workers for baking Dohballs
# ESM Import handling via esm.sh
esm_sh_external: [] # List of packages to exclude from esm.sh auto-bundling (e.g., ['react', 'react-dom'])
always_esbuild: false # Whether to always run the esbuild processor after packaging
# Dohball hosts for fetching remote packages
dohball_host:
- https://main.dohball.host
- https://backup.dohball.host
# Packages to load when the host is booted (auto run)
host_load:
- core_module
- ui_module
# Files/folders to ignore during scanning
packager_ignore:
directories:
# DEFAULTS:
- node_modules
- .git
- .doh # Auto-ignored, but good practice to list
- dohballs # Auto-ignored
- dist # Often contains build artifacts not part of source
The Auto-Packager parses JavaScript files looking for Doh-specific declarations:
// Module definition
Doh.Module('myModule', ['dep1', 'dep2'], function(Pattern) {
// Module implementation
// Pattern definition
Pattern('MyPattern', ['InheritsFrom'], {
// Pattern implementation
});
});
// Package definition (repacking an npm module as a Doh module of the same name)
// Note: explicit install is NOT required for imports.
// All imports are auto-installed at 'latest' unless pinned.
// Pin by adding @version to the import specifier: "dep2@^2".
Doh.Package('dep2', {
load: ['dep1', 'global import * as dp2 from "dep2"'],
});
// CLI command registration
Doh.CLI('myModule', {
'command-name': {
file: 'path/to/script.js', // dynamically imported when command is called
help: 'help displayed in `doh help` for this command'
}
});
// Optional: Manual installation instructions (advanced)
// Prefer pinning versions in import specifiers or rely on auto-install at 'latest'.
// Doh.Install remains for edge cases or non-imported assets.
Doh.Install('myModule', {
'npm:axios': '^1.6.0'
});
// Pod configuration
Doh.Pod('myModule', {
// Pod configuration outer scope
// add keys for your module as needed
myModule: {
// your settings
}
});
The Auto-Packager also processes .doh.yaml
files for simpler package definitions:
# Simple definition with just a load dependency
simple_package: dependency_package
# Complex definition with multiple options
complex_package:
load:
- import YAML from "yaml"
- import Axios from "axios"
install:
'npm:yaml': '' # empty value for latest
'npm:axios': ''
The AutoโPackager catalogs callback parameters (including destructured forms) to enable Scoped Parameter Resolution (SPR) at runtime (see Modules โ SPR). Mental model: load arrays build scope; callback params consume scope. Modules are both producers (load arrays) and consumers (callbacks); Packages are producers only.
Doh.Module()
declaration, it extracts the callbackโs parameter list (simple identifiers and destructured identifiers).Doh.Globals[name]
(shared state) โ globalThis[name]
. If still not found, resolution yields undefined
(with a diagnostic). Legacy compatibility: enabling Doh.pod.ignore_missing_module_params
(discouraged) will autoโcreate a placeholder {}
in Doh.Globals[name]
. Prefer explicit construction via parameter defaults (e.g., param = {}
or param = { globalThis: {} }
).{ format, parse }
), each requested property name is resolved independently through the same cascade and then re-packed into the single object argument. The runtime also supports object-literal defaults to repack a single object by resolving each keyโs lookup name.param = 'aliasName'
resolves param
using 'aliasName'
as the lookup name.Doh.Globals
: function(Store = {}) {}
guarantees Doh.Globals.Store
exists and injects it.{ globalThis: {} }
defaults ensure container creation in globalThis
: function(Registry = { globalThis: {} }) {}
guarantees globalThis.Registry
exists and injects it.$
is treated as jQuery
during resolution; DohPath
is injected as an Overloaded instance bound to the module file (not globalized).Think of the load block as the scope builder and the parameter list as the scope consumer. This system allows modules to communicate without explicit import/export statements while preserving specificity across dependency layers:
// The Auto-Packager sees both modules use 'SharedData' parameter
// and ensures they receive the same object instance via Doh.Globals
Doh.Module('ModuleA', function(SharedData = {}) {
SharedData.counter = 0;
SharedData.increment = function() {
SharedData.counter++;
};
});
Doh.Module('ModuleB', ['ModuleA'], function(SharedData) {
SharedData.increment();
console.log(SharedData.counter); // 1
});
// Reuse ESM exports from an ancestor
Doh.Module('UsingESM_common', [
'import { format, parse } from "date-fns"'
], function(format, parse) { /* both resolved by name */ });
Doh.Module('consumer', [
'UsingESM_common'
], function(format) { /* resolved by name from ancestor producer list */ });
// Destructured repack variant
Doh.Module('Formats', [
'import path from "path"',
'import { format, parse } from "date-fns"'
], function(utils = { format, parse, path }) {
// utils.format and utils.parse and utils.path are the references here
});
The Auto-Packager builds and validates the dependency graph of your application:
When a cyclic dependency is detected, the Auto-Packager will report it and fail, preventing potentially problematic code from running.
The Auto-Packager handles environment-specific dependencies by:
browser??
, nodejs??
)Example:
// The Auto-Packager will ensure ui-lib is only included in browser bundles
// and fs-extra is only included in Node.js bundles
Doh.Module('FeatureModule', [
'common-lib', // Included in all environments
'browser?? ui-lib', // Only included in browser bundles
'nodejs?? server-utilities', // Only included in Node.js bundles
'Doh.pod.debug?? debug-tools' // Only included if Doh.pod.debug is true
], function() {
// Module implementation
});
The Auto-Packager generates several important manifest files in two primary locations:
/.doh/manifests/
)These manifests contain internal build information, caches, and potentially sensitive configuration. They are generally not distributed and are excluded from being served by all Doh systems by default.
cli_manifest.json
): Maps registered CLI commands to their implementation files.pod_manifest.json
): Aggregated Pod configurations from Doh.Pod()
declarations.node_esm_manifest.json
): Import map for Node.js environments, mapping package names to their local paths in node_modules
. (CURRENTLY UNUSED)ap_cache.json
): Auto-Packager file cache. Stores file metadata (mtime, hash), parsed Doh definitions, and import information to speed up subsequent runs.autopackager_problems.json
): Logs errors encountered during packaging.pattern_duplicates.json
): Lists patterns defined in multiple files.deprecated_features.json
): Logs usage of deprecated Doh features.compiled_dohball_manifest.json
): Internal cache of manifests fetched from remote Dohball hosts./doh_js/manifests/
)These manifests describe the public structure and dependencies of your application and are often used by the framework at runtime.
package_manifest.json
): The central manifest defining all discovered packages, their dependencies (load
), files, and associated metadata.patterns_manifest.json
): Maps pattern names to the module they belong to.core_patterns_manifest.json
): A list of core Doh patterns provided by the framework itself.assets_manifest.json
): Maps modules to the static assets (DohPath.DohSlash()
calls) they reference.browser_esm_manifest.json
): Import map for browser environments, mapping package names to CDN URLs (typically esm.sh
).dohball_manifest.json
): If compile_manifest
is true, this lists the locally baked Dohballs available for hosting, including version and update time. Used by remote Doh instances to discover available Dohballs on this host.module_dep_graph.json
): Represents the dependency relationships between packages/modules./dohballs/dohballs.json
)dohballs.json
): Located in the root of the Dohball hosting directory (/dohballs/
). Tracks Dohball archives (.tar.gz
files) that exist in the hosting directory but no longer correspond to an actively exposed package according to the current pod.yaml
configuration. This helps identify and manage potentially obsolete Dohballs.The Auto-Packager automatically manages ESM imports for NPM packages used in your Doh code:
import
statements and Doh.load()
calls referencing external packages.node_modules
.node_esm_manifest.json
mapping the package name to its local path.node_modules
, otherwise esm.sh
(e.g., https://esm.sh/react@18
) and adds them to browser_esm_manifest.json
. Packages listed in pod.yaml
under esm_sh_external
are excluded from esm.sh
bundling.import React from 'react'
) to work seamlessly in both environments.The Auto-Packager includes an intelligent Auto-Installer that automatically manages npm dependencies based on import statements in declarative load blocks (Doh.Module()
and Doh.Package()
). This eliminates the need for manual Doh.Install()
declarations in most cases. Important: The Auto-Installer does NOT work with dynamic Doh.load()
calls.
Doh.Module()
and Doh.Package()
declarations for import statements (NOT Doh.load()
calls)react@^18.0.0
)doh upgrade
is run (which first runs doh update
to get current manifest info)doh update
again after installations to refresh manifestsThe Auto-Installer recognizes these import patterns in load blocks:
Doh.Module('my_module', [
// Named imports with version pinning
'import { useState, useEffect } from "react@^18.0.0"',
// Default imports
'import React from "react"',
// Namespace imports
'import * as utils from "date-fns@2.30.0"',
// Side-effect only imports
'import "normalize.css"',
], function(useState, useEffect, React, utils) {
// Dependencies are automatically installed
});
The Auto-Installer supports full semver version specifications:
import React from "react"
- Installs latest versionimport React from "react@^18.0.0"
- Installs compatible with 18.0.0import React from "react@~18.2.0"
- Installs compatible within 18.2.ximport React from "react@18.2.0"
- Installs exact version 18.2.0The Auto-Installer makes most Doh.Install()
statements obsolete:
// OLD WAY (deprecated but still works)
Doh.Install('my_module', {
'npm:react': '^18.0.0',
'npm:axios': '',
'npm:date-fns': '~2.30.0'
});
// NEW WAY (recommended)
Doh.Module('my_module', [
'import React from "react@^18.0.0"',
'import axios from "axios"',
'import * as dateFns from "date-fns@~2.30.0"'
], function(React, axios, datefns) {
// Dependencies auto-installed by doh upgrade
});
The Auto-Installer is triggered by the doh upgrade
command:
# Scan for missing npm dependencies and install them
doh upgrade
# Or upgrade specific packages and install any missing npm deps
doh upgrade my_specific_package
Note: doh update
only scans and reports - it does NOT install dependencies. doh upgrade
provides complete dependency management by running doh update
before and after installations to ensure manifests are always current.
The Auto-Packager has special handling for environment-specific code:
Load statements with environment conditions (browser??
, nodejs??
, etc.) are processed to ensure they only apply in the appropriate environments:
// Auto-Packager processes these conditionals correctly
Doh.Module('CrossPlatformFeature', [
'core-dependency', // Always included
'browser?? dom-utilities', // Only included in browser bundles
'nodejs?? server-utilities', // Only included in Node.js bundles
'Doh.pod.debug?? debug-tools' // Only included if Doh.pod.debug is true
], function() {
// Module implementation
});
The Auto-Packager recognizes and optimizes for the common bifurcated module pattern:
// Main package with environment-specific branch loading
Doh.Package('feature_name', {
load: [
'feature_name_common',
'browser?? feature_name_ui',
'nodejs?? feature_name_server'
]
});
// Common shared functionality
Doh.Module('feature_name_common', [], function() {
// Common implementation
});
// Browser UI implementation
Doh.Module('feature_name_ui', ['feature_name_common'], function() {
// Browser-specific implementation
});
// Server API implementation
Doh.Module('feature_name_server', ['feature_name_common'], function() {
// Server-specific implementation
});
The Auto-Packager ensures that:
Dohballs are created ("baked") based on the dohball_deployment
configuration in pod.yaml
:
expose_packages
, ignore_packages
, and ignore_paths
.doh bake [names...]
or doh rebake [names...]
, the list is filtered./modules/user/
might be baked into /dohballs/modules/user.tar.gz
).dohball.json
file within the source directory (e.g., /modules/user/dohball.json
). The version
field in this file determines the Dohball version.ap_cache.json
) for the last bake. A Dohball is only baked if the content has changed or if rebake
is forced (doh rebake
or pod.yaml
)..tar.gz
archive) is parallelized using a worker pool (size configured by worker_pool_size
)./dohballs/
directory, mirroring the source structure (e.g., /modules/user/
becomes /dohballs/modules/user.tar.gz
).compile_manifest
is true, /doh_js/manifests/dohball_manifest.json
is updated with information about the baked Dohball (version, updated time)./dohballs/dohballs.json
manifest is updated to track any orphaned Dohballs (existing archives not matching current exposed packages).The Auto-Packager handles Dohballs differently depending on whether they are local or remote:
Dohballs support two complementary distribution approaches for sharing code between different Doh instances:
.tar.gz
files are committed to Git repositoriesdoh install package_name
to pull from remote hosts.tar.gz
files are served directly via HTTP/HTTPS using the dohball_host
moduledoh install
or doh upgrade
Configuration for HTTP Hosting:
# In hosting instance pod.yaml
dohball_deployment:
compile_manifest: true
host_load:
- dohball_host
# In consuming instance pod.yaml
dohball_host:
- https://your-host.com
- http://localhost:3001
The Auto-Packager employs several techniques to ensure speed:
Utilizes /.doh/manifests/ap_cache.json
to store file modification times, content hashes, and parsed Doh definitions (modules, packages, patterns, imports). On subsequent runs, only files that have changed since the last run are re-parsed, significantly speeding up the process.
Dohball creation (bake
and rebake
) uses a configurable worker pool (worker_pool_size
in pod.yaml
) to perform the CPU-intensive tasks of hashing, compressing, and archiving files in parallel.
The Auto-Packager can be configured to ignore specific directories to reduce unnecessary file processing.
The Auto-Packager integrates with the Pod system by:
pod.yaml
Doh.Pod()
declarationsIntegration with the Module system includes:
Doh.Module()
definitionsThe Auto-Packager uses the Load System for:
browser??
, nodejs??
, etc.)The Auto-Packager interacts with the Pattern system by:
// Define a cross-platform feature with environment branching
Doh.Package('user_management', {
load: [
// Common functionality
'user_management_common',
// Environment-specific implementations
'browser?? user_management_ui',
'nodejs?? user_management_server'
]
});
// The Auto-Packager:
// 1. Detects the environment branching pattern
// 2. Tracks dependencies for each branch
// 3. Includes common code in all environments
// 4. Bundles UI code only for browsers
// 5. Bundles server code only for Node.js
// Define a module with configuration-based features
Doh.Module('admin_panel', [
// Base functionality
'admin_core',
// Conditional features based on configuration
'Doh.pod.features.analytics?? admin_analytics',
'Doh.pod.features.permissions?? admin_permissions',
// Environment-specific and config-specific
'browser&&Doh.pod.features.dashboard?? admin_dashboard'
], function() {
// Module implementation
});
// The Auto-Packager:
// 1. Parses the complex conditions
// 2. Tracks configuration-dependent features
// 3. Handles combined environment and config conditions
The Auto-Packager generates several diagnostic files within /.doh/manifests/
:
autopackager_problems.json
: Logs errors or inconsistencies found during scanning and packaging.pattern_duplicates.json
: Lists any patterns defined with the same name in multiple files.deprecated_features.json
: Records usage of deprecated Doh functions or parameters.dohballs.json
(in /dohballs/
): Tracks orphaned Dohball archives.If the Auto-Packager detects cyclic dependencies, it will report an error. To fix:
Doh.load()
instead of static dependenciesIf a module is referenced but not found:
Doh.Module()
doh update
to refresh manifestsIf environment-specific code is not working correctly:
browser??
, nodejs??
) are properly appliedIsBrowser()
, IsNode()
)Organize Related Components
Use Environment Branching Consistently
Avoid Cyclic Dependencies
Leverage Parameter Sharing
Regular Updates
doh update
when adding new componentsManifest Awareness
Maintain Clean Dependencies
# Run the auto-packager to scan and generate manifests (no remote installation)
doh update
# Complete dependency management - runs update โ installs โ update again
doh upgrade
# Bake Dohballs whose content has changed since the last bake.
# Can optionally specify package names to only check those.
doh bake [package_name...]
# Force rebaking of Dohballs, regardless of content changes.
# Can optionally specify package names to only rebake those.
doh rebake [package_name...]
Doh.Module(name, dependencies, callback)
: Define a moduleDoh.Package(name, config)
: Define a packagePattern(name, inherits, idea)
: Define a pattern in a Doh moduleDoh.Install(name, assets, callback)
: Define installation requirementsDoh.CLI(package, commands)
: Register CLI commandsDoh.Pod(package, config)
: Register pod configurationglobal
, direct
, etc.)^/
in paths within load
arrays for references relative to the file's location (see DohPath)