Overview

extmod is a CLI, runtime wrapper and API that allows Node applications to dynamically load remote ESM modules at runtime. To put it simply, extmod enables http(s)-based import statements within Node applications.

extmod leverages already existing Node machinery, namely ESM module loaders and module permissions, to give both stability and security to the modules you are loading into your application.

extmod also updates your modules in real-time, driven by etags or max-age Cache-Control headers, to enable the freshest code to be used, all out of the box.

Use extmod to:

• Pull remote configuration files, ala feature flagging.
• Leverage shared, common code in your organization.
• Automatically bundle and serve frontend components in SSR/RSC applications, ala Federated Modules.

Getting Started

Node 18.x or greater is required

Installation

yarn add extmod

npm i extmod

pnpm add extmod

Initialization

The next step is to generate a .extmod.json configuration file.

npx extmod init

pnpm exec extmod init

Allow Remotes

By default, remote modules are not enabled. You need to opt-in URLs that you have deemed safe to run within your application.

To allow a remote URL to be loaded, you can use the config add-remote command.

npx extmod config add-remote https://my.module.com/

pnpm exec extmod config add-remote https://my.module.com/

Import Remotes

You can now import your remote module in your application as you would any other ESM module.

import utilities from "https://my.module.com/utilities.mjs";

console.log(utilities.magicNumber) // 42!

import utilities from "https://my.module.com/utilities.mjs";

console.log(utilities.magicNumber) // 42!

Application Bootstrap

The final requirement is to leverage the extmod run command wrapper when you bootstrap your application. extmod will pass through anything you add after "run" to Node, however it will inject runtime flags which enable the extmod ESM loader as well as apply the module permissions policy you have defined.

extmod run index.mjs

extmod run next dev

Mechanisms of Action

extmod goes through various stages depending on the modules being requested, their ESM cache status and the module contents.

Resolving

When extmod finds an import starting with /^https?/, it will automatically make a HEAD request to that module in order to determine whether:

1. It has seen this specific module before.
2. It has a changed etag value, indicating it has changed.
3. It has a max-age Cache-Control header.

Depending on the answers to the questions above, extmod will either load the existing module in the local ESM cache or it will make a GET request for an updated verison of the module.

If no etag or max-age Cache-Control header is found, once loaded, the remote module will be cached within the ESM cache indefinitely. There is currently no way to "bust" or remove a loaded ESM module from the ESM cache, so the cache will grow unbounded if not restricted.

By default, bare module imports (such as "react" or "lodash") are resolved locally. This aligns with how most bundlers work with "externals" defined. You should treat bare module imports in your remote modules as being external, where extmod will look to the local project for that dependency. If you wish for your remote modules to leverage different versions other than what is supplied locally, you should pre-bundle your module with the dependency.

Permissions

Pulling content from potentially untrusted sources has inherent security risks. extmod tries to allievate this concern by leveraging Node's module permission infrastructure, which validates only approved modules when they are loaded by the runtime, and rejects all others.

Remote modules are not enabled by default. You must use the extmod CLI's config actions to add allowable remote module URLs (or domains) that you trust in order for them to be loaded into your applications.

See the Configuration section on how to do this.

Loading

After it has been determined a remote module should be loaded, a GET request against the module URL is made. Upon successful request, the remote module is fed through the same import process

Error Handling

The points of failure in the resolution and loading process should be well known in order to guard effectively in your applications. extmod tries to make handling these points of failure easier by bubbling up the error into your application code as data, rather than as a bonefide error.

There are two primary classes errors: network errors (both in resolving and loading modules) and module errors (loading a CommonJS vs ESM module for instance).

Handling these errors involves using the extmod supplied error key and determining whether an error exists within your application code.

// Import our remote component
import * as utilities from "https://my.module.com/utilities.mjs";
// Import extmod error keys
import { EXTMOD_ERROR, EXTMOD_ERROR_CODE, EXTMOD_ERROR_REASON, Extmod } from "extmod";
// Deconstruct out a possible error
const { [EXTMOD_ERROR]: error, default } = utilities as Extmod<{ default: any }>;

if (error) {
  const {
    [EXTMOD_ERROR_CODE]: code,
    [EXTMOD_ERROR_REASON]: reason,
  } = error;
  // handle code and reason
} else {
  // use your default (if applicable) import
}

// Import our remote component
import * as utilities from "https://my.module.com/utilities.mjs";
// Import extmod error keys
import { EXTMOD_ERROR, EXTMOD_ERROR_CODE, EXTMOD_ERROR_REASON, Extmod } from "extmod";
// Deconstruct out a possible error
const { [EXTMOD_ERROR]: error, default } = utilities;

if (error) {
  const {
    [EXTMOD_ERROR_CODE]: code,
    [EXTMOD_ERROR_REASON]: reason,
  } = error;
  // handle code and reason
} else {
  // use your default (if applicable) import
}

Server Side Rendering

Since extmod is leveraging normal Node import machinery, all imported modules are assumed to be tailored for a server environment by default. However, with the rise of Server Side Rendering frameworks, the lines have been blurred on what imported code is being ran on the server and ran on the client (browser).

Currently, only React 18+ is supported.

When extmod detects an import is to be used in a browser environment, it will automatically write it and any imported dependencies to disk, bundle them (via esbuild) and make it available to the browser for rendering.

Import Signaling

There are two ways to trigger client bundling through extmod, import attributes or the "use client" directive in the remote module.

Import Attribute

A new feature of ECMAScript allows attributes to be attached to import statements in order to inform the ESM loader of particularities around what is being imported. extmod leverages this with a new client attribute to singal the loader that the imported module should be treated as a client module.

import { MyModule } from "https://my.module.com/index.mjs" with { type: "client" };

import { MyModule } from "https://my.module.com/index.mjs" with { type: "client" };

"use client" Directive

React and frameworks have aligned on the "use client" Javascript directive to describe the server/client boundry in Server Side Rendering. extmod looks for this directive in remotely loaded modules to enforce that boundry and create the client bundle dynamically.

You do not add "use client" in your application code, rather you define it in your remote module and extmod will detect it automatically during runtime.

"use client";

// Browser-centric code

"use client";

// Browser-centric code

Suspense + Bundling

In order for a seamless UX, extmod uses React's Suspense to pause component rendering while the bundling process occurs; the frontend should continue rendering unabated. When you declare a client import, extmod immediately returns a default export containing a Suspense-based component that will resolve into a <script> tag containing the location of your bundle on disk.

import { FC } from "react";
import Component from "https://my.module.com/index.mjs" with { type: "client" };

export const MyRemoteComponent: FC = () => (
  // Initially Component resolves to a placeholder while the bundle is created.
  // After bundling is completed, a <script> tag is rendered here with a src
  // corresponding to the completed bundle location on the server.
  <Component />
);

import Component from "https://my.module.com/index.mjs" with { type: "client" };

export const MyRemoteComponent = () => (
  // Initially Component resolves to a placeholder while the bundle is created.
  // After bundling is completed, a <script> tag is rendered here with a src
  // corresponding to the completed bundle location on the server.
  <Component />
);

Client-side Utility

An optional but encouraged step is to use the exported ExtmodSuspense component. After your client side component has successfully resolved to a <script> and the bundle has been loaded into the browser, you need a way to ultimately render that component in-place, including adding an optional fallback.

ExtmodSuspense uses a MutationObserver to watch when a <script> tag is rendered within it and then replaces that <script> tag with the bundle code that was loaded into the browser - all completely automatically.

import { FC } from "react";
import { ExtmodSuspense } from "extmod/client";
import Component from "https://my.module.com/index.mjs" with { type: "client" };

export const MyRemoteComponent: FC = () => (
  // The wrapper waits until our <script> tag is rendered via Suspense.
  // It then replaces it with our bundled client component.
  <ExtmodSuspense 
    component={<Component />}
    fallback={<div>Loading...</div>} 
  />
);

import { ExtmodSuspense } from "extmod/client";
import Component from "https://my.module.com/index.mjs" with { type: "client" };

export const MyRemoteComponent = () => (
  // The wrapper waits until our <script> tag is rendered via Suspense.
  // It then replaces it with our bundled client component.
  <ExtmodSuspense 
    component={<Component />}
    fallback={<div>Loading...</div>} 
  />
);

Configuration

extmod configuration lives in a .extmod.json file at the root of your project and drives key features. Because of the complexity of managing module permissions, it is advised to use the extmod CLI unless you are versed in how Node module permissions are used. That being said, extmod uses an opinionated configuration of the module permission spec speficially tailored for remote modules.

For further information on the mechanics about how Node module permissions work, consult the Node documentation.

The .extmod.json configuraiton strictly validated before any CLI command is ran. To avoid errors, do not manually change the configuration file. Rather, use the CLI to perform operations against the configuraiton file.

{
  interface ExtmodConfiguration {
    // The current configuration verison
    version: string;
    // A mapping of aliases to remote module URLs
    aliases: Record<string, string>;
    // The nested Node module permissions configuration
    policy: {
      // Individual remote file permissions
      resources: {
        [url: string]: {
          dependencies: boolean;
          integrity?: string;
        };
      };
      // Remote, cascading path permissions
      // e.g. https://my.module/*, allowing any module under the my.module domain
      scopes: {
        [url: string]: {
          dependencies: boolean;
          integrity?: string;
        }
      };
    };
  }
}

CLI

The extmod CLI is helpful in managing it's configuration and running programs to enact said configuration.

init

Create a new .extmod.json configuration file with defaults. Note, remote modules are disabled by default.

npx extmod init

pnpm exec extmod init

ls

List out the current configuration in an easily consumable format within the console.

npx extmod init

pnpm exec extmod init

config

Config subcommands aid in the management of the extmod configuration file.

add-remote

Add a remote module path. This allows any resources under the given URL to be imported into your application.

# Allow any resource under myProject/ to be imported.
# For example https://s3-us-east-1.amazonaws.com/myOrg/myProject/myModule.mjs
npx extmod config add-remote https://s3-us-east-1.amazonaws.com/myOrg/myProject/

# Allow any resource under myProject/ to be imported.
# For example https://myaccount.blob.core.windows.net/mycontainer/myblob/myModule.mjs
npx extmod config add-remote https://myaccount.blob.core.windows.net/mycontainer/myblob/

update-remote

Update the remote URL via the original URL or the alias.

# Assume we've ran "npx extmod config add-remote https://s3-us-east-1.amazonaws.com/myOrg/myProject/"
npx extmod config update-remote https://s3-us-east-1.amazonaws.com/myOrg/myProject/ https://s3-us-east-1.amazonaws.com/myOtherOrg/myOtherProject/

# Assume we've ran "npx extmod config add-remote -a my-project https://myaccount.blob.core.windows.net/mycontainer/myblob/"
npx extmod config update-remote my-project https://myaccount.blob.core.windows.net/myothercontainer/myotherblob/

delete-remote

Delete a remote URL via the original URL or the alias.

# Assume we've ran "npx extmod config add-remote https://s3-us-east-1.amazonaws.com/myOrg/myProject/"
npx extmod config delete-remote https://s3-us-east-1.amazonaws.com/myOrg/myProject/

# Assume we've ran "npx extmod config add-remote -a my-project https://myaccount.blob.core.windows.net/mycontainer/myblob/"
npx extmod config delete-remote my-project

run

Run a Node command with extmod enabled.

extmod run index.mjs

extmod run next dev

validate

Validate an extmod configuration file. Outputs an error and description if configuration is malformed.

npx extmod validate

pnpm exec extmod validate

API

The externally facing API of extmod is quite simple and usually not needed.

extmod

The exported extmod() function is available for situations where using the normal import Node semantics are not ideal. This is usually done in cases where the application is using a bundler that transforms import into require or statically evalutes import at build-time and is not equipt to handle remote imports.

export type extmod = (
  url: string,
  type?: "json" | "client",
) => Promise<any>

Under the hood, extmod() simply wraps the normal import in an eval to "hide" until runtime, to which things perform as expected.