config: apply extends

antongolub

Anton Golub

Posted on February 5, 2024

config: apply extends

As we know, the forms and approaches to describing configuration files are extremely varied. I discussed this in the previous note. But there are several techniques and practices that may one day become standards.

One of this – the extends directive. Many tools provide this feature, but it works a little differently in each place. For example, tsconfig applies deep merge to compilerOptions, while prettier concatenates overrides array sections, commitlint joins plugin, etc. So the authors have to implement these nuances on site every time, and it's tiring and annoying a bit. I believe we will come up with a more efficient way to handle this routine.

Proposal

import { populate, populateSync } from '@topoconfig/extends'

const tsconfig = await populate('tsconfig.json', {
  compilerOptions: 'merge'
})
const prettierCfg = populateSync('.prettierrc.json', {
  overrides: 'merge'
})
Enter fullscreen mode Exit fullscreen mode

20 minutes adventure

At first glance, the extends may seem like a simple combination of Object.assign and fs.readFile. Well, technically it is, but not quite so. Let's see how the real scenario unfolds. To simplify the details, we will use pseudocode.

The starting point:

input = {
  foo: 'bar',
  extends: './base.json'
}
Enter fullscreen mode Exit fullscreen mode
  • pick extends
resource = input.extends
Enter fullscreen mode Exit fullscreen mode
  • load the referenced ./base.json
extra = JSON.parse(fs.readFileSync(resource, 'utf8'))
Enter fullscreen mode Exit fullscreen mode
  • merge input and extra
result = {
  ...input,
  ...extra,
  extends: undefined,
}
Enter fullscreen mode Exit fullscreen mode

How about ./base.js? Reasonable request.

  • provide js files support
input = { extends: './base.js' }
Enter fullscreen mode Exit fullscreen mode

require hurries to the resque:

load = (ref) => isJs(ref) ? require(ref) : JSON.parse(fs.readFileSync(ref, 'utf8'))
Enter fullscreen mode Exit fullscreen mode

./base.mjs?

  • Ooops. Here the API switches to async mode to handle mjs/esm inputs. Of course, we can patch require with esm package, or load module via a process.spawnSynced nodejs. We can, but we won't
load = isJs(ref)
  ? (await import(ref))?.default
  : fs.promises.readFile(ref, 'utf8')
Enter fullscreen mode Exit fullscreen mode

What if the value is passed without an extension, like base?

  • add support for shared npm configs
  • provide configurable resolve
resolve = (ref, cwd) =>
  ref.startsWith('.') || ref.startsWith('/') || path.extname(ref)
    ? path.resolve(cwd, ref)
    : ref
Enter fullscreen mode Exit fullscreen mode

But what if extends is a nested object? It definitely is.

  • apply everything recursively:
handle = input = ({
  ...input,
  ...handle(load(input.extends)),
  extends: undefined,
})
Enter fullscreen mode Exit fullscreen mode

In this case, circular references are something to pay attention to.

  • add index to avoid infinite loops
memo = new Map()

memoizedLoad = ref => {
  if (memo.has(ref)) {
    return memo.get(ref)
  }
  memo.set(ref, load(ref))
  return load(ref)
}
Enter fullscreen mode Exit fullscreen mode

Relative resource path should use the closest parent dirname as a base to work properly.

  • resolve cwd for each extends reference
base = path.resolve(process.cwd(), _opts.cwd ?? '.')
[cwd, _config] = typeof config === 'string'
  ? [path.resolve(base, path.dirname(config)), path.basename(config)]
  : [base, config]
Enter fullscreen mode Exit fullscreen mode

Multiple extends maybe?

  • let extends hold several references
input = {
  foo: 'bar',
  extends: ['./extra.json', './mixin.js']
}
extras = [input.extends].flat().map(load)
Enter fullscreen mode Exit fullscreen mode

How to read toml, yaml, ini, env? Obviously, custom formats are required too.

  • provide parse option
parse = (ref, contents) => {
  if (ref.endsWith('.yaml') || ref.endsWith('.yml')) 
      return parseYaml(contents)
  if (ref.endsWith('.json')) 
      return JSON.parse(contents)
  throw new Error(`Unsupported format: ${ref}`)
}
Enter fullscreen mode Exit fullscreen mode

How to resolve value conflicts: merge / override?

  • provide rules to handle prop-specific join strategies
rules = {
  '*': 'merge',
  foo: 'override'
}
rule = getRule(propPath, rules)
result[key] = isObject(value) && rule === 'merge'
  ? extend({
    sources: [value],
    rules,
    prefix: p,
    index
  })
  : value
Enter fullscreen mode Exit fullscreen mode

What about arrays?

  • implement array merge as concat
if (rule === 'merge') {
  arr.push(...sources.flat(1))
} else {
  arr.length = 0
  arr.push(...sources.slice(-1).flat(1))
}
Enter fullscreen mode Exit fullscreen mode

It would be nice to avoid mutations.

  • introduce clone
safeLoad = compose(memoize, clone, load)
Enter fullscreen mode Exit fullscreen mode

Conclusion

Summarizing:

  • search extends statements
  • load & parse referenced resources
    • support async and sync flows
    • use cloning to avoid mutations
    • .js, .cjs, .mjs, .json
    • provide support for custom formats
    • resolve cwd for the nested resource by its closest parent resource
    • support custom resolvers
  • walk recursively
    • handle circular references
  • provide sync and async API
  • assemble the result
    • handle arrays / plain objects
    • apply prop-specific merge

Additionally:

  • build as cjs and esm
  • provide types
  • handle windows specific issues
  • add dozens of tests
  • handle different runtimes specifics (node, deno, bun)

There is an obvious benefit to implementing this algorithm on your own – it clarifies and makes controllable every aspect of config processing in place.
However, we do not always have enough time to do this exercise. Therefore, maybe someone will find this shortcut useful:

💖 💪 🙅 🚩
antongolub
Anton Golub

Posted on February 5, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

config: apply extends
javascript config: apply extends

February 5, 2024