config: apply extends
Anton Golub
Posted on February 5, 2024
As we know, the forms and approaches to describing configuration files are extremely varied. I discussed this in the previous note. But there are several techniques and practices that may one day become standards.
One of this – the extends
directive. Many tools provide this feature, but it works a little differently in each place. For example, tsconfig
applies deep merge to compilerOptions
, while prettier
concatenates overrides
array sections, commitlint joins plugin
, etc. So the authors have to implement these nuances on site every time, and it's tiring and annoying a bit. I believe we will come up with a more efficient way to handle this routine.
Proposal
import { populate, populateSync } from '@topoconfig/extends'
const tsconfig = await populate('tsconfig.json', {
compilerOptions: 'merge'
})
const prettierCfg = populateSync('.prettierrc.json', {
overrides: 'merge'
})
20 minutes adventure
At first glance, the extends
may seem like a simple combination of Object.assign
and fs.readFile
. Well, technically it is, but not quite so. Let's see how the real scenario unfolds. To simplify the details, we will use pseudocode.
The starting point:
input = {
foo: 'bar',
extends: './base.json'
}
- pick
extends
resource = input.extends
- load the referenced
./base.json
extra = JSON.parse(fs.readFileSync(resource, 'utf8'))
- merge
input
andextra
result = {
...input,
...extra,
extends: undefined,
}
How about ./base.js
? Reasonable request.
- provide js files support
input = { extends: './base.js' }
require
hurries to the resque:
load = (ref) => isJs(ref) ? require(ref) : JSON.parse(fs.readFileSync(ref, 'utf8'))
./base.mjs
?
- Ooops. Here the API switches to async mode to handle mjs/esm inputs.
Of course, we can patch
require
withesm
package, or load module via aprocess.spawnSync
ed nodejs. We can, but we won't
load = isJs(ref)
? (await import(ref))?.default
: fs.promises.readFile(ref, 'utf8')
What if the value is passed without an extension, like base
?
- add support for shared npm configs
- provide configurable
resolve
resolve = (ref, cwd) =>
ref.startsWith('.') || ref.startsWith('/') || path.extname(ref)
? path.resolve(cwd, ref)
: ref
But what if extends
is a nested object? It definitely is.
- apply everything recursively:
handle = input = ({
...input,
...handle(load(input.extends)),
extends: undefined,
})
In this case, circular references are something to pay attention to.
- add index to avoid infinite loops
memo = new Map()
memoizedLoad = ref => {
if (memo.has(ref)) {
return memo.get(ref)
}
memo.set(ref, load(ref))
return load(ref)
}
Relative resource path should use the closest parent dirname as a base to work properly.
- resolve
cwd
for eachextends
reference
base = path.resolve(process.cwd(), _opts.cwd ?? '.')
[cwd, _config] = typeof config === 'string'
? [path.resolve(base, path.dirname(config)), path.basename(config)]
: [base, config]
Multiple extends
maybe?
- let
extends
hold several references
input = {
foo: 'bar',
extends: ['./extra.json', './mixin.js']
}
extras = [input.extends].flat().map(load)
How to read toml, yaml, ini, env? Obviously, custom formats are required too.
- provide
parse
option
parse = (ref, contents) => {
if (ref.endsWith('.yaml') || ref.endsWith('.yml'))
return parseYaml(contents)
if (ref.endsWith('.json'))
return JSON.parse(contents)
throw new Error(`Unsupported format: ${ref}`)
}
How to resolve value conflicts: merge / override?
- provide
rules
to handle prop-specific join strategies
rules = {
'*': 'merge',
foo: 'override'
}
rule = getRule(propPath, rules)
result[key] = isObject(value) && rule === 'merge'
? extend({
sources: [value],
rules,
prefix: p,
index
})
: value
What about arrays?
- implement array merge as concat
if (rule === 'merge') {
arr.push(...sources.flat(1))
} else {
arr.length = 0
arr.push(...sources.slice(-1).flat(1))
}
It would be nice to avoid mutations.
- introduce
clone
safeLoad = compose(memoize, clone, load)
Conclusion
Summarizing:
- search
extends
statements - load & parse referenced resources
- support async and sync flows
- use cloning to avoid mutations
- .js, .cjs, .mjs, .json
- provide support for custom formats
- resolve cwd for the nested resource by its closest parent resource
- support custom resolvers
- walk recursively
- handle circular references
- provide sync and async API
- assemble the result
- handle arrays / plain objects
- apply prop-specific merge
Additionally:
- build as cjs and esm
- provide types
- handle windows specific issues
- add dozens of tests
- handle different runtimes specifics (node, deno, bun)
There is an obvious benefit to implementing this algorithm on your own – it clarifies and makes controllable every aspect of config processing in place.
However, we do not always have enough time to do this exercise. Therefore, maybe someone will find this shortcut useful:
Posted on February 5, 2024
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.