The Effect Data Types: Managed & Layer
Michael Arnaldi
Posted on October 19, 2020
In the previous post of the series we started to explore the Effect
data type in details.
The Effect
type is itself used to construct a variety of data-types that are thought to cover many of the common patterns of software development.
This time we will explore Managed
and Layer
both constructed using Effect
, the first uses to safely allocate and release resources and the second used to construct environments.
In the code snippets we will simulate data stores by making usage of a third data-type built on Effect
the Ref
data-type, Ref<T>
encodes a mutable Reference
to an immutable value T
and can be used to keep track of State
through a computation.
Bracket
Let's start by taking a look at a classic pattern we encounter many times: bracket
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
// connect to the database
export const connectDb = pipe(
// make a reference to an empty map
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
// get the current map and lookup k
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
// updates the reference to an updated Map containing k->v
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
// clean up (for simulation purposes)
clear: ref.set(Map.empty)
})
)
)
)
// write a program that use the database
export const program = pipe(
// acquire the database connection
connectDb,
T.bracket(
// use the database connection
(_) =>
pipe(
T.do,
T.tap(() => _.put("ka", "a")),
T.tap(() => _.put("kb", "b")),
T.tap(() => _.put("kc", "c")),
T.bind("a", () => _.get("ka")),
T.bind("b", () => _.get("kb")),
T.bind("c", () => _.get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`)
),
// release the database connection
(_) => _.clear
)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(s)
})
),
T.runMain
)
Bracket is used to safely acquire Resources
like DbConnection
to be used and safely released post usage.
Managed
The idea behind Managed
is to generalize the behaviour of bracket
by isolating the declaration of acquire
and release
into a specific data-type.
The inner details of how this data-type works are out of the scope of this post but the design is a very interesting (more advanced) topic to explore. It is based on the implementation of ZManaged
in ZIO
that is itself based on the ResourceT
designed by Michael Snoyman in haskell.
Let's take a look at the same code when isolated into a Managed
:
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
// connect to the database
export const managedDb = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear)
)
// write a program that use the database
export const program = pipe(
// use the managed DbConnection
managedDb,
M.use((_) =>
pipe(
T.do,
T.tap(() => _.put("ka", "a")),
T.tap(() => _.put("kb", "b")),
T.tap(() => _.put("kc", "c")),
T.bind("a", () => _.get("ka")),
T.bind("b", () => _.get("kb")),
T.bind("c", () => _.get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`)
)
)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(s)
})
),
T.runMain
)
We can see how we moved the release
step from our program into the data-type of managedDb
, now we can use the Resource
without having to track its closing state.
Composing Managed Resources
One other advantage that we gained is we can now easily compose multiple Resources
together, let's take a look at how we might do that:
import * as Array from "@effect-ts/core/Classic/Array"
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
// simulate a connection to a message broker
export interface BrokerConnection {
readonly send: (message: string) => T.UIO<void>
readonly clear: T.UIO<void>
}
// connect to the database
export const managedDb = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear)
)
// connect to the database
export const managedBroker = pipe(
Ref.makeRef(<Array.Array<string>>Array.empty),
T.chain((ref) =>
T.effectTotal(
(): BrokerConnection => ({
send: (message) =>
pipe(ref, Ref.update<Array.Array<string>>(Array.snoc(message))),
clear: pipe(
ref.get,
T.chain((messages) =>
T.effectTotal(() => {
console.log(`Flush:`)
messages.forEach((message) => {
console.log("- " + message)
})
})
)
)
})
)
),
// release the connection via managed
M.make((_) => _.clear)
)
// write a program that use the database
export const program = pipe(
// use the managed DbConnection
managedDb,
M.zip(managedBroker),
M.use(([{ get, put }, { send }]) =>
pipe(
T.do,
T.tap(() => put("ka", "a")),
T.tap(() => put("kb", "b")),
T.tap(() => put("kc", "c")),
T.bind("a", () => get("ka")),
T.bind("b", () => get("kb")),
T.bind("c", () => get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`),
T.tap(send)
)
)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(`Done: ${s}`)
})
),
T.runMain
)
We have added a second Managed
called managedBroker
to simulate a connection to a message broker and we composed the 2 using zip
.
The result will be a Managed
that acquires both Resources
and release both Resources
as you would expect.
There are many combinators in the Managed
module, for example a simple change from zip
to zipPar
would change the behaviour of the composed Managed
and will acquire
and release
both Resources
in parallel.
Layer
The Layer
data-type is closely related to Managed
, in fact the idea for the design of Layer
has its roots in a pattern commonly discovered in the early days of ZIO
by early adopters.
We have looked at the Effect
data-type already and we notice how the environment R
can be used to embed services in order to gain great testability and code organization.
The question now is, how do I construct those environments?
It seems simple in principle but when you get to write an app this embedding of services becomes so pervasive that is completely normal to find yourself embedding hundreds of services in your "main" program.
Many of the services are going to be dependent on database connections and stuff like that, and as we saw before Managed
is a good use case for those.
The community has started to construct their environments by using Managed
so that services are constructed at bootstrap time and the result is used to provide the environment.
This pattern progressively evolved to an independent data-type that is based on Managed
tailored to specifically to construct environments.
Let's start by simply transforming our code to use Layers
, it will be straightforward given we have already designed it using Managed
:
import * as Array from "@effect-ts/core/Classic/Array"
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as L from "@effect-ts/core/Effect/Layer"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
// simulate a connection to a message broker
export interface BrokerConnection {
readonly send: (message: string) => T.UIO<void>
readonly clear: T.UIO<void>
}
// connect to the database
export const DbLive = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromRawManaged
)
// connect to the database
export const BrokerLive = pipe(
Ref.makeRef(<Array.Array<string>>Array.empty),
T.chain((ref) =>
T.effectTotal(
(): BrokerConnection => ({
send: (message) =>
pipe(ref, Ref.update<Array.Array<string>>(Array.snoc(message))),
clear: pipe(
ref.get,
T.chain((messages) =>
T.effectTotal(() => {
console.log(`Flush:`)
messages.forEach((message) => {
console.log("- " + message)
})
})
)
)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromRawManaged
)
export const ProgramLive = L.all(DbLive, BrokerLive)
// write a program that use the database
export const program = pipe(
// access Db from environment
T.access(({ get, put }: DbConnection) => ({ get, put })),
// access Broker from environment
T.zip(T.access(({ send }: BrokerConnection) => ({ send }))),
// use both
T.chain(([{ get, put }, { send }]) =>
pipe(
T.do,
T.tap(() => put("ka", "a")),
T.tap(() => put("kb", "b")),
T.tap(() => put("kc", "c")),
T.bind("a", () => get("ka")),
T.bind("b", () => get("kb")),
T.bind("c", () => get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`),
T.tap(send)
)
)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(`Done: ${s}`)
})
),
// provide the layer to program
T.provideSomeLayer(ProgramLive),
T.runMain
)
What we did is we turned our Managed
into Layers
by using the fromRawManaged
constructor and we then plugged the results into a single ProgramLive
layer.
We then transformed our program to consume the DbConnection
and BrokerConnection
services from enviroment.
At the very end we provided the ProgramLive
to our program
by using provideSomeLayer
.
The behaviour itself remained the same but we have now fully separated program definition from resources and services location.
The Has & Tag utilities
In the last article of the series we scratched the surface of the Has
& Tag
apis, we are going to see now a bit more details about the inner working and how it integrates into Layers.
Up to now we haven't noticed issues because we took care of keeping function names like send
and get
separated but in reality it is not a good idea to intersect casual services into a big object because conflicts might easily arise.
One conflict we had without even noticing (because it's used only locally inside managed) is the "clear" effect of both DbConnection
and BrokerConnection
, that will get shadowed at runtime in the intersected result.
In order to solve this issue, and in order to improve the amount of types you end up writing in your app we designed two data-types that solve the issue.
We took inspiration from the work done in ZIO
where the Has
type is explicit and the Tag
type is implicit in typescript we had to come up with an explicit solution.
The trick works like that:
import { tag} from "@effect-ts/core/Classic/Has"
/**
* Any interface or type alias
*/
interface Anything {
a: string
}
/**
* Tag<Anything>
*/
const Anything = tag<Anything>()
/**
* (r: Has<Anything>) => Anything
*/
const readFromEnv = Anything.read
/**
* (_: Anything) => Has<Anything>
*/
const createEnv = Anything.of
const hasAnything = createEnv({ a: "foo" })
/**
* Has<Anything> is fake, in reality we have:
*
* { [Symbol()]: { a: 'foo' } }
*/
console.log(hasAnything)
/**
* The [Symbol()] is:
*/
console.log((hasAnything as any)[Anything.key])
/**
* The same as:
*/
console.log(readFromEnv(hasAnything))
/**
* In order to take ownership of the symbol used we can do:
*/
const mySymbol = Symbol()
const Anything_ = tag<Anything>().setKey(mySymbol)
console.log((Anything_.of({ a: "bar" }) as any)[mySymbol])
Fundamentally Tag<T>
encodes the capability of reading T
from a type Has<T>
and encodes the capability of producing a value of type Has<T>
given a T
.
With the guarantee that given T0
and T1
the result Has<T0> & Has<T1>
is always discriminated.
Let's see it practice by rewriting our code:
import * as Array from "@effect-ts/core/Classic/Array"
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as L from "@effect-ts/core/Effect/Layer"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
import { tag } from "@effect-ts/system/Has"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
export const DbConnection = tag<DbConnection>()
// simulate a connection to a message broker
export interface BrokerConnection {
readonly send: (message: string) => T.UIO<void>
readonly clear: T.UIO<void>
}
export const BrokerConnection = tag<BrokerConnection>()
// connect to the database
export const DbLive = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(DbConnection)
)
// connect to the database
export const BrokerLive = pipe(
Ref.makeRef(<Array.Array<string>>Array.empty),
T.chain((ref) =>
T.effectTotal(
(): BrokerConnection => ({
send: (message) =>
pipe(ref, Ref.update<Array.Array<string>>(Array.snoc(message))),
clear: pipe(
ref.get,
T.chain((messages) =>
T.effectTotal(() => {
console.log(`Flush:`)
messages.forEach((message) => {
console.log("- " + message)
})
})
)
)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(BrokerConnection)
)
export const ProgramLive = L.all(DbLive, BrokerLive)
// write a program that use the database
export const program = pipe(
// access Db from environment
T.accessService(DbConnection)(({ get, put }) => ({ get, put })),
// access Broker from environment
T.zip(T.accessService(BrokerConnection)(({ send }) => ({ send }))),
// use both
T.chain(([{ get, put }, { send }]) =>
pipe(
T.do,
T.tap(() => put("ka", "a")),
T.tap(() => put("kb", "b")),
T.tap(() => put("kc", "c")),
T.bind("a", () => get("ka")),
T.bind("b", () => get("kb")),
T.bind("c", () => get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`),
T.tap(send)
)
)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(`Done: ${s}`)
})
),
// provide the layer to program
T.provideSomeLayer(ProgramLive),
T.runMain
)
By this very simple change we are now safe, and as we can already see, instead of manually annotate the type DbConnection
to access the service we can now simply use the Tag
value.
We can go a bit further and derive some functions that we would normally like to write, the idea is:
T.accessService(DbConnection)(({ get, put }) => ({ get, put }))
You don't want that in code.
To avoid repeating this you end-up writing commodity functions like:
export function get(k: string) {
return T.accessServiceM(DbConnection)((_) => _.get(k))
}
and use this instead of direct access.
We can automatically derive such implementations in many of the common function types except the case of functions containing generics.
Let's see it in practice:
import * as Array from "@effect-ts/core/Classic/Array"
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as L from "@effect-ts/core/Effect/Layer"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
import { tag } from "@effect-ts/system/Has"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
export const DbConnection = tag<DbConnection>()
// simulate a connection to a message broker
export interface BrokerConnection {
readonly send: (message: string) => T.UIO<void>
readonly clear: T.UIO<void>
}
export const BrokerConnection = tag<BrokerConnection>()
// connect to the database
export const DbLive = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(DbConnection)
)
// connect to the database
export const BrokerLive = pipe(
Ref.makeRef(<Array.Array<string>>Array.empty),
T.chain((ref) =>
T.effectTotal(
(): BrokerConnection => ({
send: (message) =>
pipe(ref, Ref.update<Array.Array<string>>(Array.snoc(message))),
clear: pipe(
ref.get,
T.chain((messages) =>
T.effectTotal(() => {
console.log(`Flush:`)
messages.forEach((message) => {
console.log("- " + message)
})
})
)
)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(BrokerConnection)
)
export const ProgramLive = L.all(DbLive, BrokerLive)
export const { get, put } = T.deriveLifted(DbConnection)(["get", "put"], [], [])
export const { send } = T.deriveLifted(BrokerConnection)(["send"], [], [])
// write a program that use the database
export const program = pipe(
T.do,
T.tap(() => put("ka", "a")),
T.tap(() => put("kb", "b")),
T.tap(() => put("kc", "c")),
T.bind("a", () => get("ka")),
T.bind("b", () => get("kb")),
T.bind("c", () => get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`),
T.tap(send)
)
// run the program and print the output
pipe(
program,
T.chain((s) =>
T.effectTotal(() => {
console.log(`Done: ${s}`)
})
),
// provide the layer to program
T.provideSomeLayer(ProgramLive),
T.runMain
)
The other 2 arrays of deriveLifted
are used to generate functions to access Constant Effects
and Constants as Effects
.
Like deriveLifted
other 2 functions can be used to derive commonly used utilities and those are deriveAccess
and deriveAccessM
.
It is left as an exercise to the reader to try out those combinations and look at the result function types in order to better understand the context of each derivation.
Another way to smooth the usage of services without the need to actually write any of the utility functions is to effectively write your main program as a service and leverage the layer constructor utilities to automate the environment wiring.
Let's see how we might do that:
import * as Array from "@effect-ts/core/Classic/Array"
import * as Map from "@effect-ts/core/Classic/Map"
import * as T from "@effect-ts/core/Effect"
import * as L from "@effect-ts/core/Effect/Layer"
import * as M from "@effect-ts/core/Effect/Managed"
import * as Ref from "@effect-ts/core/Effect/Ref"
import { pipe } from "@effect-ts/core/Function"
import type { NoSuchElementException } from "@effect-ts/system/GlobalExceptions"
import { tag } from "@effect-ts/system/Has"
// simulate a database connection to a key-value store
export interface DbConnection {
readonly put: (k: string, v: string) => T.UIO<void>
readonly get: (k: string) => T.IO<NoSuchElementException, string>
readonly clear: T.UIO<void>
}
export const DbConnection = tag<DbConnection>()
// simulate a connection to a message broker
export interface BrokerConnection {
readonly send: (message: string) => T.UIO<void>
readonly clear: T.UIO<void>
}
export const BrokerConnection = tag<BrokerConnection>()
// connect to the database
export const DbLive = pipe(
Ref.makeRef(<Map.Map<string, string>>Map.empty),
T.chain((ref) =>
T.effectTotal(
(): DbConnection => ({
get: (k) => pipe(ref.get, T.map(Map.lookup(k)), T.chain(T.getOrFail)),
put: (k, v) => pipe(ref, Ref.update(Map.insert(k, v))),
clear: ref.set(Map.empty)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(DbConnection)
)
// connect to the database
export const BrokerLive = pipe(
Ref.makeRef(<Array.Array<string>>Array.empty),
T.chain((ref) =>
T.effectTotal(
(): BrokerConnection => ({
send: (message) =>
pipe(ref, Ref.update<Array.Array<string>>(Array.snoc(message))),
clear: pipe(
ref.get,
T.chain((messages) =>
T.effectTotal(() => {
console.log(`Flush:`)
messages.forEach((message) => {
console.log("- " + message)
})
})
)
)
})
)
),
// release the connection via managed
M.make((_) => _.clear),
// construct the layer
L.fromManaged(BrokerConnection)
)
export function makeProgram({ get, put }: DbConnection, { send }: BrokerConnection) {
return {
main: pipe(
T.do,
T.tap(() => put("ka", "a")),
T.tap(() => put("kb", "b")),
T.tap(() => put("kc", "c")),
T.bind("a", () => get("ka")),
T.bind("b", () => get("kb")),
T.bind("c", () => get("kc")),
T.map(({ a, b, c }) => `${a}-${b}-${c}`),
T.tap(send)
)
}
}
export interface Program extends ReturnType<typeof makeProgram> {}
export const Program = tag<Program>()
export const ProgramLive = L.fromConstructor(Program)(makeProgram)(
DbConnection,
BrokerConnection
)
export const MainLive = pipe(ProgramLive, L.using(L.all(DbLive, BrokerLive)))
export const { main } = T.deriveLifted(Program)([], ["main"], [])
// run the program and print the output
pipe(
main,
T.chain((s) =>
T.effectTotal(() => {
console.log(`Done: ${s}`)
})
),
// provide the layer to program
T.provideSomeLayer(MainLive),
T.runMain
)
We basically introduced a new service Program
as a constructor makeProgram
that simply takes dependencies as arguments and we used the Layer
constructor fromConstructor
to glue everything.
We had to add the ProgramLive
layer to our final MainLive
cake and in the main function we called the effect from the program service (that can be easily derived as before).
Stay Tuned
In the next post of the series we will be discussing about generators
and how to get back to an imperative paradigm by using generators
to simulate a monadic do
that will look very similar to the dear old async/await but with much better typing.
Posted on October 19, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.