Introduction

Recently I've released a new gem, Matchable, which introduces class-level macro methods for Pattern Matching interfaces:

class Person
  include Matchable

  deconstruct :new
  deconstruct_keys :name, :age

  attr_reader :name, :age

  def initialize(name, age)
    @name = name
    @age  = age
  end
end

This post will explain the thinking behind Matchable and how it works.

Difficulty and Prerequisite Reading

Difficulty: Progressives - Requires some advanced knowledge, explores metaprogramming deeply.

This post will require prerequisite knowledge of the following:

• Ruby Method Decoration and Prepend (Decorating Ruby Series)
  • method_added hooks
  • prepending modules for attaching methods
• Pattern Matching (Pattern Matching Interfaces in Ruby)
• Destructuring (Destructuring Methods in Ruby)
• Class Eval (Idiosyncratic Eval)

I may later write a post on eval after this, but the above resources should be useful in the mean time for understanding the foundations of this post. While I will explain some of these concepts I will not go into great depth on them in this article.

Introducing Matchable

The idea behind Matchable was to create a nice porcelain interface for Pattern Matching hooks at the top level of a Ruby class:

class Person
  include Matchable

  deconstruct :new
  deconstruct_keys :name, :age

  attr_reader :name, :age

  def initialize(name, age)
    @name = name
    @age  = age
  end
end

Generated Code Results

This would generate the following effective code:

class Person
  MATCHABLE_KEYS = %i(name age)
  MATCHABLE_LAZY_VALUES = {
    name: -> o { o.name },
    age:  -> o { o.age },
  }

  def deconstruct
    to_a
  rescue NameError => e
    raise Matchable::UnmatchedName, e
  end

  def deconstruct_keys(keys)
    return { name: name, age: age } if keys.nil?

    deconstructed_values = {}
    valid_keys           = MATCHABLE_KEYS & keys

    valid_keys.each do |key|
      deconstructed_values[key] = MATCHABLE_LAZY_VALUES[key].call(self)
    end

    deconstructed_values
  rescue NameError => e
    raise Matchable::UnmatchedName, e
  end
end

Whereas other implementations would take a more dynamic approach like so:

class Person
  VALID_KEYS = %i(name age).freeze

  def deconstruct() = VALID_KEYS.map { public_send(_1) }

  def deconstruct_keys(keys)
    valid_keys = keys ? VALID_KEYS & keys : VALID_KEYS
    valid_keys.to_h { [_1, public_send(_1)] }    
  end
end

Why prefer the generated result? See for yourself. For objects with a substantial number of valid keys the performance difference is potentially 40-50% faster. The goal of this gem is not to produce 100% elegant code as much as produce code that will work quickly out of the box for a feature that may get very heavy use.

This all said, there's a lot behind that code, so let's start exploring shall we?

Prepended Modules

The first trick to getting macro-style methods like attr_ methods to work is including a module to make it work. In this case we're prepending so we can attach all of these methods to a distinctly named entity to make the backtrace a bit more readable later:

module Matchable
  MODULE_NAME = "MatchableDeconstructors".freeze

  def self.included(klass) = klass.extend(ClassMethods)

  module ClassMethods
    def deconstruct() = nil # TODO
    def deconstruct_keys(keys) = nil # TODO

    private def matchable_module
      if const_defined?(MODULE_NAME)
        const_get(MODULE_NAME)
      else
        const_set(MODULE_NAME, Module.new).tap(&method(:prepend))
      end
    end
  end
end

Make sure to read the Decorating Ruby Series first if this code does not look familiar.

We'll get to defining the other interface methods here in a bit, but we want to start with getting our initial module ready to tie-in the deconstruct and deconstruct_keys methods.

The matchable_module here is where we're going to be attaching all of our new methods for pattern matching. Think of it as a new module container for everything we want to inject put right before the class in terms of calling order:

MatchableDeconstructors > Object > ...

We could potentially use include here instead, but prepend opens us up to being able to hook on top of any deconstruction methods the user might make later, and leaves more API options open.

The idea is to have deconstruct define a deconstruct method on the module, and the same with deconstruct_keys. Now that we have our hook, let's look into the actual methods themselves.

Deconstruct (src)

Note that the source code is heavily commented and documented, you can find it in the header above.

deconstruct corresponds to Array-like matches in Ruby, and it's also the easier of the two to implement, so let's start here:

module Matchable
  module ClassMethods
    def deconstruct(method_name)
      return if matchable_module.const_defined?("MATCHABLE_METHOD")

      method_name = :initialize if method_name == :new
      matchable_module.const_set("MATCHABLE_METHOD", method_name)

      if method_defined?(method_name)
        attach_deconstructor(method_name)
        return [true, method_name]
      end

      @_awaited_deconstruction_method = method_name
      [false, method_name]
    end
  end
end

Constants and Reassignments

To start with you'll notice we're binding to the constant MATCHABLE_METHOD to say that we already have deconstruct defined. If it's not defined we set it shortly afterwards to gate this from being called more than once.

We're also making sure to reassign :new to :initialize, as :new is likely a more common interpretation. There's special behavior for htat methor we'll get to shortly.

Method Defined?

This is an interesting call:

if method_defined?(method_name)

Why do we need to check if the method is defined? Give it a second to consider before you keep reading, it made me scratch my head for a few minutes.

That's because macro-style methods are defined at the top of the class, before those same methods are initialized, and we need those methods to exist first as Symbols like attr_ methods won't work for us as you'll see in a moment.

So this for instance will break without that check:

class Person
  include Matchable

  deconstruct :new
  attr_reader :name, :age

  def initialize(name, age)
  end
end

...because initialize isn't defined yet. We'll get into why that's important for this code in a moment, but first we'll go over how we get around that issue.

Method Added

Right under that code you'll see a flag being set:

@_awaited_deconstruction_method = method_name

As you've either read from the prereads or seen before, there's a hook for method_added which will be called for every newly defined method. If it doesn't exist now, it will eventually, so we hook that:

def method_added(method_name)
  return unless defined?(@_awaited_deconstruction_method)
  return unless @_awaited_deconstruction_method == method_name

  attach_deconstructor(method_name)
  remove_instance_variable(:@_awaited_deconstruction_method)

  nil
end

If it's the method we're looking for we attach our deconstructor, otherwise we keep going. The nil return is because we don't care about the return and don't want people using that later.

Now that we have a way to intercept the methods when they finally get around to being created, we want to take a look at attach_deconstructor and how that works, and why this dance is even necessary.

Attach Deconstructor

To start with, if you haven't yet, be sure to read Destructuring Methods in Ruby unless you already know what parameters does and how that might be useful here in a second.

Now then, let's take a look at the attach_deconstructor method:

private def attach_deconstructor(method_name)
  deconstruction_code =
    if method_name == :initialize
      i_method    = instance_method(method_name)
      param_names = i_method.parameters.map(&:last)
      "[#{param_names.join(', ')}]"
    else
      method_name
    end

  matchable_module.class_eval <<~RUBY, __FILE__ , __LINE__ + 1
    def deconstruct
      #{deconstruction_code}
    rescue NameError => e
      raise Matchable::UnmatchedName, e
    end
  RUBY

  nil
end

Forking Initialize Behavior

We start by checking whether or not this is an initialize call, if it is we want to do something special. We want to take all the param names, wrap them in an Array, and return that instead of directly calling a method.

In the case of Person that'd return this:

def deconstruct() = [name, age]

Just make sure those attr_ methods are in there or you'll have issues.

Regular Method Behavior

If it's not initialize we return the method name to call.

That makes this work a lot like alias, except we want to wrap it in a custom exception, so it'd end up looking more like this for to_a:

def deconstruct
  to_a
rescue NameError => e
  raise Matchable::UnmatchedName, e
end

This is done to wrap NameError with an extra message to indicate a few methods are missing to make this thing work.

Class Eval

This comes to the interesting part: class_eval.

We're using this to evaluate the code we've assembled a bit more manually inside of the module we defined above, which is then prepended before the class to give it our pattern matching methods:

matchable_module.class_eval <<~RUBY, __FILE__ , __LINE__ + 1
  def deconstruct
    #{deconstruction_code}
  rescue NameError => e
    raise Matchable::UnmatchedName, e
  end
RUBY

Why not use a block? Why a String? Because we want to do some things we can't do unless we make Ruby write its own Ruby, and for that we need eval methods to "compile" things for us to give us a more optimized method.

The mentions of __FILE__ and __LINE__ + 1 here are to make sure that this evaluated code is put in the proper flow of our Ruby program for debuggers, backtraces, and other tools to find it later.

Read more into why this is here, but the short version is that it's much easier to troubleshoot and work with later.

That brings us to the end of deconstruct, which means our next section on deconstruct_keys is coming up, and it's a trip.

Deconstruct Keys (src)

Note that the source code is heavily commented and documented, you can find it in the header above.

Now this is a fair bit more complicated than deconstruct as we have a few considerations to work with:

1. We have a list of keys to deal with for optimization purposes
2. Those keys mean different responses
3. If keys are nil we need to return every possible key

That can make this a bit harder to reason about and dynamically compile against, but there are still tricks for making this work.

Let's start with a look at the code:

def deconstruct_keys(*keys)
  return if matchable_module.const_defined?('MATCHABLE_KEYS')

  sym_keys = keys.map(&:to_sym)

  matchable_module.const_set('MATCHABLE_KEYS', sym_keys)
  matchable_module.const_set('MATCHABLE_LAZY_VALUES', lazy_match_values(sym_keys))

  matchable_module.class_eval <<~RUBY, __FILE__ , __LINE__ + 1
    def deconstruct_keys(keys)
      if keys.nil?
        return {
          #{nil_guard_values(sym_keys)}
        }
      end

      deconstructed_values = {}
      valid_keys           = MATCHABLE_KEYS & keys

      valid_keys.each do |key|
        deconstructed_values[key] = MATCHABLE_LAZY_VALUES[key].call(self)
      end

      deconstructed_values
    rescue NameError => e
      raise Matchable::UnmatchedName, e
    end
  RUBY

  nil
end

Constants and Keys

To start out with we have a few constants being defined, and our keys are getting mapped to Symbols just to be sure:

return if matchable_module.const_defined?('MATCHABLE_KEYS')

sym_keys = keys.map(&:to_sym)

matchable_module.const_set('MATCHABLE_KEYS', sym_keys)
matchable_module.const_set('MATCHABLE_LAZY_VALUES', lazy_match_values(sym_keys))

We're using MATCHABLE_KEYS here as our VALID_KEYS to guard against unknown values. If it's defined we know we're already done with this method and can return early.

If it's not defined we get our Symbol keys and set it.

Afterwards we have something interesting, MATCHABLE_LAZY_VALUES. Let's take a look into that.

Matchable Lazy Values

What exactly is this constant doing? It's providing a mapping of method_name to a lazy way to fetch its value:

def lazy_match_values(method_names)
  method_names
    .map { |method_name| "  #{method_name}: -> o { o.#{method_name} }," }
    .join("\n")
    .then { |kv_pairs| "{\n#{kv_pairs}\n}"}
    .then { |ruby_code| eval ruby_code }
end

keys are synonymous to method_names in this gem, as we use methods to fetch values. With each of those names we want to create a Hash key-value pair, the method_name pointing to a lambda that takes an Object and calls the method directly on it. For name it would look like this:

name: -> o { o.name }

This is faster than public_send by a decent amount, and provides us a way to get at individual values without calculating them directly at match time.

The original method to do this was:

<<~RUBY
  if keys.nil? || keys.include?(:#{method_name})
    deconstructed_values[:#{method_name}] = method_name
  end
RUBY

...which incurred the unfortunate inner-loop, slowing down the code substantially.

Anyways, after we have those key-value pairs we join them together, wrap them in Hash brackets, and eval it into a Ruby Hash and we're off to the races again. We'll get into what this is used for here in a moment.

If keys are nil

Back to our class_eval, we'll see this code:

matchable_module.class_eval <<~RUBY, __FILE__ , __LINE__ + 1
  def deconstruct_keys(keys)
    if keys.nil?
      return {
        #{nil_guard_values(sym_keys)}
      }
    end

    # ...

Remembering back a bit, if keys are nil we need to return every value. The above code removed the necessity of the if keys.nil? || keys.include? check, but didn't do anything about the nil case.

That's where this comes into play with nil_guard_value.

nil Guard Values

This method is actually rather boring:

def nil_guard_values(method_names)
  method_names
    .map { |method_name| "#{method_name}: #{method_name}" }
    .join(",\n")
end

It transforms all the method names into key-value pairs without the added need for laziness in a partial match, joins them together, and leaves it up to the code above to wrap it in Hash brackets to give us our branch for all keys being required.

Deconstructed Values and Valid Keys

After we deal with the all keys branch we need to filter things down. To start with we want a place to store our new values, and a set of keys we should be returning:

deconstructed_values = {}
valid_keys           = MATCHABLE_KEYS & keys

The Hash is self-explanatory for adding more values to, but valid_keys is a bit more interesting. What we're doing here is finding the intersection between the valid matchable keys we know, and the keys we were provided. This is to make sure we don't get any strange keys that we don't know how to handle.

Getting Lazy Values

Next up we want to actually get the values based on those valid keys:

valid_keys.each do |key|
  deconstructed_values[key] = MATCHABLE_LAZY_VALUES[key].call(self)
end

deconstructed_values

We iterate over every one of the valid keys and set a value in deconstructed_values by referencing our lazy value fetcher we mentioned above. We call this with self to get the value out of the object, and after we're done here we return back all the values we've extracted.

Musing on Macros

WARNING: The below musing is not valid Ruby, just me dreaming

There is part of me that wishes I could do somthing like this instead of hacking around public_send being slower:

valid_keys.each do |key|
  deconstructed_values[key] = ${key}
end

deconstructed_values

...where ${} would be a macro system that could directly inline the associated code rather than need to send the value. Matz mentioned we may one day get macros, but today is not that day, so we continue to dream a bit while we add new and interesting patches to get around it.

That gets us to the end of deconstruct_keys, so let's wrap this up.

Benchmarking

So is this all worth it? Feel free to take a look at the benchmarks to find out, but I'll summarize here.

We have two types of objects being tested, one with two attributes and one with twenty six. Here's what we came up with, with Dynamic being the normal way and Macro being the way mentioned above:

Person (2 attributes):
  Full Hash:
    Dynamic: 5.027M
    Macro:   5.542M
    Gain:    9.3%
  Partial Hash:
    Dynamic: 7.551M
    Macro:   8.436M
    Gain:    10.5%
  Array:
    Dynamic: 7.105M
    Macro:   10.689M
    Gain:    33.5%

BigAttr (26 attributes):
  Full Hash:
    Dynamic: 984.300k
    Macro:   3.248M
    Gain:    69.7%
  Partial Hash:
    Dynamic: 2.594M
    Macro:   2.956M
    Gain:    12.3%
  Array:
    Dynamic: 1.488M
    Macro:   7.957M
    Gain:    81.3%

The more attributes you have the more gains this starts to see, especially around Arrays, though I would suggest against having that many attributes to match against in an Array.

So was it worth it? Honestly most of this was a test to see how it would work, the performance gains on things were a convenient side effect. I do not think the gains are substantial enough to justify implementing this all yourself, but if you want to use this gem to get those benefits you're more than welcome to.

Wrapping Up

That was quite a ride through a lot of metaprogramming, and a lot of interesting things in Ruby. Some of it I had to spend a bit researching and musing on before I came to decent solutions, but that's most of the fun of it.

If I entirely know what I'm doing what's the fun in that? It's fun to go and explore, to try new things, and to test the boundaries of what's possible in programming. Go out and give it a try yourself sometime, it's great fun!

Anyways, that's all I have for this one, I hope you've enjoyed this little dive into using eval for compiling dynamic code.