If you have used Ruby for any length of time you have probably, knowingly or not, used a gem. It could've been already installed on your system as part of your Ruby release. You may also have downloaded it with the RubyGems package manager. Irrespective of how you got hold of a gem, you required it as a dependency in your file and used it. But, have you ever wondered what a gem is? What's inside a Ruby gem? Having written a Ruby gem manifest file in the Writing a Ruby Gem Specification article, it's only natural to use it to learn how a Ruby package is built.

In this article, I'd like to take you to the RubyGems factory to see where it all begins. We will first learn how to create a Ruby gem and then unpack it and look inside the internals. Whether you intend to share your own gems or continue using third-party ones, knowing the inner workings of a gem will be useful. If anything it will make you appreciate the elegance with which the whole Ruby ecosystem works.

Now, on to the fun.

What is a Gem?

Any popular programming language has an ecosystem of packages that solve various problems. These packages capture reusable functionality and save tons of time for everyone else. To be accessible to everyone, libraries need a central storage place. A service for automated uploading and downloading of named packages referred to as - a registry.

Examples of popular registries are NPM (www.npmjs.com) for JavaScript, Hex (hex.pm) for Elixir and Erlang code, and Crates (crates.io) for Rust. Ruby also has an official place to store and distribute its packages: the rubygems.org. This is a service that provides you with the ability to browse and download hundreds of thousands of Ruby packages called a gem.

Each gem in a registry has a unique name and a release number. The Ruby community has a taste for quirky and whimsical names. Don't believe me, look at these:

• Cucumber - tool for running automated tests written in plain language.
• Jekyll - a static site generator.
• Nokogiri - HTML and XML parser.
• Sinatra - a minimal web framework.
• Unicorn - a web server.

The vibrant and large ecosystem of Ruby gems plays a considerable role in making the Ruby language as popular as it is now. To be part of this success and to create our own gems, we need to learn about the tool that makes it all possible, the RubyGems package manager.

The RubyGems Package Manager

When a Ruby language is installed on your system it includes a handful of gems. One of these gems is the RubyGems gem (this is a bit meta, a gem to manage gems). The RubyGems is a powerful package manager that will do things for you like:

• Installation of third-party gems.
• Dependencies resolution.
• Searching local and remote packages.
• Inspection of gems including their files and metadata.
• Building and serving RDoc documentation.
• Creation and publishing of gems.

This is not an exhaustive list but it should give you a taste of how many tasks a full-blown package manager handles. As of this writing, Ruby 2.7 installs RubyGems 3.1.2 on my system. The RubyGems gem provides you with a command-line tool that will support all the before listed tasks called the gem. I agree this is a bit confusing. Both a Ruby package and the tool to manage these packages are named the same.

You may also have used a tool called Bundler to install gems. Both Bundler and RubyGems are partners in the same crime. They help you above all manage gem dependencies. Bundler's focus is on dealing with the installation of gems for a local application. In contrast, RubyGems manages all the gems installed on your system under a given Ruby version. The plot thickens even more starting from Ruby version 2.6. In this version, the Bundler has been merged into the RubyGems as a default gem alongside packages like irb, logger or fileutils.

For us, the important feature of RubyGems is that it can convert a gem specification into a package. That's what we're going to take a closer look at next.

Building a Gem

Most likely you used the gem command to install or uninstall a dependency. There are more than 30 commands available at your fingertips. In particular, the gem build command is what we're going to look at. Its job is to take a manifest file and turn a bunch of files into a Ruby gem. The typical layout of a gem follows a well-established pattern for organising files. For example, the layout of the emoticon gem that we used to write a manifest file for looks like this:

emoticon/
├── exe
│   └── emote
├── lib
│   ├── emoticon
│   │   └── version.rb
│   └── emoticon.rb
├── CHANGELOG.md
├── LICENSE.txt
├── README.md
└── emoticon.gemspec

For our demonstration, we're going to use the already created emoticon.gemspec that contains all the information necessary to bake a new gem.

To create a gem from our gemspec, run the gem build command:

$ gem build emoticon.gemspec

This will display confirmation information in the console:

Successfully built RubyGem
Name: emoticon
Version: 0.1.0
File: emoticon-0.1.0.gem

At this point, you will have the emoticon-0.1.0.gem file in the same directory. Nice!

Under the covers the build command delegates its work to the Gem::Package class and the class level build method. This method accepts as an argument Gem::Specification instance and few options. That's exactly the instance we used to describe metadata about the emoticon gem in our manifest file. As a reminder here is the emoticon.gemspec snippet, for the sake of clarity:

# frozen_string_literal: true

require_relative "lib/emoticon/version"

gemspec = Gem::Specification.new do |spec|
  spec.name        = "emoticon"
  spec.version     = Emoticon::VERSION
  ...
end

Instead of relying on the gem build command, we will create a package manually to learn what's involved. To begin, we create a new file build.rb and load the "rubygems/package" dependency. As it happens, all we need to do to build a gem is add a couple lines of code. First, we load the gem specification. Next, we invoke the Gem::Package's build method with a gemspec as an argument:

require "rubygems/package"

gemspec = Gem::Specification.load("emoticon.gemspec")
Gem::Package.build(gemspec)

With these two lines in place, we can now execute our build.rb file:

$ ruby build.rb

This produces exactly the same result as before - an emoticon-0.1.0.gem file. Now, we have a gem for further exploration.

What's Inside a Gem?

The emoticon-0.1.0.gem has an unusual .gem extension name. But, this file is nothing more than an archived directory packaged using the tar Unix utility. We can take a look inside of the archive and list its content to the console with the tar utility. All we need to do is pass the -t flag for printing to the terminal and -f option for reading from a file:

$ tar -tf emoticon-0.1.0.gem

Inside the tar archive file, we find a set of three gzipped files:

checksums.yaml.gz
data.tar.gz
metadata.gz

If you build your gem and sign it with a cryptographic key then the archive will have a total of six files. Each gzipped file will have a matching signature file:

checksums.yaml.gz
checksums.yaml.gz.sig
data.tar.gz
data.tar.gz.sig
metadata.gz
metadata.gz.sig

What are these signature files?

To create a signature file, the RubyGems selects a digest algorithm from the standard library:

# lib/rubygems/security.rb

module Gem::Security
 DIGEST_ALGORITHM =
    if defined?(OpenSSL::Digest::SHA256)
      OpenSSL::Digest::SHA256
    elsif defined?(OpenSSL::Digest::SHA1)
      OpenSSL::Digest::SHA1
    else
      require 'digest'
      Digest::SHA512
    end
end

In the first step, the digest algorithm is applied to every file in the archive to create a summary of its content. The summary is a unique value of a fixed-length often referred to as a file checksum or digest. Then, a signing key and a certificate are combined into a signing instance of Gem::Security::Signer. The signer signs the file's digest and creates a digital signature that is then saved to a file with .sig extension. To see the content of the signature, a base64 encoded binary file, we can use the base64 command:

$ base64 data.tar.gz.sig

In this case, we see some random characters in the terminal output, truncated to save the space:

bAclqtMmrTKFSMFn74w8kwjRrmv/8Wlc1prNXxHfYtqHypaFdnoIOKKPhYAp3D5MKiqmqIsY8VTb8gQvbT/rEcMm40C57fh5JEQ3I7Tkvs0D76nL+cIaqnnqJ1H5Irknhkj+fu...

The signature can be thought of as a software equivalent of what we do in everyday life when we sign a paper document with a pen. By signing a document we verify its accuracy and authenticity. In our case, the signature files provide an extra layer of protection when installing gems in two ways. First, they allow us to verify that the gem source files haven't changed since they were originally signed. Second, they ensure that the signature belongs to the gem's author who owns the private signing key.

By default gems are installed as unsigned without checking their file signatures. This leaves the possibility open that a gem may have been tampered with by an attacker. To take advantage of the signatures during the gem's installation process, you need to use the - -trust-policy option or short flag -P with the highest and most secure policy:

$ gem install emoticon -P HighSecurity

With the signatures files out of the way, we can come back to our checksums, data and metadata compressed files. Our good friend the Gem::Package has more to teach us about how it performs its packaging process. Diving into the build method, we discover that all these three archived files are generated using the Gem::Package::TarWriter class. The tar writer does its work by exposing a few helper methods via an internal DSL (Domain Specific Language) that add content to the main gem archive file:

# lib/rubygems/package.rb

@gem.with_write_io do |gem_io|
  Gem::Package::TarWriter.new gem_io do |gem|
    add_metadata gem
    add_contents gem
    add_checksums gem
  end
end

Let's go ahead and extract all the files from the emoticon gem archive:

$ tar -xf emoticon-0.1.0.gem

One by one, we're going to examine files from the emoticon gem archive so we can learn their purpose.

Inspecting the Files

The first file that we're going to take under a microscope is a compressed YAML file called checksums.yaml.gz. The file compression is done with a lesser-known zlib gem from the Ruby standard library. This gem provides an interface to general purpose and portable zlib utility for compressing files. Among the formats it supports is the gzip (.gz) format.

In more detail, the Gem::Package offloads compression of all the archived files to the gzip_to method. This method accepts as an argument a file content represented as IO stream. Then it uses the Zlib::GzipWriter to turn the stream into a gzipped version:

# lib/rubygems/package.rb

def gzip_to(io)
  gz_io = Zlib::GzipWriter.new io, Zlib::BEST_COMPRESSION
  gz_io.mtime = @build_time
  yield gz_io
ensure
  gz_io.close
end

Since the checksums.yaml.gz is a binary file we need to do something to view its content. Fortunately, we don't have to unpack it to look inside, as we can use the gzcat utility:

$ gzcat checksums.yaml.gz

What we see inside at the top level are the keys that correspond with the names of the digest algorithms used. In our case, the sha256 and sha512 algorithms. Underneath each algorithm name are pairs of gzipped file name and its checksum. The checksums are digital summaries of the file content generated with the particular algorithm. For example, the file checksums under the sha256 key are represented as a string of 64 hexadecimal encoded characters. Thus it all looks something like this:

---
SHA256:
  metadata.gz: bdac75ac0dab99ec3cdb10692ef70f8b0966ef0aff7fb2c40b3c0ebf0440667d
  data.tar.gz: f0004274a83869c47e487eeb06c655b13c14601350eb8f6f4299ea6e0259b2d4
SHA512:
  metadata.gz: 03d8c338cee67683cc5e032a4e54153c9a73496e8960173a60b74b31fbe372e8786b86979efd6a923e0842dc132b2d4bdf020eff299b13a970018a31b5503325
  data.tar.gz: c5c52e2f82d90cc9d249b2dcc3500b3e29216f5db4049557b365f5051f34df19e5152519793b60b9691e1b951ec54fb6fa8e6c39ef673522cc1e19d72bb01bbf

The purpose of the checksums file is to help verify the integrity of the data.tar.gz and metadata.gz files during the gem installation process.

The next file we're going to zoom in on is the data.tar.gz. Since this is a gzipped tar archive, to see inside we use the familiar technique:

$ tar -ztf data.tar.gz

This file contains the source code files that were listed in the gem specification. Here is the list from emoticon gem:

CHANGELOG.md
LICENSE.txt
README.md
exe/emote
lib/emoticon.rb
lib/emoticon/version.rb

The final metadata.gz file is a YAML archive that includes all the metadata extracted from the gem's manifest file. If we look inside the file:

$ gzcat metadata.gz

We're greeted with a long output akin to the following truncated version:

--- !ruby/object:Gem::Specification
name: emoticon
version: !ruby/object:Gem::Version
  version: 0.1.0
platform: ruby
authors:
- Piotr Murach
autorequire: 
bindir: exe
cert_chain: []
date: 2020-03-01 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
  name: tomlrb
  requirement: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
  type: :runtime
  prerelease: false
  version_requirements: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
...

What's interesting here are all the !ruby/object notations. What are these? The YAML format allows us to serialize simple values like strings, arrays or hashes. But it also can serialize pretty much any Ruby object. By default, YAML will serialize all the object's instance variables and their values. For example, if we create a Gemspec class from a Struct object with some attributes:

require "yaml"
Gemspec = Struct.new(:name, :date, :version)
gemspec = Gemspec.new("emoticon", Time.now, "0.1.0")

And then print the object content in a YAML format to the console:

puts YAML.dump(gemspec)

We will see all the instance variables serialized:

--- !ruby/struct:Gemspec
name: emoticon
date: 2020-04-10 14:00:41.077868000 +01:00
version: 0.1.0

Yet, we're not stuck with only serializing object's attributes. We can control what attributes YAML is going to use with the to_yaml_properties method. For example, this is how the Gem::Requirement specifies it's YAML transformation under the hood:

# lib/rubygems/requirement.rb

class Gem::Requirement
  def to_yaml_properties # :nodoc:
    ["@requirements"]
  end
end

In our case, the entire Gem::Specification is serialized this way. All its attributes that may point to other objects are recursively serialized and saved in the YAML file.

If you wish to take a look at the specification of any gem installed on your system, you can run the gem spec command:

$ gem spec package-name

This will print all the gem's metadata information in your console.

You should also notice that the gem specification file is only needed when building a gem. It shouldn't be part of your release. The metadata is extracted to a YAML file and that's what is used during a gem installation.

Conclusion

You may have thought of gems as nondescript blobs of functionality that somehow appear on your system under some weird and wonderful name. My hope is that this article clarified the structure of a gem and how it accomplishes its functionality in the Ruby ecosystem.

After all, we learnt that there is nothing new under the sun. The old and trusted Unix tools for archiving and compression serve as the foundation for packaging gems. We experienced how the succinct YAML format keeps the data readable and machine friendly. We saw how the OpenSSL toolkit adds a security layer to make our gems resilient and dependable.

I don't know about you but I like to know the nitty-gritty details of the tools I use. I'm in awe and have nothing but great admiration for the developers who provide such solutions. It takes an ingenious effort to turn the concept of reusable code into a practical solution that scales and makes millions of Ruby projects possible!

Thanks for taking the time to read the article and I hope you found it useful.

Stay hungry and creative!

This article was originally published on PiotrMurach.com.

Photo by Jason D on Unsplash.