Originally published on crunchingnumbers.live

In the last two blog posts, you learned that Ember treats testing as a first-class citizen. Out of the box, Ember provides 3 types of tests so that you can fine-tune test coverage and performance. It also supports a variety of addons and debugging tools to improve your developer experience in testing.

Today, we address an important question: How should you write tests? By the end of this post, you will learn 5 simple rules that I like to follow. The rules aren't do-this-or-do-that's (cold hard facts). Instead, they carry nuance and interesting side stories. To keep your learning experience fun, I will transcribe my talk at EmberFest 2019 (rather than summarizing it) to engage in a dialogue with you.

0. Introduction

Hello. How is everyone doing? Show thumbs-up if you're doing awesome, thumbs-down if you're jet lagged and disoriented, and thumbs-middle if you're just fine.

That's what my yoga instructor likes to do before class: thumbs-up, thumbs-down, or so-so. It's a good exercise for your wrists.

a. Who Am I

My name is Isaac. I have been an Ember developer for two years. I'm still very new to web development.

Show of hands. How many of you are also new to Ember—two years or less? Please, give them a round of applause for taking time to attend EmberFest and hone their skills. Once again, show of hands please. How many of you are mentors to new developers? This time, I thank you from the bottom of my heart. I believe it's all of you—the mentors and the new developers—who will keep Ember strong and innovative as we move to the year 2020. That's just in two months. Can you believe that?

b. Overview

Today, I'll talk about two things that I love with passion: mathematics and writing tests. It's kinda funny. As an applied mathematician, I studied fracture mechanics. It's the study of how things break. Now, as a web developer, I'm still studying how things break—especially on IE 11 and Edge, the banes of our society.

Joking aside, I want to talk about writing tests especially for developers who are new to Ember like me, and for the mentors who are guiding them. I believe Ember gives new developers the power to feel productive—the power to feel good about themselves after work—from day one.

As a new developer, one of the things that you can do is to poke around the app and describe what you saw. That's exactly what writing tests is. The only question that remains is, how should you write tests? Are there best practices that you can follow?

I'm here to tell you, the answer is yes. Every one of you has the power to write tests like a mathematician. By the end of this talk, you will learn 5 simple rules that I like to follow when I write tests:

1. If, if, if
2. Use common, everyday words
3. Write less with theorems and new terms
4. All your basis are belong to us (Hmm, what does that mean?)
5. 1 picture = 1000 words ("A picture is worth a thousand words.")

c. Motivation

I came up with these rules based on my experience of writing proofs in college and graduate school. See, when I was in college, I wrote proofs that were horrendous—an abomination. I made plain-wrong assumptions; the logic didn't follow; and oftentimes, my proof worked for some cases but not all like it's supposed to. (Does that sound like you and your tests?) In college, I actually hated writing proofs, and sometimes I hated myself, because I wasn't good at it like all the other mathematicians.

It wasn't until I entered graduate school, when I had to study countless proofs day and night, and had to write many in return, that I began to see patterns—best practices, if you will—for writing proofs. Once I saw these patterns, each month, I got better at writing proofs, and soon, I was even able to teach others how to write theirs.

When I look back upon this experience, I wish I had known sooner in college how to write proofs because I'm sure I would have been fantastic at it. I wish I had had someone—a mentor—to show me how.

That's why I give this talk today to you. If you are a new developer, who perhaps feels inadequate when writing tests, I want you to know that you are not alone in how you feel—I was like you—that you can get better with practice like I did. If you are a mentor, I want you to think about what are some best practices that you can teach and how your mentees might feel when they write their tests.

1. If, If, If

With that said, let's get the show started! Rule number one—If, If, If—is about making good assumptions.

a. Motivation

When mathematicians write proofs—you will be surprised—they are actually telling you a story. They rely on conditional statements, also known as if-then statements, to describe what happens to a character in the beginning and what happens to them at the end. If-then statements allow mathematicians to connect different scenes to tell one cohesive story.

I think it's a powerful thought, that writing tests is nothing more than telling someone a story. When the user visited the signup page, they saw a form that was majestic. The user filled out the form with ease and the submit button was…enabled. The user clicked on submit and they were redirected to—whoa!—dashboard. They are now on level 2, hoping to find the princess in yet another castle. (Wouldn't it be awesome if your tests spoke to you like that?)

b. What to Watch Out for

Here's where things go wrong. To illustrate, I will share a personal fact. Yesterday was my birthday. I turned 31 and I'm this close to a mid-life crisis.

To cheer me up, I was hoping that we could do something special. We have about 200 people here. Mathematics tells us, there should be about 16 other people born in October like me. Raise your hands, October people.

Now, the rest of you, will you please sing Happy Birthday for me and these wonderful friends of yours whom you haven't met until yesterday? I promise you, afterwards, you will feel good about yourself. Are you ready? On the count of three…

In the beginning, I mentioned that rule #1 is about making good assumptions. On the surface, this test worked perfectly. I got personal, you sang Happy Birthday, and now you are happy (I hope).

Well, I lied. Yesterday wasn't my birthday. (Of course not. That's a 1 out of 365 chance.) I am 31, though, and I may or may not be close to a mid-life crisis. As you can see, not all assumptions were correct, yet you were able to sing Happy Birthday and feel happy.

This example, although it's contrived, illustrates a problem that you will often face. You have to write assertions based on assumptions, but in reality—when your tests are running—not all of your assumptions may have happened. To make matters worse, the reason for that may not be obvious, especially if you are new to Ember.

c. Making Wrong Assumptions

In my experience, there are 5 things that cause me to make wrong assumptions. I'll explain what they are and how Ember can help us solve these problems.

Keep in mind that these reasons are not at all obvious and deal with advanced topics such as state transitions. Don't feel overwhelmed if any of these are new. What I want you to see is that, when your tests fail, don't be discouraged. They are an opportunity to pause and ask yourself: Are my assumptions correct?

First, there are observers and computed properties. As your app grows, it becomes hard to ensure that these are triggered at the right time and all of your dependencies are listed and up-to-date. Things are better in Octane, thanks to async observers and tracked properties. Still, managing state is not an easy task and requires thoughtful design.

Second is data. The form builder that I use relies on observers and two-way bindings. As a result, the data that I expect to have and what actually exists in Ember Data store can be out of sync if I am not careful. In general, when you're designing components, please use Data Down, Actions Up.

Third is unsettled state. Ember's tests run synchronously, one assertion after another, but Ember's test helpers aren't aware of all async operations that happen in your app. If you animate elements or read files—things that take time—you will want to make sure that your app has settled before making assertions. You can use Ember's waitFor and waitUntil helpers to do so.

Fourth is leaky tests. They are a headache. Ideally, each of your tests runs in an isolated environment. Unfortunately, the assumptions made in one test can sneak into another if you're not careful with asyncs, global variables, mocks, and stubs. As a result, whether one test passes starts to depend on another. To my knowledge, the only solution here is prevention. Ember QUnit can find async leakage, while Ember Sinon can prevent stub leakage. You can also use Ember Exam to randomize test order to identify leaks early.

Last but not least is permissions. As developers, we are used to writing code as an admin because we get to see all parts of the application. As a result, we may end up biasing our tests thinking that the user does also. Please write tests for non-admins too. You can use nested modules to easily set up and group tests for admins and tests for non-admins.

2. Use Common, Everyday Words

Whew, that was a lot. Let's all take a deep breath.

I've been doing yoga since last December. In yoga, people practice ujjayi breath. You close your mouth and make a guttural sound in your throat like this. I swear, you will find it soothing every time except if you're on a video call, forget to turn off your mic, and everyone hears you, heaving. That happened to me in a meeting with the Ember Learning Team and the person who had to point out was Tom Dale. Talk about first impressions. (He was nice about it.)

a. Motivation

Speaking of video calls, rule number two is about how to communicate with others. Remember that, by writing tests, you are telling someone a story. You want to make sure that the other person understands you easily.

You can do so by using common, everyday words. Here, common means convention; it's something many people agreed to. Everyday means simple; something that many are familiar with.

b. Everyday

When mathematicians write proofs, they use everyday words, such as continuous and zero, to tell their story. You will be surprised that proofs aren't always about numbers and complex equations. There are words. There are pictures. There's always a story.

Just like mathematicians, you can write tests with words that you use in your daily life. Thanks to addons like Ember Test Selectors, Ember CLI Page Object, QUnit DOM, and Chai DOM, your tests can be easily understood by other developers.

c. Common

Mathematicians also follow a common language. For example, the letter f often stands for a function, while brackets and parentheses indicate the domain of f—the input of the function. Thanks to standard notations, mathematicians easily understand each other, no matter where they come from.

When writing tests, I recommend creating a standard that is both easy to remember and easy to type. For example, if you use Ember Test Selectors, always refer to links by data-test-link and buttons by data-test-button. To check data, you can use the name data-test-field in both edit and view modes.

You may have noticed that I've been using the label or ARIA label to identify data-test tags. This practice of identifying elements by what's visible on the screen nicely complements accessibility-driven testing. This is something that addons like Semantic Test Helpers are trying to solve.

3. Write Less with Theorems and New Terms

a. Motivation

Mathematicians are very much like developers. They love writing proofs that are short and DRY (don't repeat yourself).

b. Theorems

Remember this statement? Mathematicians can prove this with a single line, "Use the Intermediate Value Theorem." If they are really ambitious, they will just write "IVT" and call it a day!

It's absurd but a powerful idea. You can write short tests if you know that a part of your test is proven already. Short tests mean less code that you have to maintain later. That's a good thing. Mathematicians call what's already proven a theorem. We developers call it a test helper.

Thanks to Ember's new testing API, writing a test helper is simple. You write an ordinary function—async, in this case—to describe what the user does, then you call the function in your test. Think of writing test helpers as extracting a part of your test code for reusability.

Another test helper that I use every day is called authenticate. It assigns the user the right permissions in my test. Given these two examples, can you now think of ways to refactor your test code?

c. New Terms

There is another way for mathematicians to write less. It's to create a new term to describe an idea that happens over and over, in different contexts.

For example, you will see continuous functions everywhere—in linear algebra, calculus, topology, optimization, and machine learning. To avoid having to write continuous each time, mathematicians introduced the C-notation. Now, they can write less yet precisely tell their stories.

When you write tests, you will find assertions that happen over and over, but QUnit DOM or Chai DOM doesn't provide the API out of the box. It simply can't satisfy all of your business logic. But not to worry. Just like mathematicians who create new terms, you can create custom assertions.

A custom assertion, like test helpers, is just a function. You can define it however you'd like. Here, the assertion isEnabled hides the complexity of checking a button's state. It's a simple example. You can definitely get more ambitious by checking complex data structures such as tables and D3 visualizations—arrays and trees.

By creating good abstractions, you get to enjoy two benefits. One, you can write short, meaningful tests. (Again, short means maintainable.) Two, others can write similar tests without having to know the technical details on day one. They get to be productive thanks to you.

4. All Your Basis Are Belong to Us

a. Motivation

When mathematicians write proofs, they have to cover all possible cases. Often, that's an infinite amount. With tests, however, you can't cover infinite cases. Your tests would run forever!

Instead, you can find a basis to guarantee that your app works almost always, in the fewest number of tests possible.

b. Basis

In math, a basis means independent vectors that span the entire space. Span means "to cover" (like test coverage). You probably haven't heard of the term, basis, but I bet that you're already familiar with the idea.

Latitude and longitude are an example of a basis. These coordinates are independent and, for every combination, you can find a unique location in space, which is infinite. For example, (55.676°, 12.568°) takes you to Copenhagen and (30.267°, -97.743°) to Austin, where I come from. Just two numbers—three, counting elevation—to describe the infinite space.

RGB scheme is another example. There are infinitely many colors but each is a unique combination of red, green, and blue—three numbers.

For us, a basis means finding a small number of tests that can be considered building blocks. If we prove that our building blocks work, then mathematics tells us, our app works no matter what the user does.

c. Examples

Let's make this idea of a basis more concrete by looking at 3 examples.

Suppose you're designing this page where the user sees data and can take 5 different actions: Create, Edit, Delete, Clone, and Import. You want to prove that the table will always show the right data.

The problem is, there are infinitely many sequences of actions that the user can take. For example, they can create a record, then delete another. They can do import, edit, then clone. These are just two examples out of infinitely many. There's no way you can write a test for each and every possible sequence.

What if, instead, you guarantee that each action returns the user to the right state? When the user visits the page, everything works and they are happy. When the user takes an action, you make sure that they return to the happy state where everything works. That way, when the user exits the page, everything is correct.

The beauty of this approach is that, now, you only have to consider 5 test cases—one for each action. You can be confident that your app is in the right state when the user visits and exits the page.

Example two. Here is a component that I created to visualize LDAP filters. You can enter a filter to understand its logical structure and edit the expression with confidence. I can also find errors and help you fix them.

There are infinitely many expressions that an LDAP filter can take, but you can always break it down into these four basic types and three operations. 7 vectors to check.

The last example is a D3 component for visualizing a DSL (domain-specific language). You can interact with this graph to write your code in reverse.

Again, there are infinitely many programs that you can write, but the graph always boils down to a few basic types. It's how I know that my D3 visualization is correct.

5. 1 Picture = 1000 Words

Up till now, you learned several strategies to level up your testing. You have to make good assumptions, choose the right words, write short tests, and break down large problems into small ones.

I have to ask: Are you as overwhelmed as I? I simply had to read my script; you actually had to listen and learn, which is infinitely worse!

The final rule, in yoga terms, is shavasana. It's a moment to wind down, reflect on your journey, and wake up re-energized.

a. Motivation

One tragic thing about proofs and tests is that they ask us to be perfect. The things that we build are ambitious and complex, but if we make one mistake, a proof gets rejected by the community while an app faces vocal customers. It's a terrible feeling to have, to constantly worry whether our tests and our app are good enough.

The last rule that I want to share asks you to let go of your worries and be okay with not being perfect. What you go for instead is incremental change. Each day—each sprint, even—you work on writing better tests. The most important thing above all is to have fun—to enjoy what you do.

b. Start out Simple

Suppose you are designing a component that is complex and you're not sure how to write tests for it. Start out with the most simple test: assert.ok(true). Even this one line offers valuable information—namely, that your component won't blow up just by existing in your app.

Similarly, if you are working on a complex page, you don't have to write thorough application tests right away. You can use Percy or Backstop to take visual snapshots while you gradually introduce written tests.

It feels counterintuitive, right? That visual snapshots are as good as written tests?

c. Proof-by-Pictures

Let's take a look what mathematicians do one last time. I bet that, before, this meant very little to you because you didn't understand the words. Am I right?

Here's the same statement in drawing. From the picture, it's clear that, if my function is continuous and is positive on one end and negative on the other, it must equal to zero somewhere in-between. That's all the statement said!

Furthermore, from this picture, we can imagine a powerful technique called bisection method. It's how Git bisect works!

Let me show you one more example to illustrate that pictures are powerful. I will prove to you that there are as many numbers between 0 and 1 (this small little piece) as there are between -∞ and ∞ (all numbers). It's a mind-blowing fact about infinity, that a part can have the same size as the whole.

This green line represents all numbers between 0 and 1. Each number is a point on the line.

From the center, I'm going to draw a semicircle.

These dotted lines show that there are as many points on the green line as there are on the blue line. Do you agree with that?

From the center to each point on the circle, I can draw a straight line and stop at the bottom.

Just like before, these dotted lines show that there are as many points on the blue line as there are on the bottom green line. Notice that the bottom green line goes to infinity in both directions.

Therefore, you can find as many numbers between 0 and 1 as you can between -∞ and ∞. How's that for a slice of fried gold?

6. Conclusion

Let's go over the rules once more.

1. Writing tests is nothing more than telling someone a story. You describe what happens to a character in the beginning and what happens to them at the end. Please check whether your assumptions are met in your tests.

2. To tell stories effectively, you will want to use words that others can easily understand. Come up with a convention that you and your team would love to use every day.

3. Thanks to Ember's new testing API, you can create test helpers and custom assertions to write less code. You can write short, meaningful tests and help others be productive.

4. When you face a large problem, see if you can break it down into smaller ones. In particular, if you find independent vectors that span the entire problem, you can guarantee that your app works almost always.

5. Finally, it's okay not to be perfect. Be happy with who you are today and whom you can be tomorrow. The most important thing above all is to have fun—to enjoy writing tests.

It's time for me to start running and say goodbye for a little while, and I know you're going to miss me so I'll leave you with this. No, not an MCR song. A quote that I love most dearly.

To me, this quote is about continuous learning and not giving up when things are hard. It goes like this: "We shall not cease from exploration / and the end of all our exploring / will be to arrive where we started / and know the place for the first time."

I thank you very much for listening. If you'd like to know more about me, please reach out on LinkedIn and Discord.

Notes

"All Your Basis Are Belong to Us" is a reference to a video game. Basis is singular, just like base.

I said that a basis guarantees correctness almost always (theory suggests always) because how we write tests for the building blocks still matters. If we write bad ones, this rule won’t help much.

You can find my presentation slides here (Keynote, PDF).