Exploring testing automation with AI tools

Intro

I am skeptical about all this AI hype, I did try AI to generate the code, tried few LLM as well. If you ask AI to solve some common problem that was solved dozens of times before, it can do it, but for a real application that contains some custom logic it can't.
Can't really use "autocomplete on steroids" for writing production code.

But, in some cases AI models could be useful, for example writing some boilerplate unit tests, generating JSDoc comments (at least a boilerplate for it), and translating documentation.

Action!

Let's see it in action with real code.

I'm stumbled upon this service www.unit-test.dev, I'm not an author and not affiliated with them in any way.
Will use my library for Next.js, I wrote that code, and it is complex enough to be considered a real world example.

Low level pure function

Let's see how it can handle relatively simple low-level function.

export function encode(payload: unknown): string {
  const type = typeOf(payload);

  switch (type) {
    case 'function':
    case 'symbol':
      return '';
    case 'date':
      return SYMBOLS.date + (payload as Date).toISOString();
    case 'string':
      return `${SYMBOLS.string}${encodeURIComponent(payload as string)}`;
    case 'number':
      return SYMBOLS.number + String(payload as number);
    case 'boolean':
      return SYMBOLS.boolean + String(payload as boolean);
    case 'object':
    case 'array':
      return JSON.stringify(payload as object, replacer).replaceAll('"', "'");
    case 'null':
      return SYMBOLS.null;
    case 'undefined':
      return SYMBOLS.undefined;
    default:
      return String(payload);
  }
}

const replacer = (key: string, value: unknown) => {
  const type = typeOf(value);
  return key && type !== 'object' && type !== 'array' ? encode(value) : value;
};

const typeOf = (val: unknown): Type => {
  const isNull = val === null;
  const isArray = Array.isArray(val);
  const isDate = val instanceof Date;
  const isObject = !isNull && !isDate && !isArray && typeof val === 'object';

  return (
    (isNull && 'null') ||
    (isDate && 'date') ||
    (isArray && 'array') ||
    (isObject && 'object') ||
    typeof val
  );
};

const SYMBOLS = {
  string: '◖',
  boolean: '🗵',
  null: '∙null',
  undefined: '∙undefined',
  number: '∓',
  date: '⏲',
};

Model generated this tests


// All necessary imports here
import { encode } from './encoder'; // File path should correspond to where the function is stored.

describe('encode function testing', () => {
  test('encodes strings correctly', () => {
    expect(encode('hello')).toEqual('◖hello');
    expect(encode('foo bar')).toEqual('◖foo%20bar');
    expect(encode('')).toEqual('◖');
  });

  test('encodes numbers correctly', () => {
    expect(encode(123)).toEqual('∓123');
    expect(encode(-123.456)).toEqual('∓-123.456');
    expect(encode(0)).toEqual('∓0');
  });

  test('encodes booleans correctly', () => {
    expect(encode(true)).toEqual('🗵true');
    expect(encode(false)).toEqual('🗵false');
  });

  test('encodes null and undefined correctly', () => {
    expect(encode(null)).toEqual('∙null');
    expect(encode(undefined)).toEqual('∙undefined');
  });

  test('encodes dates correctly', () => {
    const date = new Date(2023, 0, 1); // Jan 1, 2023 expect(encode(date)).toEqual(`⏲${date.toISOString()}`);
  });

  test('encodes arrays correctly', () => {
    expect(encode([1, 'two', true])).toEqual("['∓1','◖two','🗵true']");
    expect(encode([])).toEqual('[]');
  });

  test('encodes objects correctly', () => {
    expect(encode({ a: 1, b: 'two', c: true })).toEqual("{'a':'∓1','b':'◖two','c':'🗵true'}");
    expect(encode({})).toEqual('{}');
  });

  test('encodes symbols and functions by returning an empty string', () => {
    const symbol = Symbol('symbol');
    expect(encode(symbol)).toEqual('');
    expect(encode(() => {})).toEqual('');
  });

  test('handles unsupported types by coercing them to strings', () => {
    // Assuming BigInt is unsupported by the rest of the code
    expect(encode(BigInt(1234))).toEqual('1234'); // Direct coercion of BigInt to string might be required });

  test('checks robustness with nested complex objects', () => {
    const complexObj = {
      num: 1,
      str: "test",
      bool: false,
      date: new Date('2021-01-01T00:00:00.000Z'),
      arr: [1, 'two', { a: 3 }],
      nested: { a: "hello", b: true },
    };
    const result = encode(complexObj);
    const expectedValue = "{'num':'∓1','str':'◖test','bool':'🗵false','date':'⏲2021-01-01T00:00:00.000Z','arr':['∓1','◖two',{'a':'∓3'}],'nested':{'a':'◖hello','b':'🗵true'}}";
    expect(result).toEqual(expectedValue);
  });
});

Most of the generated tests are similar to tests that I wrote, that is pretty good.

Except here expect become part of the comment somehow const date = new Date(2023, 0, 1); // Jan 1, 2023 expect(encode(date)).toEqual(⏲${date.toISOString()});

Let's run generated tests

Code lack of }) at the end, after it fixed.

Now it somehow put one test case inside another test, easy enough to fix it.

Now almost all tests are passing, that good.
Last test case is weird, is there a bug? Looks like it, I did test date case, but not inside an object. However, it works correctly in a demo page and related high-level Playwright tests.
Culprit is replacer in JSON.stringify works not as I expected, it's calling .toString() on a date instance and it's mutating object.

Verdict

Some AI tools could be used to generate simple unit tests and save time.

Tnx for reading, share your experience in comments, or maybe you want to see more example (React component for example).

Blog

Exploring testing automation with AI tools

Alex

Intro

Action!

Low level pure function

Let's run generated tests

Verdict

Join Our Newsletter. No Spam, Only the good stuff.

Related