Learning to write custom Expressions

Hi folks! In the following article, I want to discuss Expressions. For example, I'll show you how code works with an approach we habitually use and how to implement the same functionality with an Expression. In the end, we'll see performance benchmarks.

What is Expression?

Expression is an abstract class that provides the base class from which the classes representing expression tree nodes are derived. It also contains static factory methods to create the various node types. You often use Expression when doing LINQ queries. Under the hood of LINQ, Expression was implemented. LINQ is a wrapper for more convenient query use. Using LINQ, you can use methods that were implemented in LINQ. Expression allows you to create your custom queries and expressions. Expression has methods such as Expression.Property, Expression.Constant, Expression.Equal, and Expression.Call that generate dynamic code at runtime. It can be modified and executed in runtime. For what aims is it needed? The common aim is to use custom expressions to create dynamic queries.

Using LINQ

I created a simple console project. You do not need any NuGet packages. First of all, I added JSON data, which we need to parse and filter.

[
  {
    "FirstName": "Molly",
    "LastName": "Moore",
    "Age": 74
  },
  {
    "FirstName": "Scott",
    "LastName": "Alexander",
    "Age": 44
  },
  {
    "FirstName": "Sarah",
    "LastName": "Ortiz",
    "Age": 68
  }
]

You need a model to filter data, but the custom expression allows you to do it without a model.

public record Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
}

I created the Filter.cs class and added the LinqFiltering method.

    public void LinqFiltering()
    {
        var json = File.ReadAllText("data.json");

        var people = JsonSerializer.Deserialize<List<Person>>(json);

        if (people == null) return;
        foreach (var person in people.Where(x => x.Age > 52))
        {
            Console.WriteLine(JsonSerializer.Serialize(person));
        }
        Console.WriteLine();
    }

As you can see, we used the LINQ method Where, which uses Expressions under the hood. You can use more conditions for more complicated scenarios. As a rule, it covers the majority of cases.

Using Expression

What if you need dynamic or more complicated queries? Then, it helps you do it by using Expression. I added another JSON with the query:

{
  "Age": {
    "$gte": 52
  },
  "$or": [
    {
      "LastName": "Doe"
    }
  ]
}

I declared filter names. You can add different custom filters.
In the second step, I added the QueryParser.cs class and declared the Expression parameter:

    private static readonly ParameterExpression
        DictionaryParameter = Expression.Parameter(typeof
            (IReadOnlyDictionary<string, object>), "input");

Since the JSON structure is a key-value pair, using a Dictionary is more logical. In this class, we'll parse keys and values separately.

static Expression GetLeftValueExpression(JsonProperty
        parentProperty, JsonProperty property)
    {
        var keyParam =
            Expression.Constant(parentProperty.Name);
        var indexer = typeof(IReadOnlyDictionary<string,
            object>).GetProperty("Item");
        var indexerExpr = Expression.Property(
            DictionaryParameter, indexer, keyParam);
        return property.Value.ValueKind switch
        {
            JsonValueKind.Number =>
                Expression.Unbox(indexerExpr, typeof(int)),
            JsonValueKind.String =>
                Expression.TypeAs(indexerExpr, typeof(string)),
            JsonValueKind.True or JsonValueKind.False =>
                Expression.TypeAs(indexerExpr, typeof(bool)),
            _ => indexerExpr
        };
    }

The left side contains keys. This method sets each property and checks the type.

static Expression GetRightValueExpression(JsonProperty
        property)
    {
        return property.Value.ValueKind switch
        {
            JsonValueKind.Number =>
                Expression.Constant(property.Value.GetInt32()),
            JsonValueKind.String => Expression.Constant(
                property.Value.GetString()),
            JsonValueKind.True or JsonValueKind.False =>
                Expression.Constant(property.Value
                    .GetBoolean()),
            _ => Expression.Empty()
        };
    }

The right side contains values. This method sets values for each property and checks the type.

static Expression GetNestedFilterExpression(JsonProperty
        property)
    {
        Expression? currentExpression = null;
        foreach (var expressionProperty in
                 property.Value.EnumerateObject())
        {
            var getValueExpression = GetLeftValueExpression(
                property, expressionProperty);
            var valueConstantExpression =
                GetRightValueExpression(expressionProperty);
            Expression comparisonExpression =
                expressionProperty.Name switch
                {
                    "$lt" => Expression.LessThan(
                        getValueExpression, valueConstantExpression),
                    "$lte" => Expression.LessThanOrEqual(
                        getValueExpression, valueConstantExpression),
                    "$gt" => Expression.GreaterThan(
                        getValueExpression, valueConstantExpression),
                    "$gte" => Expression.GreaterThanOrEqual(
                        getValueExpression, valueConstantExpression),
                    _ => Expression.Empty()
                };
            if (currentExpression is not null)
            {
                currentExpression = Expression.And(
                    currentExpression, comparisonExpression);
            }
            else
            {
                currentExpression = comparisonExpression;
            }
        }
        return currentExpression ?? Expression.Empty();
    }

This method allows you to parse filters, set Expression conditions, and join them to one filter, just as you do in LINQ with the && operator.

static Expression GetFilterExpression(JsonProperty
        property)
    {
        return property.Value.ValueKind switch
        {
            JsonValueKind.Object =>
                GetNestedFilterExpression(property),
            _ => Expression.Equal(GetLeftValueExpression(
                property, property), GetRightValueExpression(
                property))
        };
    }

This method gets all filters from the left and right sides.

static Expression GetOrExpression(Expression expression,
        JsonProperty property)
    {
        foreach (var element in property.Value.EnumerateArray())
        {
            var elementExpression = GetQueryExpression(element);
            expression = Expression.OrElse(expression,
                elementExpression);
        }
        return expression;
    }

This method gets an additional filter equal to that used in LINQ as the || operator.

static Expression GetQueryExpression(JsonElement element)
    {
        Expression? currentExpression = null;
        foreach (var property in element.EnumerateObject())
        {
            Expression expression = property.Name switch
            {
                "$or" => GetOrExpression(currentExpression,
                    property),
                _ => GetFilterExpression(property)
            };
            if (currentExpression is not null && expression is
                    not BinaryExpression)
            {
                currentExpression = Expression.And(
                    currentExpression, expression);
            }
            else
            {
                currentExpression = expression;
            }
        }
        return currentExpression ?? Expression.Empty();
    }

This method handles the operators of the || and &&.

public static Expression<Func<IReadOnlyDictionary<string, object>,
        bool>> Parse(JsonDocument json)
    {
        var element = json.RootElement;
        var query = GetQueryExpression(element);
        return Expression.Lambda<Func<IReadOnlyDictionary<string,
            object>, bool>>(query, DictionaryParameter);
    }

For converting the Dictionary, use this custom converter:

public class DictionaryStringObjectJsonConverter : JsonConverter<IReadOnlyDictionary<string, object>>
{
    public override IReadOnlyDictionary<string, object>? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        var dictionary = new Dictionary<string, object>();
        if (reader.TokenType != JsonTokenType.StartObject)
        {
            throw new JsonException();
        }

        while (reader.Read())
        {
            if (reader.TokenType == JsonTokenType.EndObject)
            {
                return dictionary.ToFrozenDictionary();
            }

            string key = reader.GetString()!;

            reader.Read();
            object value;

            if (reader.TokenType == JsonTokenType.String)
            {
                value = reader.GetString()!;
            }
            else if (reader.TokenType == JsonTokenType.Number)
            {
                value = reader.GetInt32();
            }
            else if (reader.TokenType == JsonTokenType.True || reader.TokenType == JsonTokenType.False)
            {
                value = reader.GetBoolean();
            }
            else
            {
                value = JsonDocument.ParseValue(ref reader).RootElement.Clone();
            }

            dictionary[key] = value;
        }

        return dictionary.ToFrozenDictionary();
    }

    public override void Write(Utf8JsonWriter writer, IReadOnlyDictionary<string, object> value, JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        foreach (var kvp in value)
        {
            writer.WritePropertyName(kvp.Key);
            JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
        }
        writer.WriteEndObject();
    }
}

This method returns an Expression, which is called the lambda function, and returns filtered data.
To run it, you should add this code:

internal static class Program
{
    private static void Main()
    {
        var parse = new Filter();
        parse.LinqFiltering();
        parse.ExpressionFiltering();
    }
}

The result should be like this:

{"FirstName":"Molly","LastName":"Moore","Age":74}
{"FirstName":"Sarah","LastName":"Ortiz","Age":68}
{"FirstName":"Kevin","LastName":"Caldwell","Age":76}
{"FirstName":"Rachel","LastName":"Lawson","Age":78}
{"FirstName":"Timothy","LastName":"Williamson","Age":70}
{"FirstName":"Joseph","LastName":"Robinson","Age":73}
{"FirstName":"Tamara","LastName":"Mccoy","Age":57}
...

Benchmarks

Now, let's check the performance. We wrote a lot of code, and Expression seems to have the worst performance. Sure, it does not make any sense to use Expression for a few data. However, if you have at least 1000 items, it can be a good solution.

Let's modify Program.cs and add this code:

internal static class Program
{
    private static void Main()
    {
        // var parse = new Filter();
        // parse.LinqFiltering();
        // parse.ExpressionFiltering();
        BenchmarkRunner.Run<Filter>(new BenchmarkConfig());
    }
}

public class BenchmarkConfig : ManualConfig
{
    public BenchmarkConfig()
    {
        AddJob(Job.ShortRun
            .WithRuntime(CoreRuntime.Core90)
            .WithJit(Jit.Default)
            .WithPlatform(Platform.X64)
        );

        AddLogger(ConsoleLogger.Default);
        AddColumnProvider(BenchmarkDotNet.Columns.DefaultColumnProviders.Instance);
        AddExporter(BenchmarkDotNet.Exporters.HtmlExporter.Default);
    }
}

Remember to add the [Benchmark] attribute for the LinqFiltering() and ExpressionFiltering() methods.
If you run it, you'll get the benchmark result table:

As you can see, the result is approximately equal.

Conclutions

The Expression doesn't set a goal to replace LINQ filters. It has a narrow using area and should be used when dynamic queries are needed. Expression is complicated enough to resolve simple tasks.

The source code is here.

I hope this article was helpful to you. See you next week, and happy coding.

Blog

Learning to write custom Expressions

Serhii Korol

What is Expression?

Using LINQ

Using Expression

Benchmarks

Conclutions

Join Our Newsletter. No Spam, Only the good stuff.

Related