TheLazy.Dev
Portfolio GitHub

Blog

serverless |aws |devops |productivity

AWS Step Functions, Intrinsic Functions In-Depth | Serverless

awedis

awedis

Posted on March 7, 2023

AWS Step Functions, Intrinsic Functions In-Depth | Serverless

AWS Step Functions enables you to orchestrate serverless workflows by integrating with different AWS services.

📋 Note: If you want to read more about AWS Step Functions before starting this article you can read AWS Step Functions In-Depth | Serverless

The main parts of this article:

  1. About Intrinsic functions
  2. Examples

1. what are Intrinsic functions?

Intrinsic functions allows you to run directly on your state machine some simple processing.

Well, let's take a look at how we used to manipulate our data in Step Functions. In order to do so, we needed a state and a Lambda function which would take the data as input, update it, and make all the necessary changes before passing the data to the next state. However, upon closer examination, it became clear that our architecture required excessive resources and involved too many steps.

The Amazon States Language provides several intrinsic functions, also known as intrinsics, that can help you perform basic data processing operations without using a Task state. These intrinsics are constructs that resemble functions in programming languages and can assist payload builders in processing the data that goes to and from the Resource field of a Task state.

Using intrinsic functions can help you build applications where you don't have to rely on Lambda functions, which can save you time and money, reduce complexity (by consolidating logic in one place), minimize throughput, and reduce the payload between different tasks.

Examples

Now let's build simple examples and try all the options that we have on Intrinsic functions, the following are the different operations that we will cover

  • Intrinsics for arrays: States.Array, States.ArrayPartition, States.ArrayContains, States.ArrayRange, States.ArrayGetItem, States.ArrayLength, States.ArrayUnique

  • Intrinsics for data encoding and decoding: States.Base64Encode, States.Base64Decode

  • Intrinsic for hash calculation: States.Hash

  • Intrinsics for JSON data manipulation: States.JsonMerge, States.StringToJson, States.JsonToString

  • Intrinsics for Math operations: States.MathRandom, States.MathAdd

  • Intrinsic for String operation: States.StringSplit

  • Intrinsic for unique identifier generation: States.UUID

  • Intrinsic for generic operation: States.Format

Before adding intrinsic functions, let's first build our Step Functions. To keep it simple, we will have only two states, and a successful return at the end. Furthermore, our Lambda functions for each state will be simple, only logging out the data and passing the value to the next state.

firstLambdaARN: &FIRST_ARN arn:aws:lambda:${env:region}:${env:accountId}:function:${self:service}-${env:stage}-firstState
secondLambdaARN: &SECOND_ARN arn:aws:lambda:${env:region}:${env:accountId}:function:${self:service}-${env:stage}-secondState
states:
  IntrinsicExample:
    name: IntrinsicExample
    definition:
      Comment: "Intrinsic Functions Example"
      StartAt: firstState
      States:
        firstState:
          Type: Task
          Resource: *FIRST_ARN
          Next: secondState
        secondState:
          Type: Task
          Resource: *SECOND_ARN
          Next: success
        success:
          Type: Succeed
Enter fullscreen mode Exit fullscreen mode 
"use strict";

module.exports.firstState = async (event) => {
  console.log('firstState event =>', event);

  return event;
};

module.exports.secondState = async (event) => {
  console.log('secondState event =>', event);

  return {
    status: 'SUCCESS'
  };
};
Enter fullscreen mode Exit fullscreen mode

Everything seems ready, let's start adding our intrinsic functions one by one and start seeing the results.

📋 Note: I'm not going with description of each intrinsic function, if you want more documentation how each one is working you can visit the link Intrinsic functions

To use intrinsic functions you must specify .$ in the key value in your state machine definitions.

You can nest up to 10 intrinsic functions within a field in your workflows.

Intrinsics for arrays

In the first one we are going to pass the whole payload as it is to input also we are going to get the length. Inside the state we can see that Parameters are now added

  • States.ArrayLength 
firstLambdaARN: &FIRST_ARN arn:aws:lambda:${env:region}:${env:accountId}:function:${self:service}-${env:stage}-firstState
secondLambdaARN: &SECOND_ARN arn:aws:lambda:${env:region}:${env:accountId}:function:${self:service}-${env:stage}-secondState
states:
  IntrinsicExample:
    name: IntrinsicExample
    definition:
      Comment: "Intrinsic Functions Example"
      StartAt: firstState
      States:
        firstState:
          Type: Task
          Resource: *FIRST_ARN
          Parameters:
            input.$: "$"
            length.$: "States.ArrayLength($.inputArray)"
          Next: secondState
        secondState:
          Type: Task
          Resource: *SECOND_ARN
          Next: success
        success:
          Type: Succeed
Enter fullscreen mode Exit fullscreen mode

Perfect! Now we need to execute our Step Function. The easiest way to do so is by creating a simple function that can be triggered from an API call. We can then use the AWS SDK to run our Step Function.

"use strict";
const { StepFunctions } = require('aws-sdk');
const stepFunctions = new StepFunctions();

module.exports.runIntrinsicFunction = async (event) => {
  try {
    const stepFunctionResult = stepFunctions.startExecution({
      stateMachineArn: process.env.INTRINSIC_EXAMPLE_STEP_FUNCTION_ARN,
      input: JSON.stringify({
        inputArray: [
          {
            id: 1,
            title: 'book',
          },
          {
            id: 2,
            title: 'pen',
          },
          {
            id: 3,
            title: 'watch',
          },
          {
            id: 4,
            title: 'laptop',
          }
        ]
      })
    }).promise();
    console.log('stepFunctionResult =>', stepFunctionResult);

    return {
      statusCode: 200,
      body: JSON.stringify({
        message: `This is test API`,
      }, null, 2),
    };
  } catch (error) {
    console.log(error);
  }
};
Enter fullscreen mode Exit fullscreen mode

Now we can see that our first state takes the following input. It's amazing to see how powerful this is and how it can help minimize our code logic. With this new functionality, we can simplify our code and make it more efficient. Let's now move on to the other examples.
Image description

  • States.Array, States.ArrayPartition

Payload:

input: JSON.stringify({
  inputInteger: 12345678,
  inputForPartition: [1, 2, 3, 4, 5, 6, 7, 8]
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  buildId.$: "States.Array($.inputInteger)"
  partition.$: "States.ArrayPartition($.inputForPartition, 2)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "partition": [
    [
      1,
      2
    ],
    [
      3,
      4
    ],
    [
      5,
      6
    ],
    [
      7,
      8
    ]
  ],
  "BuildId": [
    12345678
  ],
}
Enter fullscreen mode Exit fullscreen mode
  • States.ArrayContains, States.ArrayRange

Payload:

input: JSON.stringify({
  inputArray: [1,2,3,4,5,6,7,8,9],
  lookingFor: 8
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  containsResult.$: "States.ArrayContains($.inputArray, $.lookingFor)"
  rangeResult.$: "States.ArrayRange(1, 20, 3)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "containsResult": true,
  "rangeResult": [
    1,
    4,
    7,
    10,
    13,
    16,
    19
  ]
}
Enter fullscreen mode Exit fullscreen mode
  • States.ArrayGetItem, States.ArrayUnique

Payload:

input: JSON.stringify({
  inputArray: [1,2,3,3,3,3,3,3,4],
  index: 5,
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  getItemResult.$: "States.ArrayGetItem($.inputArray, $.index)"
  uniqueResult.$: "States.ArrayUnique($.inputArray)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "getItemResult": 3,
  "uniqueResult": [
    1,
    2,
    3,
    4
  ]
}
Enter fullscreen mode Exit fullscreen mode

Intrinsics for data encoding and decoding

  • States.Base64Encode, States.Base64Decode

Payload

input: JSON.stringify({
  input: "Data to encode",
  base64: "RGF0YSB0byBlbmNvZGU="
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  encodeResult.$: "States.Base64Encode($.input)"
  decodeResult.$: "States.Base64Decode($.base64)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "decodeResult": "Data to encode",
  "encodeResult": "RGF0YSB0byBlbmNvZGU="
}
Enter fullscreen mode Exit fullscreen mode

Intrinsic for hash calculation

  • States.Hash The hashing algorithm: MD5, SHA-1, SHA-256, SHA-384, SHA-512

Payload:

input: JSON.stringify({
  Data: "input data", 
  Algorithm: "SHA-1"
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  result.$: "States.Hash($.Data, $.Algorithm)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "result": "aaff4a450a104cd177d28d18d74485e8cae074b7"
}
Enter fullscreen mode Exit fullscreen mode

Intrinsics for JSON data manipulation

  • States.JsonMerge, States.StringToJson, States.JsonToString

Payload:

input: JSON.stringify({
  json1: { "a": {"a1": 1, "a2": 2}, "b": 2, },
  json2: { "a": {"a3": 1, "a4": 2}, "c": 3 },
  escapedJsonString: "{\"foo\": \"bar\"}",
  unescapedJson: {
    foo: "bar"
  }
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  output.$: "States.JsonMerge($.json1, $.json2, false)"
  toJsonResult: "States.StringToJson($.escapedJsonString)"
  toStringResult: "States.JsonToString($.unescapedJson)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "output": {
    "a": {
      "a3": 1,
      "a4": 2
    },
    "b": 2,
    "c": 3
  },
  "toJsonResult": {
    "foo": "bar"
  },
  "toStringResult": "{\"foo\":\"bar\"}"
}
Enter fullscreen mode Exit fullscreen mode

Intrinsics for Math operations

  • States.MathRandom, States.MathAdd

Payload:

input: JSON.stringify({
  start: 1,
  end: 999,
  value: 100,
  step: 2
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  randomResult.$: "States.MathRandom($.start, $.end)"
  additionResult.$: "States.MathAdd($.value, $.step)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "randomResult": 110,
  "additionResult": 102
}
Enter fullscreen mode Exit fullscreen mode

Intrinsic for String operation

  • States.StringSplit

Payload:

input: JSON.stringify({
  inputStringOne: "1,2,3,4,5",
  splitterOne: ",",
  inputStringTwo: "This.is+a,test=string",
  splitterTwo: ".+,="
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  resultOne.$: "States.StringSplit($.inputStringOne, $.splitterOne)"
  resultTwo.$: "States.StringSplit($.inputStringTwo, $.splitterTwo)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "resultOne": [
    "1",
    "2",
    "3",
    "4",
    "5"
  ],
  "resultTwo": [
    "This",
    "is",
    "a",
    "test",
    "string"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Intrinsic for generic operation

  • States.UUID

Intrinsic:

Parameters:
  uuid.$: "States.UUID()"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "uuid": "c23c5e29-e9df-4507-8152-c677792511a4"
}
Enter fullscreen mode Exit fullscreen mode

Intrinsic for generic operation

  • States.Format

Payload:

input: JSON.stringify({
  template: "Hello, this is {}."
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  formatResult.$: "States.Format($.template, $.name)"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "formatResult": "Hello, this is Awedis."
}
Enter fullscreen mode Exit fullscreen mode

Now lets see how we can nest some intrinsic functions together.

First I want to merge two objects together and then escape it.

Payload:

input: JSON.stringify({
  json1: { id: '001', title: 'book' },
  json2: { id: '001', category: 'products', label: 'business' }
})
Enter fullscreen mode Exit fullscreen mode

Intrinsic:

Parameters:
  time.$: "$$.State.EnteredTime"
  output.$: "States.JsonToString(States.JsonMerge($.json1, $.json2, false))"
Enter fullscreen mode Exit fullscreen mode

Result:

{
  "output": "{\"id\":\"001\",\"label\":\"business\",\"title\":\"book\",\"category\":\"products\"}",
  "time": "2023-03-05T13:17:18.682Z"
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

Using intrinsic functions can make your work easier since software engineers use these functions almost daily. Intrinsic functions can help minimize and simplify your code, making it much cleaner.

This article is part of the "Messaging with Serverless" series that I have been writing for a while. If you are interested in reading about other serverless offerings from AWS, feel free to visit the links below.

💖 💪 🙅 🚩
awedis
awedis

Posted on March 7, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

AWS Step Functions, Intrinsic Functions In-Depth | Serverless
serverless AWS Step Functions, Intrinsic Functions In-Depth | Serverless

March 7, 2023