TheLazy.Dev
Portfolio GitHub

Blog

generativeai |amazonbedrock |javascript |node

Amazon Bedrock For JavaScript and TypeScript Developers

depaa

Matteo Depascale

Posted on November 6, 2023

Amazon Bedrock For JavaScript and TypeScript Developers

Introduction

Hey! If you are here, it means you are planning to use Amazon Bedrock, and every example you see uses Python. So, you might have typed on Google something like '"Amazon" "Bedrock" "Typescript"' and somehow you ended up here (not complaining though 😜).

This blog post covers examples in Node.js, both JavaScript and TypeScript, for using Amazon Bedrock SDK V3. You can expect two things from this blog post: examples and my thoughts.

Did I mention there is one example for each model available in Amazon Bedrock? 🚀

Let's jump right through it.

What? Wait a sec! If you have never used Amazon Bedrock, I have an article waiting for you after this one: The Complete Guide to Amazon Bedrock for Generative AI.

Initializing Clients

"Clients"! Yes, you read it right, this time it's not my typo. We need to install two clients:

npm install @aws-sdk/client-bedrock
npm install @aws-sdk/client-bedrock-runtime
Enter fullscreen mode Exit fullscreen mode
  • client-bedrock: SDK for creating and managing Amazon Bedrock models;
  • client-bedrock-runtime: SDK for invoking Amazon Bedrock models and running inference on them.

Now, let's import the libraries. Amazon Bedrock is available in different regions, so we need to select one, or it will use the default one.

import { BedrockRuntimeClient, InvokeModelCommand, InvokeModelCommandInput, InvokeModelCommandOutput, InvokeModelWithResponseStreamCommand, InvokeModelWithResponseStreamCommandInput, InvokeModelWithResponseStreamCommandOutput } from "@aws-sdk/client-bedrock-runtime";

import { BedrockClient, CreateModelCustomizationJobCommand, GetModelCustomizationJobCommand, ListFoundationModelsCommand, CreateModelCustomizationJobCommandInput, CreateModelCustomizationJobCommandOutput, GetModelCustomizationJobCommandInput, GetModelCustomizationJobCommandOutput, ListFoundationModelsCommandInput, ListFoundationModelsCommandOutput } from '@aws-sdk/client-bedrock';

const client = new BedrockRuntimeClient({ region: process.env.REGION || 'us-east-1' });
const client = new BedrockClient({ region: process.env.REGION || 'us-east-1' });
Enter fullscreen mode Exit fullscreen mode

With that done, we're ready to start with the client-bedrock-runtime SDK.

Invoke Model With Response Stream

This API streams the Generative AI model response to us. For this example, I used the model Anthropic Claude Instant V1.

const MODEL_ID = process.env.MODEL_ID || 'anthropic.claude-instant-v1';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';

const params: InvokeModelWithResponseStreamCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      prompt: `\n\nHuman:${PROMPT}\n\nAssistant:`,
      max_tokens_to_sample: 300,
      temperature: 0.5,
      top_k: 250,
      top_p: 1,
    }),
};
const command = new InvokeModelWithResponseStreamCommand(param);
const res = await client.send(command);

const chunks = [];

for await (const event of res.body) {
    if (event.chunk && event.chunk.bytes) {
        const chunk = JSON.parse(Buffer.from(event.chunk.bytes).toString("utf-8"));
        chunks.push(chunk.completion); // change this line
    } else if (
        event.internalServerException ||
        event.modelStreamErrorException ||
        event.throttlingException ||
        event.validationException
    ) {
        console.error(event);
        break;
    }
};
console.log({
    prompt: PROMPT,
    completion: chunks.join(''),
})
Enter fullscreen mode Exit fullscreen mode

Above, we can see how to integrate the 'InvokeModelWithResponseStreamCommand' with TypeScript. One thing worth mentioning is that if we are using a Lambda Function, this integration may not be as useful if we simply take it from this example. It becomes more useful when integrated with Lambda response streaming so we can effectively stream the response from the Generative AI model back to our users. You can find an example here: Lambda Streaming First Byte (TTFB) Pipeline.

Invoke Model

Starting from the basics, we will go through every single model provider available in Amazon Bedrock as of the time I'm writing this blog post (I'll probably maintain this blog post updated if I see you like it 🙂).

Here is the complete structure to invoke the model:

const MODEL_ID = process.env.MODEL_ID || '';
const PROMPT = process.env.PROMPT || '';

const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify(/*Here we place prompt and inference parameters, every model has its own structure 😩*/),
};
const command = new InvokeModelCommand(params);
const res = await client.send(command);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
    prompt: PROMPT,
    completion: /* Here we place the response from "modelRes", every model has its own response 😩*/,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Each model has its own body and response parameters. Starting from this code I'll show the 'params' and how to retrieve the 'response' for each model provider.

I understand that these are all different providers, and generative AI is still too young to have a strong standard, but having to map different parameters and different response objects can be quite confusing. Please, model providers, talk to each other and provide a single, user-friendly way for parameters and model responses 🙏.

⚠️Depending on the region you are in, you may not see every model available.

Invoke Model: Anthropic

Here are the models available for Anthropic:

  • "anthropic.claude-v1"
  • "anthropic.claude-instant-v1"
  • "anthropic.claude-v2"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'anthropic.claude-instant-v1';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      prompt: `\n\nHuman:${PROMPT}\n\nAssistant:`,
      max_tokens_to_sample: 300,
      temperature: 0.5,
      top_k: 250,
      top_p: 1,
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        completion: modelRes.completion,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Invoke Model: AI21 Labs

Here are the models available for AI21 Labs:

  • "ai21.j2-mid-v1"
  • "ai21.j2-ultra-v1"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'ai21.j2-mid';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      prompt: PROMPT,
      maxTokens: 200,
      temperature: 0.7,
      topP: 1,
      stopSequences: [],
      countPenalty: { scale: 0 },
      presencePenalty: { scale: 0 },
      frequencyPenalty: { scale: 0 },
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        completion: modelRes.completions[0].data.text,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Invoke Model: Cohere

Here are the models available for Cohere:

  • "cohere.command-text-v14"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'cohere.command-text-v14';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      prompt: PROMPT,
      max_tokens: 400,
      temperature: 0.75,
      p: 0.01,
      k: 0,
      stop_sequences: [],
      return_likelihoods: "NONE",
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        completion: modelRes.generations[0].text,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Invoke Model: Amazon Text

Here are the models available for Amazon Text:

  • "amazon.titan-text-lite-v1"
  • "amazon.titan-text-express-v1"
  • "amazon.titan-text-agile-v1"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'amazon.titan-text-lite-v1';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      inputText: PROMPT,
      textGenerationConfig: {
        maxTokenCount: 300,
        stopSequences: [],
        temperature: 0,
        topP: 0.9,
      }
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        completion: modelRes.results[0].outputText,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

⚠️Those models are still in preview, but the documentation show a detailed overview on what Amazon' models need and give.

Invoke Model: Amazon Embedding

Here are the models available for Amazon Embedding:

  • "amazon.titan-embed-text-v1"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'amazon.titan-embed-text-v1';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      inputText: PROMPT,
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        embedding: modelRes.embedding,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Invoke Model: Stability AI

Here are the models available for Stability AI:

  • "stability.stable-diffusion-xl-v0"

And here is the code:

const MODEL_ID = process.env.MODEL_ID || 'stability.stable-diffusion-xl-v0';
const PROMPT = process.env.PROMPT || 'Hi, who are you?';
const params: InvokeModelCommandInput = {
    modelId: MODEL_ID,
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      text_prompts: [{ text: PROMPT }],
      cfg_scale: 10,
      seed: 0,
      steps: 50,
    }),
};
const res = await invokeModel(params);
const jsonString = new TextDecoder().decode(res.body);
const modelRes = JSON.parse(jsonString);

const bodyRes = {
        prompt: PROMPT,
        image: modelRes.artifacts[0].base64,
};
console.debug(bodyRes);
Enter fullscreen mode Exit fullscreen mode

Additionally, we can save the image in a file. I'll leave this code to the AI 🤖... I'm joking, it's in my repository 😜.

On a side note, don't you want to know what the Stable Diffusion XL V0 response is to the question: "Hi, who are you?"? Here's the result 👇

Personification of Stability Diffusion XL V0

Perfect, now that we have explored everything there is in the client-bedrock-runtime SDK, it's time to learn how we can use the client-bedrock SDK 🚅.

List Foundation Models

The easiest of them all, it will list foundation models available on Amazon Bedrock. Here's the code:

const params: ListFoundationModelsCommandInput = {
    byInferenceType: "ON_DEMAND", // or "PROVISIONED"
}
const command = new ListFoundationModelsCommand(param);
const res = await client.send(command);
Enter fullscreen mode Exit fullscreen mode

First of all, shouldn't this API return every model without specifying parameters? Why did I use 'byInferenceType'?

Unfortunately, this API has a bug, and without parameters, it throws errors in the 'byProvider' parameter. Also, using the 'byProvider' parameter will throw error status 400 because the regex for the model provider name is not correct. Using 'byInferenceType' is the least impacting parameter if you are starting with Amazon Bedrock 😊.

⚠️Here's the issue: https://github.com/aws/aws-sdk-js/issues/4519

Create Model Customization Job

This command was taken right from the V3 documentation, which is really well done.

Here's the code:

const BUCKET_URI = process.env.BUCKET_URI || 's3://S3_BUCKET_NAME';
const ROLE_ARN = process.env.ROLE_ARN || 'arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME';
const BASE_MODEL_IDENTIFIER = process.env.BASE_MODEL_IDENTIFIER || 'arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-text-express-v1';

const now = new Date();
const params: CreateModelCustomizationJobCommandInput = {
    jobName: `job-${now.getTime()}`, // required
    customModelName: `titan-text-express-v1-${now.getTime()}`, // required
    roleArn: ROLE_ARN, // required
    baseModelIdentifier: BASE_MODEL_IDENTIFIER, // required
    jobTags: [ // TagList
      { // Tag
        key: 'bedrock', // required
        value: 'true', // required
      },
    ],
    customModelTags: [
      {
        key: 'custom-bedrock', // required
        value: 'true', // required
      },
    ],
    trainingDataConfig: {
      s3Uri: `${BUCKET_URI}/training/dataset.jsonl`, // required
    },
    outputDataConfig: {
      s3Uri: `${BUCKET_URI}/output`, // required
    },
    hyperParameters: { // required
      'epochCount': '1',
      'batchSize': '4',
      'learningRate': '0.02',
      'learningRateWarmupSteps': '0',
    },
    // customModelKmsKeyId: 'STRING_VALUE',
    // clientRequestToken: 'STRING_VALUE',
    // validationDataConfig: { // ValidationDataConfig
    //   validators: [ // Validators // required
    //     { // Validator
    //       s3Uri: 'STRING_VALUE', // required
    //     },
    //   ],
    // },
    // vpcConfig: { // VpcConfig
    //   subnetIds: [ // SubnetIds // required
    //     'STRING_VALUE',
    //   ],
    //   securityGroupIds: [ // SecurityGroupIds // required
    //     'STRING_VALUE',
    //   ],
    // },
  };
const command = new CreateModelCustomizationJobCommand(param);
const res = await client.send(command);

console.log(res.jobArn)
Enter fullscreen mode Exit fullscreen mode

Fine tuning is still in preview, but from this documentation https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/bedrock/command/CreateModelCustomizationJobCommand/, we can have a detailed sneak peek 👀 of what's coming.

Get Model Customization Job

Nothing too special about this one, it gets the job information using its identifier.

const JOB_ARN = process.env.JOB_NAME || 'JOB_NAME';
const params: GetModelCustomizationJobCommandInput = {
    jobIdentifier: JOB_ARN,
};
const command = new GetModelCustomizationJobCommand(params);
const res = await client.send(command);
Enter fullscreen mode Exit fullscreen mode

Here we can check the 'status' and also the 'failureMessage', which is really handy to receive by email on Friday at 17.55 😈.

⚠️There are other APIs from 'client-bedrock' that I won't cover because they are really simple or not as useful as these 3.

Conclusion

There you have it, folks! With these code snippets in mind, you can use Amazon Bedrock like a pro 💻. We went through both SDKs and found workarounds for bugs, I think this was a nice ride, and hopefully, you too will be able to enjoy your ride better after this article.

Here you can find the GitHub repository: https://github.com/Depaa/amazon-bedrock-nodejs 😉

If you enjoyed this article, please let me know in the comment section or send me a DM. I'm always happy to chat! ✌️

Thank you so much for reading! 🙏 Keep an eye out for more AWS-related posts, and feel free to connect with me on LinkedIn 👉 https://www.linkedin.com/in/matteo-depascale/.

Reference

Disclaimer: opinions expressed are solely my own and do not represent the views or opinions of my employer.

💖 💪 🙅 🚩
depaa
Matteo Depascale

Posted on November 6, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Amazon Bedrock For JavaScript and TypeScript Developers
generativeai Amazon Bedrock For JavaScript and TypeScript Developers

November 6, 2023