TheLazy.Dev
Portfolio GitHub

Blog

dotnet |docker |csharp |iac

API's From Dev to Production - Part 11 - Pulumi - IAC

peteking

Pete King

Posted on June 7, 2021

API's From Dev to Production - Part 11 - Pulumi - IAC

Series Introduction

Welcome to Part 11 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

  • Infrastructure as Code
    • Pulumi

TL;DR

We got to grips with IaC and made use of our C# skills to build our own infrastructure - Go shift-left! Pulumi has been a joy to use, just so easy, the integration of GitHub Actions and Pulumi has equally been awesome.

We were able to preview our infrastructure changes from a Pull Request using our CI Workflow and achieved C-Deployment by deploying our infrastructure upon merging into main using our CD Workflow.

The Pulumi GitHub Bot works great and the Pulumi Console is simply invaluable. 🤓

GitHub Repository

GitHub logo peteking / Samples.WeatherForecast-Part-11

This repository is part of the blog post series, API's from Dev to Production - Part 11 on dev.to. Based on the standard .net standard Weather API sample.

Requirements

We will be picking-up where we left off in Part 10, which means you’ll need the end-result from GitHub Repo - Part 10 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Don't forget to ensure you have setup Code Climate Quality with your repository.

If you want to follow along from the code from Part 10, that is really ideal, I have everything in order and it should all work fine. If however, you want to skip to the end and run it all, you can too, but there there are some things you'll need to do.

  1. Clone the GitHub Repo (Part-11) into your own GitHub
  2. Install/Obtain Access to Azure CLI
  3. Install Pulumi CLI
  4. Navigate → ./src/Samples.Weatherforecast.Infra
  5. Follow the instructions in this post about: Create a Service Principal
  6. Follow the instructions in this post about: Securely store Azure Service Principal credentials
  7. Execute → pulumi preview

This should create the Pulumi project and its stack.

Introduction

This post is all about Infrastructure as Code, we want to be more autonomous to build and manage our own infrastructure instead of being fully dependant on others. Infrastructure as Code has been around a while and there are many solutions, each cloud provider such as Microsoft Azure, AWS, GCP and more all have their own native solutions, and of course the number one name for some time has been Hashicorp's Terraform (OSS, Cloud, Enterprise). However, they all have something in common... they are all in their own languages, from declarative to DSL's and JSON. Using DSL's can be cumbersome as is learning a new language; however, a purpose built language could have some advantages. Pulumi is one of those solutions that takes a different angle, the ability to get to your end goal of having your infrastructure written as code but in a language you use all the time.

In this post we will dive head-first into, Pulumi!

Let's get started. 🤘

Microsoft Azure

Get access to Azure, this is kind of mandatory, we will be using Microsoft Azure - If however, you don't have an account, you can get one for FREE! 😁

Sign-up to Microsoft Azure.

Install/Obtain access to Azure CLI

Azure CLI on your OS

OR

Azure CLI in Docker

OR

Azure Cloud Shell in BASH

OR

If you already have chocolately installed, you can simply execute:

choco install azure-cli
Enter fullscreen mode Exit fullscreen mode

As long as you can get access, that's all we need, so use whichever method you prefer.

Pulumi

What is Pulumi?

Pulumi is a modern infrastructure as code platform built for engineers; developers and infrastructure.

The beauty of the product is you can build, deploy and manage your applications and infrastructure using languages and tools you are familiar with. Follow your existing engineering practices and guidelines, and take advantage of the same toolchain you use today.

This means if you major in C# for instance, you have all the power of the C# language and all your existing tools. It means you don't need to learn a bespoke language just to manage and provision your infrastructure = Learning curve is minimised.

When you deploy applications, everyone can work together, or you can choose to be isolated; maybe I'm trying something out or I have something very unique. You can work together and share infrastructure packages no matter what language it is written in, deploy with confidence by validating beforehand and automate it all!

If that wasn't enough, with Pulumi you can use policies to provide guardrails for teams, allowing them to feel empowered with autonomy (shift-left), and with the Pulumi Console you can see a full audit history. This is known as Policy as Code.

Pulumi is a breath of fresh air compared to other solutions, now I know what you're thinking, Terraform... Yes, I love it too, and I've worn that T-shirt, it's a good one, everyone has their go-to favourite T. However, I have a preference for Pulumi, its simplicity by using the same language I'm building API's in, whether that is C#, Go, JS/TS, Python etc. The Terraform T-shirt is now washed, ironed, and in the cupboard for a while. Luckily Pulumi can connect to Terraform also, so even if you have a centralised Platform team building foundational infrastructure, initial virtual networking topology, firewalls, SIEM etc., engineering teams can shift-left to build and manage their own components whilst using the outputs of Terraform. If for example I needed the output of a VNET from Terraform, Pulumi can get that and then I can provision my product using Pulumi as a pretty-well autonomous engineering team.

Sign-up

Pulumi has a free tier and they say it's free forever (for individual use) 😁

Sign-up to Pulumi.

Pulumi CLI

Pulumi is controlled primarily using its command line interface (CLI). It works with the Pulumi service to deploy changes to cloud apps and infrastructure. It also keeps a history of who updated what in your team and when; which is awesome! The CLI has been designed for productivity and ultrafast feedback, in addition to the ability to use it in CI/CD scenarios.

Install Pulumi CLI.

(Again) If you already have chocolately installed, you can simply execute:

choco install pulumi
Enter fullscreen mode Exit fullscreen mode

Requirements check

✅ Everything we've done previously, i.e. Part 10 etc.
✅ Azure account
✅ Azure CLI - Version used v2.24.0
✅ Pulumi account (Console)
✅ Pulumi CLI - Version used v2.14.0

Let's get started with Pulumi!

Fire-up your console of choice and navigate to the src directory of the WeatherAPI.

Create a folder called, Samples.WeatherForecast.Infra and navigate to it

mkdir Samples.WeatherForecast.Infra
cd Samples.WeatherForecast.Infra
Enter fullscreen mode Exit fullscreen mode

Inside the infra directory - Create a new Pulumi Project.

pulumi new azure-csharp
Enter fullscreen mode Exit fullscreen mode

From here, you'll be prompted by the Pulumi CLI for some values.

  • project name: Samples.WeatherForecast.Infra
  • project description: Infra for WeatherForecast
  • stack name: dev
  • azure-native:location: The Azure location to use: uksouth

I've chosen UK South given my location, for you, please choose whatever suits your needs best.

ℹ️ TIP
If you don't know what locations are available, below is a very quick & easy way to find out.

az login

# Once login successful, execute
az account list-locations --output table
Enter fullscreen mode Exit fullscreen mode

Open VS Code by executing code .

You should see a nice, brand new Pulumi C# program! 😄

If you also login to the Pulumi Console, you should see your project there too!

blog-11-01

blog-11-02

Our API Infrastructure

blog-post-11-arch

Steps

Step 1

Open VS Code for the WeatherForecast-API if you haven't already, and Navigate → /src/WeatherForecastApi-Infra/

Step 2

Rename MyStack.cs to AzureStack.cs

Step 3

Inside AzureStack.cs, replace all code with the following

using Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Web;
using Pulumi.AzureNative.Web.Inputs;

class AzureStack : Stack
{
    public AzureStack()
    {
        var config = new Pulumi.Config();

        // Obtain our docker image from config
        var dockerImage = config.Require("docker-image");

        // Resource Group
        var rg = new ResourceGroup("rg-weatherforecastapi-uks-");

        // AppService Plan
        var appServicePlan = new AppServicePlan("appplan-weatherforecastapi-uks-", new AppServicePlanArgs() {
            ResourceGroupName = rg.Name,
            Location = rg.Location,
            Kind = "Linux",
            Reserved = true,
            Sku = new SkuDescriptionArgs
            {
                Name = "B1",
                Tier = "BASIC"
            },
        });

        // WebApp for Containers
        var app = new WebApp("app-weatherforecastapi-uks-", new WebAppArgs() 
        { 
            ResourceGroupName = rg.Name,
            ServerFarmId = appServicePlan.Id,
            SiteConfig = new SiteConfigArgs
            {
                AppSettings = new[] 
                { 
                    new NameValuePairArgs
                    {
                        Name = "WEBSITES_ENABLE_APP_SERVICE_STORAGE",
                        Value = "false"
                    },
                    new NameValuePairArgs
                    {
                        Name = "DOCKER_REGISTRY_SERVER_URL",
                        Value = "https://ghcr.io"
                    },
                    new NameValuePairArgs
                    {
                        Name = "WEBSITES_PORT",
                        Value = "8080" // Our custom image exposes port 8080. Adjust for your app as needed.
                    }
                },
                AlwaysOn = true,
                LinuxFxVersion = $"DOCKER|{dockerImage}"
            },
            HttpsOnly = true
        });

        this.Endpoint = Output.Format($"https://{app.DefaultHostName}/weatherforecast");
        this.HealthEndpoint = Output.Format($"https://{app.DefaultHostName}/health"); 
    }

    [Output] public Output<string> Endpoint { get; set; }

    [Output] public Output<string> HealthEndpoint { get; set; }
}
Enter fullscreen mode Exit fullscreen mode

I appreciate there is a chunk of code there, so let's break it down just a little.

The first couple of lines is simply getting the config from Pulumi. I have chosen the Require method because I would rather it fails miserably if the config setting is not present; without it, we shouldn't continue because we'll have no container image. 😆

Up next is the resource group creation, note that we do not actually specify its location - The location comes from the config file - Pulumi.dev.yaml, it's value is from, azure-native:location. 😉

There are a bunch of other Azure native settings, for more information on these, please see, Azure Native Configuration

Following-on from there is the creation of the Azure AppService Plan, this is essentially our underlying Virtual Machine (VM), I have picked the smallest available WebApp for Containers (Linux) I can; so if you are paying for this I recommend when you test, to destroy afterwards - Still, if you didn't, it would be a under GBP£10 per month anyway.

Then we have the WebApp itself, this is the AppService, there are a few things we need to set here, mainly stating where the Docker registry is, in our case, GitHub Container Registry - Please note that the default is always DockerHub, you can easily point it to ACR (Azure Container Registry) also. We set AlwaysOn and HttpsOnly to true and we of course need to set the Docker Image, this is the, LinuxFxVersion property.

We need a way to output values, we want to output the address of the host as the end of the resource names will be dynamically generated. We set one for the weatherforecast and just to be helpful we set one for the health endpoint too.

Finally, the last 2 lines of the file are the declarations of the output properties, what denotes these as outputs is simply the 'Output' attribute.

Step 4

Modify the Pulumi.dev.yaml file.
Replace its contents with the below:

config:
  azure-native:location: uksouth
  Samples.WeatherForecast.Infra:docker-image: ghcr.io/peteking/samples-weatherforecast-part-11:3dae90cc47eff6861687444a6e91be89a65bed9f
Enter fullscreen mode Exit fullscreen mode

We've added our own config here which is our docker image, therefore, it's configuration instead of C# code.

This should be the location of your Docker image.

If you're using the GitHub Container Registry, it should be a simple swap.

Step 5

We have enough to give it a good manual "interactive" test with the Pulumi CLI, just to make sure everything is working as we expect it to.

Open your terminal and Navigate → /src/WeatherForecastApi-Infra/

Execute:

pulumi up
Enter fullscreen mode Exit fullscreen mode

Here you can see it has executed preview as part of the up command, it highlights what it plans to do, in this case, it will create 4 resources.

blog-11-04

Select → yes

Once you've selected yes, Pulumi will go-ahead and create those resources, below is the response.

blog-11-05

You can see the Pulumi outputs we created, one that links to the weatherforecast endpoint and one to the health endpoint; we've done this because the resources are dynamic, i.e. at the end of our Azure Resources is a random string to ensure uniqueness. If this doesn't suit your use case, you can ensure consistency instead - For more information, see Pulumi - Autonaming

We now have the opportunity to test our cloud infrastructure as code deployment and see if our API works!

You can use the links from the output to save typing.

The weather forecast gets returned! 🎉

blog-11-08

Even the /health endpoint is reporting good news! 🎆

blog-11-09

If you want to minimise your costs 💲, you can of course destroy 💥 your resources, and with the powers of Pulumi that is also a breeze with the, destory command.

Execute:

pulumi destory
Enter fullscreen mode Exit fullscreen mode

blog-11-06

Select → yes

Once you've selected yes, Pulumi will go-ahead and destroy those resources, below is the output. 👍

blog-11-07

ℹ️
You'll notice the output at the bottom stating the stack is still available - All this means is you have all the details, history and everything about your stack in the Pulumi Console; which is all rather helpful, don't you think?

GitHub Actions with Pulumi

All of what we have achieved is great, we have got everything we need infrastructure wise, built it all as code, we've even deployed and tested it for real. However, it's all rather interactive, we need to bake this into both our CI Workflow and our CD Workflow.

Our CI Workflow will execute pulumi preview and utilise the Pulumi GitHub Bot for instant feedback in the Pull Request; by doing this, the person reviewing the PR will be able to see what resources will be created/destroyed/updated - If anything is a miss then you have the opportunity to do something about it.

Our CD Workflow will execute pulumi up and deploy our resources to Azure; but only once all our tests pass, code coverage pass and our container scan is all good too.

In order to interact with Azure in GitHub Actions we will create a Service Principal.

Azure Service Principal

What is it?

A Service Principal is an object in a single tenant or directory in Azure AD. It is the identity of an application instance, it defines access for the application and what resources it can access. Instead of using a user identity, we can create a service identity and manage its security and access.

Create Azure Service Principal

Step 1

Create an, Azure Subscription (if you don't already have one).

Step 2

Obtain the subscription id:

az account list #It's the 'id' field.
Enter fullscreen mode Exit fullscreen mode

Step 3

Set the subscription:

az account set --subscription=<id> #<id> is the 'id' field from above
Enter fullscreen mode Exit fullscreen mode

Step 4 

az ad sp create-for-rbac --name sp-weather-api
Enter fullscreen mode Exit fullscreen mode

You should see a result from Step 4 with the following values

{
  "appId": "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW",
  "displayName": "ServicePrincipalName",
  "name": "http://ServicePrincipalName",
  "password": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "tenant": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"
}
Enter fullscreen mode Exit fullscreen mode

ℹ️ TIP
If you'd rather it in another format to save time later, you can use the --sdk-auth parameter.

Securely store Azure Service Principal credentials

Now we have the Service Principal, we need to be able to store the details securely.

We can store these in GitHub Secrets which is totally fine right, but that means they are not linked to our Pulumi Stack, so another way is to store them inside Pulumi; which can also store secrets

From the output of our Service Principal creation command, the following values can be mapped like so:

Service Principal Value SDK Name
appId clientId
password clientSecret
tenant tenantId

This is where the --sdk-auth parameter can help, it will automatically map these for you.

Pulumi Secrets ㊙️

By storing these in Pulumi we get multi-user access for deployment which includes using it interactively; if we stored them in GitHub Secrets, we can only use them directly from the pipeline itself; there are pros and cons this of course - Choose what suites you best.

pulumi config set azure-native:clientId "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW"
Enter fullscreen mode Exit fullscreen mode 
pulumi config set azure-native:clientSecret "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" --secret
Enter fullscreen mode Exit fullscreen mode

❗ Important
Note the --secret at the end of the command, this is very important; otherwise, the value will be stored in plane text, whereas we want it to be encrypted and stored as a secret.

pulumi config set azure-native:tenantId "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"
Enter fullscreen mode Exit fullscreen mode 
pulumi config set azure-native:subscriptionId "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ"
Enter fullscreen mode Exit fullscreen mode

Once you have executed these commands, you'll notice your Pulumi.dev.yml file has been updated accordingly.

CI Workflow

Let's modify our CI Workflow ./github/workflows/ci-pull-request.yml

Add a new job at the bottom of the Workflow.

infra:
  needs: ci
  runs-on: ubuntu-latest

  steps:
    - name: Checkout repo
      uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0

    - name: Install Pulumi CLI
      uses: pulumi/action-install-pulumi-cli@v1.1.0

    - name: Set Pulumi config values
      working-directory: ./src/Samples.WeatherForecast.Infra
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
      run: |
        pulumi stack select dev
        pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}

    - name: Pulumi Preview
      uses: pulumi/actions@v3.1.0
      with:
        command: preview
        stack-name: dev
        work-dir: ./src/Samples.WeatherForecast.Infra
        github-token: ${{ secrets.GITHUB_TOKEN }}
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

Where do I get the Pulumi Personal Access Token from?

  1. Navigate → Pulumi Console - https://app.pulumi.com
  2. Navigate → Your profile (top-right)
  3. Click → Access Tokens - Quick link here: https://app.pulumi.com/YOUR_PULUMI_USER_NAME/settings/tokens
  4. Click → Create token
  5. Copy the secret generated
  6. Safely store the secret in GitHub Actions Secrets with the name of PULUMI_ACCESS_TOKEN

Install the Pulumi GitHub Bot

Unless you want to dive into the Actions log to find out how the pulumi preview went, you better install the Pulumi GitHub Bot

Step by step guide on Pulumi Docs

Once you have the bot installed, when you initiate a PR, you'll see something like the below:

blog-11-10

Even better though, is you can view more details in the Pulumi Console like so:

blog-11-11

blog-11-12

CD Workflow

Let's modify our CD Workflow ./github/workflows/build-and-publish.yml

Add a new job at the bottom of the Workflow.

infra:
  needs: build-and-push
  runs-on: ubuntu-latest

  steps:
    - name: Checkout repo
      uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0

    - name: Install Pulumi CLI
      uses: pulumi/action-install-pulumi-cli@v1.1.0

    - name: Set Pulumi config values
      working-directory: ./src/Samples.WeatherForecast.Infra
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
      run: |
        pulumi stack select dev
        pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}

    - name: Pulumi Up
      uses: pulumi/actions@v3.1.0
      with:
        command: up
        stack-name: dev
        work-dir: ./src/Samples.WeatherForecast.Infra
        github-token: ${{ secrets.GITHUB_TOKEN }}
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

Commit your code

It's time to commit you code, but before you do, let's update our branch protection rule - Now we have an infra job in our Workflow which does a pulumi preview, we most likely want this to be mandatory and pass.

blog-11-13

Now that we have our branch protection rules updated and CD Workflow file completed, let's ensure it all works.

  1. Raise your PR and see the CI Workflow kick-off.
  2. Verify everything is fine by looking at the Pulumi GitHub Bot or the Pulumi Console.
  3. Wait until all GitHub Status Checks pass and merge your PR into main.
  4. Verify your Workflow is successful.
  5. Either:
    • Go to the Pulumi Console and see your Activity.
    • OR see the output from the infra job in the CD Workflow.
  6. Try the output links to verify your API is running.

I prefer the Pulumi Console, see below, you can see the merge of my branch and the creation of the infrastructure once it was merged into main.

blog-11-14

I was then able to validate that both the weatherforecast and health endpoints was functional.

What have we learned? 🤔

We have learned all about IaC - Infrastructure as Code, and utilised Pulumi to achieve this - We have leveraged our existing C# skills to shift-left on our infrastructure; therefore, we are now more autonomous.

We've been able to test (preview) and verify our infrastructure changes during Pull Requests as part of CI, and we have achieved C-Deployment by releasing our infrastructure in our CD Workflow.

Relief - That's it! 😌

If you've made it this far, congratulations and thank you 🤝, I know it's a long post and a lot to take in, but I hope you have found it worthwhile.

There is unfortunately or fortunately, depending on how you look at it, still a lot to do 😏 So there will be a more coming soon! 🤪

UPDATE AUGUST 27TH 2021
The below GitHub Issue has now been resolved.

📄 Note
Adding Pulumi to your repo in GitHub will break CodeQL, there is an active GitHub issue open for this.

Process failed with exit code 100 #544

peteking avatar
peteking posted on

Hi,

I have failure in the codeql-action, it has failed with an exit code of 100, looking at the docs it says it's likely a bug based on this error code.

This is an open source sample project, so all open.

The recent addition made as you can see from the PR is I've added Pulumi for its Azure infrastructure provisioning.

I think it start to look at Pulumi here:

  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space

and then slowly goes wrong and fails, out of memory.

Here is the codeql workflow run here: https://github.com/peteking/Samples.WeatherForecast-Part-11/pull/1/checks?check_run_id=2738620747

Part of the log is here for quick reference:

Finalizing csharp
  /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql database finalize --threads=2 /home/runner/work/_temp/codeql_databases/csharp
  Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11.
  Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11.
  [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list]
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list]
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space
  com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521)
  com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578)
  com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754)
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space
  com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495)
  com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521)
  com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439)
  com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578)
  com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376)
  com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754)
  com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172)
  com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495)
  com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129)
  com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439)
  com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source)
  com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
  com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source)
  java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base/java.lang.Thread.run(Unknown Source)
  com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376)
  com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172)
  com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129)
  com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source)
  com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
  com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source)
  java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base/java.lang.Thread.run(Unknown Source)
  Oops! A fatal internal error occurred.
  Oops! A fatal internal error occurred.
  java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
  java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
    at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
    at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105)
    at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98)
    at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
    at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448)
    at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
    at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105)
    at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98)
    at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
    at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209)
    at com.semmle.cli2.CodeQL.main(CodeQL.java:96)
  Caused by: java.lang.OutOfMemoryError: Java heap space
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209)
    at com.semmle.cli2.CodeQL.main(CodeQL.java:96)
  Caused by: java.lang.OutOfMemoryError: Java heap space
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100
      at Object.toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v1/lib/toolrunner-error-catcher.js:79:19)
      at processTicksAndRejections (internal/process/task_queues.js:93:5)
      at async Object.finalizeDatabase (/home/runner/work/_actions/github/codeql-action/v1/lib/codeql.js:414:13)
      at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:74:9)
      at async Object.runAnalyze (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:172:5)
      at async run (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:50:30)
      at async runWrapper (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:96:9)
View on GitHub

Next up

Part 12 in this series will be about:

  • More Infrastructure as Code with Pulumi
    • Environmental support (multiple Stacks)
    • Security
      • What can we do on the cheap in Azure?
      • What should we do in a more real scenario?
  • GitHub Actions
    • Clean-up a little and add Environment capabilities

More information

💖 💪 🙅 🚩
peteking
Pete King

Posted on June 7, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

API's From Dev to Production - Part 11 - Pulumi - IAC
dotnet API's From Dev to Production - Part 11 - Pulumi - IAC

June 7, 2021