Scraping AWS Docs to Generate PowerShell Help in VaporShell

Recently, I became interested in what code repositories were automatically generated from the AWS CloudFormation Resource Specification files. These act as schema in how CloudFormation templates must be built.

Tools that came to mind as already likely doing so were the ones meant to help with CFN template development by abstracting it away from raw JSON/YAML and into a particular language like SparkleFormation, Troposphere, and probably the AWS Cloud Development Kit.

VaporShell

I have used PowerShell quite a bit throughout my tech career, and back when I first saw the CFN resource specs, I thought it would be cool to autogen a PowerShell Module that had the same goal in mind as those other tools. I also wanted to auto-populate the comment-based help within each function, by pulling directly from the AWS CloudFormation User Guide documentation. Then I shelved it and moved on to other things. This was in 2017.

It came to mind again recently, and I found that VaporShell had been created in the meantime! I checked to see how the help documentation looked, and realized I could revisit my old code and contribute to the project.

Example Improvements

Originally, the built-in comment-based help for VaporShell resource functions were barebones. I felt that the AWS CloudFormation Resource Guide source could be used to populate parameter and function documentation.

Parameter Information (VaporShell Original)

PS> Get-Help New-VSAlexaAskSkill -Parameter VendorId

-VendorId <Object>
    Required: True    
    Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ask-skill.html#cfn-ask-skill-vendorid   
    PrimitiveType: String    
    UpdateType: Immutable

    Required?                    true
    Position?                    named
    Default value                
    Accept pipeline input?       false
    Accept wildcard characters?  false

Parameter Information (Expanded)

Includes parsed data from Alexa::ASK::Skill

PS> Get-Help New-VSAlexaAskSkill -Parameter VendorId

-VendorId <Object>
    The vendor ID associated with the Amazon developer account that 
    will host the skill. Details for retrieving the vendor ID are in 
    How to get your vendor ID: https://github.com/alexa/alexa-smartho
    me/wiki/How-to-get-your-vendor-ID. The provided LWA credentials 
    must be linked to the developer account associated with this 
    vendor ID.

    Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ask-skill.html#cfn-ask-skill-vendorid   
    PrimitiveType: String    
    UpdateType: Immutable

    Required?                    true
    Position?                    named
    Default value                
    Accept pipeline input?       false
    Accept wildcard characters?  false

NOTE: Required property can be dropped, because PowerShell help output will automatically check whether [parameter(Mandatory = $true)] which results in Required? output being true or false.

Description Information (VaporShell Original)

PS> Get-Help New-VSAlexaAskSkill

NAME
    New-VSAlexaAskSkill

SYNOPSIS
    Adds an Alexa::ASK::Skill resource to the template
...

DESCRIPTION
    Adds an Alexa::ASK::Skill resource to the template
...

Description Information (Expanded)

Includes parsed data from Alexa::ASK::Skill

PS> Get-Help New-VSAlexaAskSkill

NAME
    New-VSAlexaAskSkill

SYNOPSIS
    Adds an Alexa::ASK::Skill resource to the template
...

DESCRIPTION
    The Alexa::ASK::Skill resource creates an Alexa skill that 
    enables customers to access new abilities. For more information 
    about developing a skill, see the Build Skills with the Alexa 
    Skills Kit developer documentation: https://developer.amazon.com/
    docs/ask-overviews/build-skills-with-the-alexa-skills-kit.html.

How?

The best path seemed to be:

Pull down the latest version of the AWS CFN User Guide
Pull down the latest us-east-1 CFN spec file
Roll through the Documentation properties of everything within the AWS CFN spec files that are used to build PowerShell functions, and find the equivelant MarkDown doc in the AWS CFN User Guide source repo
Scrape resource/property descriptions, along with any parameter descriptions, to include in PowerShell function comment-based help

Proof-of-Concept Output

Using the POC code I made here, AWSCFNDocHelpers.psm1, we can get exactly what we are looking for.

Prerequisites

Tested On: PowerShell Core 6.2.2
OS: Ubuntu 18.04
Requires the following done in the $PWD code will execute in

# AWS CFN User Guide help documentation
git clone https://github.com/awsdocs/aws-cloudformation-user-guide.git

# AWS CFN spec file
Invoke-WebRequest 'https://d1uauaxba7bl26.cloudfront.net/latest/gzip/CloudFormationResourceSpecification.json' -OutFile 'CloudFormationResourceSpecification.json'

Example Code Results

PS> Import-Module ./AWSCFNDocHelpers.psm1
PS> $HelpDocs = New-CFNHelpDoc
PS> ($HelpDocs | Get-Member -Type NoteProperty).Name
Description
FunctionName
Links
Parameters
Synopsis

PS> $HelpDocs[0].FunctionName
New-VSAlexaASKSkill

PS> $HelpDocs[0].Description 
The Alexa::ASK::Skill resource creates an Alexa skill that enables customers to access new abilities. For more information about developing a skill, see the Build Skills with the Alexa Skills Kit developer documentation: https://developer.amazon.com/docs/ask-overviews/build-skills-with-the-alexa-skills-kit.html.

PS> $HelpDocs[0].Parameters[0].ParameterName
AuthenticationConfiguration

PS> $HelpDocs[0].Parameters[0].ParameterDescription
Login with Amazon LWA configuration used to authenticate with the Alexa service. Only Login with Amazon clients created through the Amazon Developer Console: https://developer.amazon.com/lwa/sp/overview.html are supported. The client ID, client secret, and refresh token are required.

VaporShell now automatically includes these changes, dynamically populating content scraped from the AWS CFN docs! Nate Ferrell owns the repository, and modified my demo code to be part of the CI pipeline:

Nate Ferrel

NOTE: To see the back-and-forth issue communication in the VaporShell repo, along with supplied demo code, checkout the issue here: https://github.com/scrthq/VaporShell/issues/61

Working on this made me realize there was some funny business going on with the AWS CFN documentation, and with the CFN resource specification files. I'll talk about these problems, and how I used GitHub Actions with Python to do some automated auditing, in another post.

In the meantime, checkout the GitHub Actions beta!

Originally published at https://icanteven.io on October 2nd, 2019

Blog