TheLazy.Dev
Portfolio GitHub

Blog

powershell |python |opensource |windows

PyPyInstaller Powershell Module Initial Code Push

cwprogram

Chris White

Posted on September 25, 2023

PyPyInstaller Powershell Module Initial Code Push

I've already written about this in a few places but I've finally done an initial release of PyPyInstaller. This is one of the projects I mentioned in my last distraction post.

Motivation

One of the articles I wanted to write was for managing multiple installations of Python. Unfortunately, I noticed that PyPy seemed to be left out in terms of Windows support. There wasn't a solid option to both add it to PATH and add the appropriate PEP 514 entries to be listed in the Python Launcher. I decided this would be an interesting project to work on as a hobby type deal.

Why Powershell

There are many Windows solutions that run off a command line or GUI executable. The problem is you can't really see what's going on. Even if the source code is available you don't really know it's the code in that executable unless you build it yourself. Making this a Powershell module means everything is a script and the user can easily see every command that's being run.

Another reason is that using commandlets allows for flexibility in how installations are done. For example, there's a command for finding the latest installation which can be piped into the command to install a PyPy version:

> Find-PyPyLatest -PythonSeries "3.9", "3.10" | Install-PyPy
Enter fullscreen mode Exit fullscreen mode

But what if you really need that specific version? You can just run Install-PyPy by itself with the appropriate option:

> Install-PyPy -PythonVersions "3.8.16"
Enter fullscreen mode Exit fullscreen mode

I think it helps strike a balance between just get it done and still allow some more controlled uses.

Code Layout

The layout of the code looks like this:

├─.github
│  └─workflows
├─.vscode
├─src
│  └─PyPyInstaller
│      ├─Functions
│      │  ├─Private
│      │  └─Public
│      └─Wrappers
└─test
    └─fixtures
Enter fullscreen mode Exit fullscreen mode

src contains the source of the module itself. Under this is a Functions directory containing the actual ps1 files. This directory is also where code coverage looks to ensure tests are built against everything. Functions are then broken up into Public and Private. Private functions are meant to support the Public versions. It's then the Public versions that are exposed to the user. test contain the Pester tests used to ensure the code works.

Test Test Test...

The is one of the first personal projects where I really went all out on testing. Part of it was because PyPyInstaller touches PATH and registry keys. At the moment the code coverage is at 99% (I still have a big question mark on how to deal with the last %):

Tests completed in 3.35s
Tests Passed: 37, Failed: 0, Skipped: 0 NotRun: 0
Processing code coverage result.
Covered 99.46% / 90%. 185 analyzed Commands in 8 Files.
Enter fullscreen mode Exit fullscreen mode

Pester is what's used for most Powershell testing. While it has some pretty amazing features, it does have issues with .NET class method mocking and sometimes scope can be weird. Here's a simple example:

BeforeAll {
    $PackageRoot = "$PSScriptRoot\..\src\PyPyInstaller\"
    Remove-Module PyPyInstaller
    Import-Module $PackageRoot
    . "$PackageRoot\Functions\Private\Utility.ps1"
}
InModuleScope PyPyInstaller {
    Describe "Update-PyPyMirror" {
        BeforeAll {
            $TestRootPath = "$env:temp"
            Mock -CommandName Read-PyPyInstallerConfig -MockWith { return @{ RootPath = $TestRootPath } }
            Mock -CommandName Out-File -ParameterFilter { $FilePath -eq "$TestRootPath\versions.json" } -MockWith { return $null }
            Mock -CommandName Invoke-WebRequest -ParameterFilter { $Uri -eq "https://buildbot.pypy.org/mirror/versions.json" } -MockWith { return @{ Content = Get-Content "$PSScriptRoot\fixtures\test_versions.json" } }
        }
        It "Passes Downloads Mirror JSON" {
            Update-PyPyMirror
            Assert-MockCalled Invoke-WebRequest -Exactly -Times 1 -ParameterFilter { $Uri -eq "https://buildbot.pypy.org/mirror/versions.json" } -Scope It
            Assert-MockCalled Out-File -ParameterFilter { $FilePath -eq "$TestRootPath\versions.json" }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

First BeforeAll imports the code that will be tested. Then InModuleScope is used to deal with the scope oddities mentioned. Describe has the function being mentioned, with an additional BeforeAll declaration. This is where the mocking happens in the this particular test case. As an example for one of the mocks:

Mock -CommandName Invoke-WebRequest -ParameterFilter { $Uri -eq "https://buildbot.pypy.org/mirror/versions.json" } -MockWith { return @{ Content = Get-Content "$PSScriptRoot\fixtures\test_versions.json" } }
Enter fullscreen mode Exit fullscreen mode

This will mock the Invoke-Request command when the -Uri argument is the PyPy mirror versions.json file. Instead of returning that it will return a test version of the file with a specific layout for my tests. When the code is run, it asserts that my mocked version of Invoke-WebRequest was called instead of the actual command:

Assert-MockCalled Invoke-WebRequest -Exactly -Times 1
Enter fullscreen mode Exit fullscreen mode

This makes sure I'm not on the bad side of PyPy mirror infra by hitting their versions file every time I run the test cases! It can also test the registry:

    Context "Registry Keys Don't Exist" {
        BeforeAll {
            New-Item -Path "TestRegistry:\Software\Python" -Force
            Mock -CommandName Test-Path -MockWith { return $false }
        }
        It "Passes Adds Entries To Registry" {
            Set-PyPyLauncherEntry -PythonVersion "3.10.12" -PyPyVersion "7.3.12" -InstallPath "C:\PyPy" -RegistryDrive "TestRegistry"

            $CoreInfo = Get-ItemProperty "TestRegistry:\Software\Python\PyPyInstaller\3.10"
            $CoreInfo.DisplayName | Should -Be "PyPy 3.10.12"
            $CoreInfo.SupportUrl | Should -Be "https://github.com/cwgem/pypy-powershell-install"
            $CoreInfo.Version | Should -Be "7.3.12"
            $CoreInfo.SysVersion | Should -Be "3.10.12"
            $CoreInfo.SysArchitecture | Should -Be "64bit"

            $InstallPathInfo = Get-ItemProperty "TestRegistry:\Software\Python\PyPyInstaller\3.10\InstallPath"
            $InstallPathInfo.'(default)' | Should -Be "C:\PyPy"
            $InstallPathInfo.ExecutablePath | Should -Be "C:\PyPy\pypy.exe"
            $InstallPathInfo.WindowedExecutablePath | Should -Be "C:\PyPy\pypyw.exe"
        }
    }
Enter fullscreen mode Exit fullscreen mode

It does so by creating a TestRegistry: entry that can be used in place of the actual registry. This is something I then pass in to the command that deals with the registry as a path, and then assert against the various registry keys.

Interesting Powershell

Of particular interest was how Powershell was easily able to handle pipeline input:

function Find-PyPyLatest {
    [CmdletBinding()]
    param (
        [Parameter(ValueFromPipeline)]
        [string[]]
        $PythonSeries
    )
Enter fullscreen mode Exit fullscreen mode

Here's an example which accepts the value of PythonSeries as a pipeline argument (I can also just pass it in as the -PythonSeries argument):

> "3.9", "3.10" | Find-PyPyLatest

pypy_version   : 7.3.12
python_version : 3.9.17
stable         : True
latest_pypy    : True
date           : 2023-06-16
files          : {@{filename=pypy3.9-v7.3.12-aarch64.tar.bz2; arch=aarch64; platform=linux; download_url=https://downlo
                 ads.python.org/pypy/pypy3.9-v7.3.12-aarch64.tar.bz2}, @{filename=pypy3.9-v7.3.12-linux32.tar.bz2; arch
                 =i686; platform=linux; download_url=https://downloads.python.org/pypy/pypy3.9-v7.3.12-linux32.tar.bz2}
                 , @{filename=pypy3.9-v7.3.12-linux64.tar.bz2; arch=x64; platform=linux; download_url=https://downloads
                 .python.org/pypy/pypy3.9-v7.3.12-linux64.tar.bz2}, @{filename=pypy3.9-v7.3.12-macos_x86_64.tar.bz2; ar
                 ch=x64; platform=darwin; download_url=https://downloads.python.org/pypy/pypy3.9-v7.3.12-macos_x86_64.t
                 ar.bz2}…}

pypy_version   : 7.3.12
python_version : 3.10.12
stable         : True
latest_pypy    : True
date           : 2023-06-16
files          : {@{filename=pypy3.10-v7.3.12-aarch64.tar.bz2; arch=aarch64; platform=linux; download_url=https://downl
                 oads.python.org/pypy/pypy3.10-v7.3.12-aarch64.tar.bz2}, @{filename=pypy3.10-v7.3.12-linux32.tar.bz2; a
                 rch=i686; platform=linux; download_url=https://downloads.python.org/pypy/pypy3.10-v7.3.12-linux32.tar.
                 bz2}, @{filename=pypy3.10-v7.3.12-linux64.tar.bz2; arch=x64; platform=linux; download_url=https://down
                 loads.python.org/pypy/pypy3.10-v7.3.12-linux64.tar.bz2}, @{filename=pypy3.10-v7.3.12-macos_x86_64.tar.
                 bz2; arch=x64; platform=darwin; download_url=https://downloads.python.org/pypy/pypy3.10-v7.3.12-macos_
                 x86_64.tar.bz2}…}
Enter fullscreen mode Exit fullscreen mode

Objects can also be accepted via pipeline:

function Install-PyPy {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, ValueFromPipelineByPropertyName = $true)]
        [string[]]
        [Alias('python_version')]
        $PythonVersions,
Enter fullscreen mode Exit fullscreen mode

This will bind the python_version property of any passed in objects to the PythonVersions argument. It's is quite a blessing from someone who has had to deal with the *NIX version of pipe handling... Another oddity I found was adding to PATH:

function Get-PyPyPathEnvironmentVariable {
    return [System.Environment]::GetEnvironmentVariable('Path', 'User')
}

function Set-PyPyPathEnvironmentVariable {
    param (
        [Parameter()]
        [String]
        $NewPath
    )
    [System.Environment]::SetEnvironmentVariable('Path', $NewPath, 'User')
}
Enter fullscreen mode Exit fullscreen mode

These are wrappers around .NET class methods to make Pester play nice as it can't mock those very well (instead it mocks the commands that wrap them). The issue I found was that normal PATH registration in Powershell doesn't fire off the WM_SETTINGCHANGE event. So instead the .NET method is used which does fire off the event. This allows for a simple restart of a terminal session to pick up the changes.

What's Next

This is far from complete and there's still a lot that can be done:

  • Code layout consolidation
  • Test case consolidation
  • List/Remove/Update PyPyInstaller Installations
  • Developer documentation
  • GitHub workflows for testing and maybe a coverage badge

Can't wait to see where this project leads. Even if it doesn't go places I learned quite a lot about Powershell!

💖 💪 🙅 🚩
cwprogram
Chris White

Posted on September 25, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

PyPyInstaller Powershell Module Initial Code Push
powershell PyPyInstaller Powershell Module Initial Code Push

September 25, 2023