CI/CD for Automating SP Migration with ShareGate
Kinga
Posted on September 5, 2022
ShareGate application allows scheduling tasks, so the migration is executed outside of office hours.
The process may also be automated using PowerShell, which is especially important if content transformation must be executed in the process.
I decided to go one step further and automate it using Azure Pipelines.
Prerequisites
ShareGate is not available as a Marketplace extention and cannot be easily used in Microsoft-hosted pipelines. The software has to be installed and activated with a license key- which is assigned per user per workstation.
In order to automate the migration process, I am using a self-hosted agent configured on a Virtual Machine with ShareGate installed and properly licensed.
The process is using the concept presented in the Schedule your SharePoint migration using PowerShell and ShareGate with the only difference being the trigger.
Pipelines
Extract and Transform
The first stage of the migration process exports content from the SharePoint lists and performs transformations.
The ShareGate_ExportToExcel.yml job is using the self-hosted agent with Sharegate installed, whilst the TransformData.yml job is executed on a Microsoft-hosted agent:
jobs:
- job: ${{parameters.jobName}}
displayName: 'ShareGate Export to Excel'
dependsOn: ${{ parameters.dependsOn }}
timeoutInMinutes: 360
pool: ShareGate # IMPORTANT
variables:
- group: ${{parameters.groupKeyVaultName}}
- name: scriptsLocation
value: "$(Build.SourcesDirectory)/Pipelines/scripts"
steps:
- checkout: self
persistCredentials: true
- task: powershell@2
name: SGMigration
inputs:
filePath: $(scriptsLocation)/SG-ExportToExcel.ps1
arguments: '-userName_SP ''$(SP-SvcAcct-Name)'' -userPassword_SP (ConvertTo-SecureString -AsPlainText $(SP-SvcAcct-Pswd) -Force) `
-listNames ''Portfolio'', ''Schedule'', ''Archive'' '
displayName: Export to Excel
```
#### Load to downstream environments
The next stages execute the content migration to selected downstream environments.
During test migrations, I'm reusing the previously exported and transformed Excel, even if the data may be slightly outdated. Being able to use the same Excel file is saving me a lot of time; it took 1,5 hours to generate it
The import process consists of several steps:
- updating lists configuration to, for example, allow older item versions which may be missing values in the fields that are currently mandatory
- importing content using source SharePoint site and the transformed Excel file
- resetting lists configuration to its previous state
- validating list configuration and list content
```yml
jobs:
# set the list configuration: required columns, fillIn choices etc
- template: jobs_SG_PreMigration.yml
parameters:
environment: ${{parameters.environment}}
groupVariablesName: ${{parameters.groupVariablesName}}
groupKeyVaultName: ${{parameters.groupKeyVaultName}}
jobName: SPOPreMigration
# import with ShareGate
- template: jobs_SG_Migration.yml
parameters:
environment: ${{parameters.environment}}
groupVariablesName: ${{parameters.groupVariablesName}}
groupKeyVaultName: ${{parameters.groupKeyVaultName}}
jobName: Import
dependsOn: SPOPreMigration
# set the list configuration back: required columns, fillIn choices etc
- template: jobs_SG_PostMigration.yml
parameters:
environment: ${{parameters.environment}}
groupVariablesName: ${{parameters.groupVariablesName}}
groupKeyVaultName: ${{parameters.groupKeyVaultName}}
jobName: SPOPostMigration
dependsOn: Import
# run list validation
- template: jobs_SPO_Validate.yml
parameters:
environment: ${{parameters.environment}}
groupVariablesName: ${{parameters.groupVariablesName}}
groupKeyVaultName: ${{parameters.groupKeyVaultName}}
jobName: SPOValidate
searchOrphanedUsers: true
dependsOn: SPOPostMigration
```
Once again, only the import has to be executed on the self-hosted agent
```yml
jobs:
- deployment: ${{parameters.jobName}}
displayName: 'ShareGate Migration'
environment: ${{ parameters.environment }}
dependsOn: ${{ parameters.dependsOn }}
timeoutInMinutes: 360
pool: ShareGate # IMPORTANT
strategy:
runOnce:
deploy:
steps:
- checkout: self
persistCredentials: true
# Copy results
- task: CopyFiles@2
condition: always()
displayName: 'Copy SG mapping files'
inputs:
contents: |
$(Build.SourcesDirectory)\SPO_Migration\ShareGateConfig\*
targetFolder: 'D:\ShareGateConfig'
flattenFolders: true
overWrite: true
- task: powershell@2
name: SGMigration
inputs:
filePath: $(scriptsLocation)/SG-Migrate.ps1
arguments: '-tenantName ''$(Az_TenantName)'' -siteName ''$(SPO_SiteName)'' `
-userName_SPO ''$(SPO-SvcAcct-Name)'' -userPassword_SPO (ConvertTo-SecureString -AsPlainText $(SPO-SvcAcct-Pswd) -Force) `
-userName_SP ''$(SP-SvcAcct-Name)'' -userPassword_SP (ConvertTo-SecureString -AsPlainText $(SP-SvcAcct-Pswd) -Force) `
-resultsFolder ''$(System.DefaultWorkingDirectory)\results'' `
-sourceFolder ''$(filesLocation)'' `
-environment ''${{ parameters.environment }}'' -buildId ''${{ parameters.environment }} $(Build.BuildNumber)'' '
displayName: ShareGate Migration
# Copy results
- task: CopyFiles@2
condition: always()
displayName: 'Archiving migration results'
inputs:
contents: |
$(System.DefaultWorkingDirectory)\results\*
targetFolder: '$(Build.ArtifactStagingDirectory)'
flattenFolders: true
- task: PublishBuildArtifacts@1
inputs:
artifactName: '${{ parameters.environment }} ShareGate'
condition: always()
displayName: 'Publish Artifact: ShareGate'
It is even possible to use set the ShareGate task name using pipeline's variables.
The -buildId ''${{ parameters.environment }} $(Build.BuildNumber)'' ' parameter is used in PowerShell
Copy-Content -TaskName "$srcListName to $dstListName ($buildId)"
allowing me to comfortably review the tasks progress from using the ShareGate application:
Posted on September 5, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.