Create a YAML pipeline to build, clone, inspect code, and deploy for a Pull Request workflow
Published 21 January 2021
Azure DevOps pipelines allow you to create powerful automated workflows.
One example workflow for database projects is involves working in feature branches and using Pull Requests. This workflow is very commonly used with Git version control.
Overview of a Pull Request Workflow
If you’re not an experienced Git user, this may sound quite abstract. Here is a visualization. In the diagram, time flows from left to right:
In this workflow, there is a main branch. The main branch represents a mainline of code – it could be named master; it could be named trunk; the naming is up to your preferences. When new work is done, a feature branch is created – in this case we show two of these, feature1 and feature2. Multiple feature branches may be active at the same time. To keep up on other work, developers working in feature branches regularly merge changes from main into their feature branch.
When the developer finishes their feature work and is ready for their code to move towards production deployment – or if they hit a point where they just want feedback or discussion -- they push their feature branch to the central Git repo and create a Pull Request.
A Pull Request is a point of collaboration where team members review the proposed changes and give feedback. When the author and reviewers agree that the code is ready to move forward, the Pull Request is approved and completed, which then merges the code into the target branch (in this case, main).
In this example, everything merged into the main branch is expected to be code that has been reviewed and is considered to be ready to be deployable to a "higher" environment – perhaps QA or a staging environment, or even production in some cases.
This means that we need provide a very high quality review of our code as it merges into main.
Pull Requests and automation are commonly used to support quality code review by validating code and creating a quality, production like environment for code review.
Configuring Pull Request Automation in Azure DevOps
Pull Request automation is configured in Azure DevOps by creating a pipeline and a branch policy.
The branch policy can require that in order to merge a change into the target branch:
- The pipeline must succeed
- One or more reviewers must review the change
An Example YAML Pipeline for Pull Request Automation
For quality review of database changes, it is essential for reviewers to have a production-like environment which contains the appropriate state of the database.
In this example, we provide this for a SQL Change Automation project by:
- Building the SQL Change Automation Project
- Creating a clone of the production environment using SQL Clone
- Creating a Release Artifact against the clone
- The Release Artifact contains human-readable reports for the reviewer, including static code analysis and summaries of database drift which may have occurred
- Additional tools and/or custom scripts may be run against the database code to inspect it if further code analysis is desired
- In this example we check the deployment script for specific TSQL commands and warn or throw an error if they are found
- There may be times when you wish to override errors from a script of this type-- we use a pipeline variable to support this scenario
- Deploying the Release Artifact to the cloned database
This pipeline pattern may be extended to also deploy applications to this review environment, to provide the reviewer with the ability to perform further tests at the application level.
Example YAML Pipeline for a SQL Change Automation Project
trigger: - main # This pipeline uses a self-hosted agent which has access to the # SQL Server we are deploying to as well as to the SQL Clone service # https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows pool: YourPoolName steps: # First we build our project - task: RedgateSqlChangeAutomationBuild@4 displayName: SQL Change Automation Build inputs: operation: 'Build SQL Change Automation Project' sqlProj: 'LiveDemo/LiveDemo.sqlproj' packageName: 'LD' tempServerTypeBuild: 'sqlServer' tempServerNameBuild: '.' authMethodBuild: 'windowsAuth' nugetPackageVersionSelector: 'Specific' nugetPackageVersion: '1.0' nugetPackageVersionUseBuildId: true requiredVersionOfDlma: 'latestInstalled' # The build publishes an artifact. We now download that # to our self-hosted agent - task: DownloadPipelineArtifact@2 displayName: Download build artifact inputs: buildType: 'current' artifactName: 'Database Build Artifact' targetPath: '$(System.DefaultWorkingDirectory)' # This PowerShell reads the name of the source branch, # replaces any / characters with _'s, and sets this as a variable. # We will be able to reference $(PRString) in future steps which # we will use to name a database. This is simply an example of # how it may be useful to use variables. - task: PowerShell@2 displayName: Set $(PRString) variable inputs: targetType: 'inline' script: | $path="$(Build.SourceBranch)" # Write-Host "$path" $path= $path -replace "/", "_" Write-Host "##vso[task.setvariable variable=PRString]$path" # We call SQL Clone to create a database clone just for this PR # The clone name contains the source branch for this PR - task: RedgateSqlCloneClone@4 displayName: Clone database inputs: cloneServer: 'sqlclone' imageNameForClone: 'LiveDemo' cloneSqlServer: 'YourServerName' cloneName: 'LiveDemo_$(PRString)' deleteClone: true # We call SQL Change Automation to create a Release Artifact # against the clone. We export the Release Artifact, which may # be useful for reference by reviewers of the PR. # In real-world cases, a highly-available fileshare # is recommended for Release Artifacts. - task: RedgateSqlChangeAutomationRelease@4 displayName: Create Release Artifact with SQL Change Automation inputs: Operation: 'Create' NuGetFile: '$(System.DefaultWorkingDirectory)' ExportPath: 'C:\DBAutomation\ReleaseArtifacts\$(PRString)\$(Build.BuildNumber)' TargetDatabaseConnectionType: 'explicit' TargetDatabaseServer: '.' TargetDatabaseName: 'LiveDemo_$(PRString)' TargetAuthMethod: 'windowsAuth' AbortOnWarning: 'none' requiredVersionOfDlma: 'latestInstalled' # This PowerShell checks the deployment script in our Release Artifact for # TSQL commands which we want to warn or throw an error on # A pipeline variable is used to allow for this to be re-run with an # override/bypass if an exception is to be made - task: PowerShell@2 displayName: Check TargetedDeploymentScript.sql for problem commands inputs: targetType: 'inline' script: | If (${env:BYPASS_CHECK_OF_TARGETEDDEPLOYMENTSCRIPT_SQL_FOR_PROBLEM_COMMANDS} -eq 'True') { Write-Output "Bypassing check of TargetedDeploymentScript.sql based on pipeline variables." exit 0 } $targetedDeploymentScript = 'C:\DBAutomation\ReleaseArtifacts\$(PRString)\$(Build.BuildNumber)\TargetedDeploymentScript.sql' $searchPatternWarnings = @( "CREATE", "EXEC[\W]+\(.*", "sp_executesql" ) $searchPatternErrors = @( "GRANT" "TRUNCATE", "DROP", "DELETE", "sp_rename" ) $warnings = @( Foreach ($s in $searchPatternWarnings) { Select-String -Path $targetedDeploymentScript -Pattern "(^|\s)$s($|\s)" -AllMatches | ` # Exclude lines specific to __SchemaSnapshot objects Select-String -Pattern "[__SchemaSnapshot]" -NotMatch -SimpleMatch | Select-Object LineNumber, Line } ) $ct = $warnings.Count if ($ct -gt 0) { Write-Output "##vso[task.logissue type=warning;code=100;] $($ct) warnings found" Write-Output $warnings | Format-Table -RepeatHeader } else { Write-Output "No warnings found." } $errors = @( Foreach ($s in $searchPatternErrors) { Select-String -Path $targetedDeploymentScript -Pattern "(^|\s)$s($|\s)" -AllMatches | ` # Exclude lines specific to __SchemaSnapshot objects Select-String -Pattern "[__SchemaSnapshot]" -NotMatch -SimpleMatch | Select-Object LineNumber, Line } ) $ct = $errors.Count if ($ct -gt 0) { Write-Output "##vso[task.logissue type=error;code=100;] $($ct) errors found" Write-Output $errors | Format-Table -RepeatHeader Exit 1 } else { Write-Output "No errors found." } # We call SQL Change Automation to deploy the Release Artifact # to the clone. - task: RedgateSqlChangeAutomationRelease@4 displayName: Deploy Release Artifact with SQL Change Automation inputs: Operation: 'DeployFromResources' ImportPath: 'C:\DBAutomation\ReleaseArtifacts\$(PRString)\$(Build.BuildNumber)' TargetDatabaseConnectionType: 'explicit' TargetDatabaseServer: '.' TargetDatabaseName: 'LiveDemo_$(PRString)' TargetAuthMethod: 'windowsAuth' requiredVersionOfDlma: 'latestInstalled'