Introduction Link to heading
In Part 1, we dived into the creation of a powerful Bash script for batch cleanup of Azure Container Registries (ACRs). It provided you with a manual method to efficiently manage your container images, including exemption lists, execution, and detailed logging and reporting.
Now, in Part 2, we take automation to the next level by introducing Azure Pipelines. Why might you want to use Azure Pipelines in conjunction with the script from Part 1? The answer is simple: automation brings numerous advantages. Here are some advantages…
- Efficiency: Automation takes out the requirement for manual execution, saving time and exertion in the cleanup cycle. Your ACR can be kept well maintained without constant supervision by scheduling pipeline runs or triggering them in response to specific events.
- Consistency: Every time Azure Pipelines runs, a consistent cleanup procedure is enforced, lowering the likelihood of human error. This consistency is critical for guaranteeing that your ACR follows picture maintenance strategies.
- Customization: Azure Pipelines allows you to set parameters, making it adaptable to various repositories, thresholds, and exemption lists. You can easily tailor the cleanup process to different environments or projects within your organization.
- Integration: Azure Pipelines seamlessly integrates with your existing CI/CD pipelines and DevOps practices. You can incorporate ACR cleanup as a routine step in your deployment pipeline, ensuring that only the most relevant images are deployed.
- Detailed Reporting: Pipelines provide built-in reporting and visualization tools. You can monitor cleanup progress, track historical data, and gain insights into the efficiency of your image management.
In this part, we’ll go through the process of setting up an Azure Pipeline to automate your ACR cleanup. We’ll explore how to create a pipeline, customize parameters, and effortlessly maintain your container images. By combining the power of Azure Pipelines with the efficiency of the Bash script from Part 1, you’ll establish a robust system for managing your Azure Container Registry, enhancing your DevOps practices, and optimizing your image management.
Pipeline Parameters: Customizing Your ACR Cleanup Link to heading
Let’s delve into one of the key aspects of setting up an Azure Pipeline for your Azure Container Registry (ACR) cleanup: defining pipeline parameters. Customizing these parameters empowers you to adapt the cleanup process to different repositories, image retention policies, and specific project requirements.
Tailoring Cleanup to Your Needs: Link to heading
Azure Pipelines offer a flexible and customizable approach to ACR cleanup. Before we explore how to create a pipeline, let’s understand the significance of defining pipeline parameters:
- Repository Selection: You can specify which ACR repositories should undergo cleanup. This parameter enables you to target specific repositories for image management while leaving others untouched.
- Threshold Configuration: Customize the maximum threshold for images to retain in each repository. Different projects or environments may have varying retention policies, and pipeline parameters allow you to set these thresholds accordingly.
- Exemption Lists: Just like in the Bash script from Part 1, you can implement exemption lists in your pipeline. Define which repositories or images should be exempted from the cleanup process, ensuring critical images are preserved.
- Automation Frequency: Adjust the frequency of pipeline runs. You can schedule regular cleanup runs at convenient times or trigger pipelines in response to specific events, such as image pushes or pull requests.
Pipeline Parameters in Action: Link to heading
Let’s consider an example: You have multiple ACR repositories, each serving different applications. Some applications require frequent updates, while others are more stable and should retain images for a longer duration. By defining pipeline parameters, you can automate cleanup for each repository, specifying unique thresholds and exemption lists.
For instance, you might configure your pipeline to:
- Clean up the “development” repository every day, retaining only the latest five images.
- Clean up the “production” repository weekly, retaining the latest ten images.
- Exempt the “critical_app” repository from cleanup to ensure essential images are always available.
By customizing these parameters, you align your ACR cleanup process with the specific needs of your applications, ensuring efficient resource utilization and adherence to image retention policies.
Job Execution: Automating ACR Cleanup with Azure Pipelines Link to heading
Now we can explore the heart of your Azure Pipeline setup: job execution. This is where the magic happens, as your pipeline takes the defined parameters and Bash script from Part 1 and automates the cleanup of your Azure Container Registry (ACR).
Pipeline Setup Overview: Link to heading
Before we dive into job execution, let’s briefly recap the pipeline setup process:
- Pipeline Creation: You initiate the process by creating a new Azure Pipeline in your Azure DevOps environment. This pipeline serves as the automation engine for your ACR cleanup.
- Parameter Configuration: As discussed in the previous section, you define pipeline parameters to customize the cleanup process. These parameters include repository selection, threshold configuration, exemption lists, and automation frequency.
- Bash Script Integration: You incorporate the Bash script developed in Part 1 into your pipeline. This script contains the logic for ACR cleanup, leveraging the defined parameters to perform image management efficiently.
- Job Execution: This is the core of your pipeline, where the Bash script job is executed based on the provided parameters. The script interacts with your ACR, identifies images that exceed retention thresholds, and initiates deletions while respecting exemption lists.
Job Execution: Link to heading
Here’s a breakdown of how job execution takes place within your Azure Pipeline:
- Parameter Input: The pipeline collects input from the defined parameters, such as the selected repositories, threshold values, exemption lists, and automation schedule. These inputs determine which ACR repositories are cleaned up and how many images are retained.
- Script Execution: The pipeline triggers the execution of the Bash script, passing the input parameters as variables. The script uses this data to connect to your ACR, list tags, and commence cleanup operations.
- Iterative Cleanup: The script iterates through the selected repositories, adhering to the defined parameters. It identifies images that exceed retention thresholds and deletes them, ensuring that critical images are exempted as specified.
- Logging & Reporting: Throughout the cleanup process, the script generates detailed log messages, just as in Part 1. These logs capture the status of image deletions, including successful deletions and any failures. This information is essential for transparency and auditing.
- Automation: Depending on your parameter configuration, the pipeline can be scheduled to run at specific intervals or triggered by events like image pushes. This ensures that your ACR cleanup remains up to date without manual intervention.
Below is a complete example of how te pipeline would look, this showcases how the various parameters, logging and reporting are incorporated.
trigger:
- '*'
pr: none
jobs:
- job: 'ACRCleanup'
pool:
vmImage: 'ubuntu-latest'
parameters:
azureServiceConnection: # Define your Azure service connection
repositoriesToCleanup: # Define repositories to clean up
retentionThreshold: # Define retention threshold
exemptionList: # Define exemption list
automationSchedule: # Define automation schedule
steps:
- task: AzureCLI@2
displayName: "Clean Up registry"
inputs:
azureSubscription: '$(azureServiceConnection)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
#!/bin/bash
# Define your Azure Container Registry name
registry_name="myregistry"
# Define the maximum threshold for images to retain
maxthreshold=$(($(date +"%d") % $(date +"%H") + 5))
# Create an array to store log messages
log_messages=()
# Initialize counters for logging
totalImagesDeleted=0
totalImagesRetained=0
totalImagesFailed=0
# Log the current date and time in the report
log_messages+=("Report generated on $(date)")
# Define the list of repositories to delete
repositories_to_delete=($repositoriesToCleanup)
# Log the repositories subject to trimming
log_messages+=("Repositories subject to trimming:")
for repo in "${repositories_to_delete[@]}"; do
log_messages+=("- $repo")
done
log_messages+=("Max allowed images per repository: $retentionThreshold")
# Log a separator
log_messages+=("----------------------------------------------------")
# Loop through repositories to delete
for repo in "${repositories_to_delete[@]}"; do
# List tags in the repository
tags=$(az acr repository show-tags --name "$registry_name" --repository "$repo" --orderby time_asc --output tsv)
totalImages="$(echo "$tags" | wc -w)"
# Log the tags to be deleted in this repository
log_messages+=("Repository: $repo")
log_messages+=("Total images: $totalImages")
if [ $totalImages -le $retentionThreshold ]; then
log_messages+=("No deletion required: $totalImages meets allowed threshold")
log_messages+=("---")
continue
fi
tagCounter=$(expr $totalImages - $retentionThreshold)
log_messages+=("Expected total images to delete: $tagCounter")
deletedTagCounter=0
log_messages+=("Deleting images...")
for tag in $tags; do
if [ $deletedTagCounter -le $(expr $tagCounter - 1) ]; then
# Delete the tag
if az acr repository delete --name "$registry_name" --image "$repo:$tag" --yes; then
log_messages+=("- $repo:$tag")
((deletedTagCounter++))
else
log_messages+=("Failed to delete: $repo:$tag")
((totalImagesFailed++))
fi
else
break
fi
done
# Log the total number of deleted tags for this repository
log_messages+=("Total images deleted in $repo: $deletedTagCounter")
log_messages+=("---")
((totalImagesDeleted+=deletedTagCounter))
((totalImagesRetained+=totalImages-deletedTagCounter))
done
# Log a separator
log_messages+=("----------------------------------------------------")
# Log summary information
log_messages+=("Cleanup summary:")
log_messages+=("Total Images Deleted: $totalImagesDeleted")
log_messages+=("Total Images Retained: $totalImagesRetained")
log_messages+=("Total Images Failed to Delete: $totalImagesFailed")
# Save the log to a text file
log_file="cleanup_log.txt"
printf "%s\n" "${log_messages[@]}" > "$log_file"
# Print the log file path
echo "Cleanup log saved as $log_file"
- task: ShahzebKhan.sap-azure-pipelines-html-report.PublishHtmlReport.PublishHtmlReport@3
displayName: 'Publish clean-up Report'
condition: always()
inputs:
tabName: "Clean-up report"
reportDir: 'deletion_report.html'
- task: CopyFiles@2
displayName: 'Copy report to: $(Build.ArtifactStagingDirectory)/clean'
inputs:
SourceFolder: $(Build.SourcesDirectory)
TargetFolder: $(Build.ArtifactStagingDirectory)/clean
Contents: 'deletion_report**'
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact: clean-rpt'
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)/clean
ArtifactName: clean-rpt
- Trigger: This pipeline is set to trigger on all branches (*) and doesn’t run for pull requests (pr: none). Adjust the trigger settings based on your requirements.
- Job Definition: We define a job named ‘ACRCleanup’ that specifies the virtual machine image as ubuntu-latest. You can choose an appropriate image for your needs.
- Azure Service Connection: The azureServiceConnection parameter should be defined outside of the pipeline YAML. It represents the connection to your Azure subscription, which allows the pipeline to interact with Azure resources.
- The pipeline includes parameters for repositoriesToCleanup, retentionThreshold, exemptionList, and automationSchedule, which are defined and utilized in the inlineScript of the AzureCLI task.
- The retentionThreshold parameter determines the maximum number of images to retain per repository, and this value is used to calculate the maxthreshold.
- The repositoriesToCleanup parameter is passed to the repositories_to_delete array, allowing you to specify which repositories to clean up.
- The exemptionList parameter, as mentioned in Part 1, can be utilized in the script to exempt specific repositories from cleanup.
For the pipeline tasks: Link to heading
- Step 1 - Clean Up Registry (AzureCLI): This step executes the cleanup script. It uses the Azure CLI (AzureCLI@2) task, which runs a Bash script. Replace “# Insert your cleanup script here” with the actual cleanup script you developed in Part 1.
- Step 2 - Publish Cleanup Report (HTML Report): This step publishes an HTML report generated during the cleanup process. It uses the “ShahzebKhan.sap-azure-pipelines-html-report.PublishHtmlReport.PublishHtmlReport@3” task. Make sure you have this extension installed or use a preferred extension that generates HTML reports.
- Step 3 - Copy Report Files: After generating the report, this step copies the report files to the artifact staging directory. This directory is where build artifacts are stored.
- Step 4 - Publish Cleanup Report Artifact: The final step publishes the cleanup report artifact. It uses the “PublishBuildArtifacts@1” task to publish the report files. The PathtoPublish input specifies the path to the report files, and the ArtifactName input defines the name of the artifact.
By configuring this pipeline, you can automate your ACR cleanup, execute the cleanup script, generate detailed HTML reports, and store them as artifacts for further analysis or sharing with your team.
In Conclusion: Mastering ACR Cleanup with Automation
In this two-part blog series, we’ve embarked on a journey to streamline and automate the cleanup of your Azure Container Registry (ACR). From crafting a powerful Bash script to orchestrating the process with Azure Pipelines, we’ve explored every facet of efficient ACR management.
The advantages of automation are evident. By combining a meticulously designed cleanup script with Azure Pipelines, you not only save valuable time but also ensure consistency, customization, and transparency in your image management practices. Automation becomes your trusted ally in maintaining an organized and optimized ACR.
As you put these insights into practice, remember the core principles:
-
Customization: Tailor your cleanup process to the unique requirements of your repositories, defining retention policies and exemption lists that suit your applications.
-
Efficiency: Let automation work for you. Schedule regular cleanup runs or trigger them in response to specific events, allowing your ACR to remain clutter-free without manual intervention.
-
Reporting: Detailed logs and reports keep you informed about the cleanup’s progress, enabling you to track performance and make informed decisions.
With the knowledge and examples shared in this blog series, you’re well-equipped to conquer the challenge of ACR cleanup. Whether you’re managing a single ACR or a multitude, automation simplifies the process, ensuring that only the most relevant and up-to-date images find their place in your registry.
As you embark on your ACR cleanup journey, may your containers remain swift, secure, and ready for the future of your applications.