Delivering the Expected - Solteq Developer Blog

Apim deployment automatization

title

Welcome again! In the previous two articles, I have shown the basic usage of APIM and how to use it to secure your backend services. If you have tried to use APIM already, you should notice that there is a lot of necessary clicking here and there. A lot of manual work needs to be done, which is tedious and error-prone. We, as developers, do not like unnecessary manual and repetitive work. So, can we do something about that?

The short answer is yes!. And the long answer, as you may guess, is: “yes, but…“. And this but is not trivial, unfortunately. However, I hope that this and the following articles will help you to make it easier to handle. So, let’s start!

In this series

  1. APIM Introduction
  2. APIM Security
  3. Ways of APIM Deployment automation

Ways of APIM deployment automation

Below I have listed the ways of automation that I recognize - if I missed something, please let me know. I will only briefly describe the first two as the rest of the article will focus on the third option.

Build-in configuration repository

If you will go to your APIM instance, you should find the Repository option:

repository

Here you can simply dump all your configuration options and settings to the git repository that will be created internally for you (Save to repository option). Similarly, you can then restore APIM from that configuration using Deploy to API Management option. You can also check out all that was saved by Azure to that repository and manually modify any part of it.

This functionality can be treated as an APIM backup and works great as such. Additionally, by checking out the repo locally, you will be able to see how to configure different options in arm templates, which can be useful knowledge.

Sadly, this approach has a lot of limitations, where the most significant ones are:

Standard ARM template deployment

As already mentioned in the previous point, we can configure most of the options in ARM templates and somehow automatically use this to restore APIM configuration. The natural conclusion here is that, if that can be done by Azure itself, it probably can be done also by us using standard ARM Azure Resource deployment. In addition to templates saved to a Git repository, we can generate the ARM template for the whole Resource Group in which our APIM is defined.

The good news is that this auto-generated template will have all the options specified, including the SSL certificates. Moreover, it can be easily parametrized so both issues from the previous method seem to be solved.

Sadly, the reality isn’t so great. If you ever have the chance to generate an ARM template from the existing resource group, you should know that those are big and hard to follow. They have all possible options defined - even default ones. And for APIM the template that we will get from Azure is not big - it is huge!

Undoubtedly, we can manually edit that template and trim to our needs. I am often creating my templates that way - first creating resources on Azure, then generating and editing the generated template, which is then used in my DevOps pipeline. Unfortunately, it will not work nicely with APIM. There is a lot of options, for instance, API policies, that are stored internally in ARM templates and passing them as parameters will be troublesome.

There is a good series of blog posts about how to make those ARM templates cleaner and easier to maintain. I recommend reading those as they also describe few features of ARM templates, that I was not using before. To name Linked Templates as a game-changer for handling broaden setup.

To sum up, by using this approach, we can save all configuration and deploy to any environment (APIM Instance). But the resulting solution will be hard to use and maintain, hence error-prone. This still suffices the Continuous Deployment requirements.

Azure CI/CD Tools

A few months back, a new project appears on MS Azure GitHub account - Azure API Management DevOps Resource Kit. Looks like MS Team noticed that APIM is hard to automate and integrate and there is a need for some tooling around that. It is partially inspired by the blog posts I mentioned before. There is a great video that explains the basic ideas behind the tooling - link. I will also borrow the graphics from MS GitHub page to illustrate the process: process illustration

This tooling is focused around ARM templates deployment. It aims to help create those templates and automatically update APIs. It supports the following workflows:

  1. Extraction - you can set up your API in Azure in a conventional way. After testing it, you can run the Extractor tool to generate ARM template. This template can be then sent as a PR to the main APIM repository.
  2. Generating from deployment artifacts - typically, each of our APIs is producing a Swagger (OpenAPI) specification file. We can use the Creator tool to run against this file and create an ARM template from it and some configuration options.
  3. Conventional ARM Templates or PowerShell scripts that can be combined with outputs from Extractor and Creator tools.

Personally, as a developer, I see the biggest power of this tooling in Creator tool and I mostly used it for my project. But the final solution requires the combination of all three options, and I am going to describe how I manage to configure Azure DevOps build and release pipeline for our sample project mentioned in the previous article.

Sample pipeline

Before I will go to the details of how we approached continuous integration and deployment pipeline with our project, let’s create a basic, starter sample.

Using CREATOR tool

Creator tool takes two main inputs - the OpenApi specification file and the YAML configuration file. Here you can find a list of the options and samples of the YAML configuration file. For a quick start, I will show a minimal working sample, to illustrate how great is this tool.

Consider that we have deployed our app and we can access its swagger file under http://www.myapp.com/swagger/myapiswagger.json. Our minimal configuration file will look like:

version: 0.0.1
apimServiceName: myapp_apim
apis:
  - name: Client
    serviceUrl: http://www.myapp.com/backend/
    openApiSpec: http://www.myapp.com/swagger/myapiswagger.json
    suffix: client
outputLocation: '.\templates\Output\'

Meaning of different options:

And that’s it! you can now run the Creator Tool using: dotnet run create --configFile CONFIG_YAML_FILE_LOCATION and you should get a sample ARM template generated:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "ApimServiceName": {
      "type": "string"
    }
  },
  "resources": [
    {
      "properties": {
        "isCurrent": false,
        "subscriptionRequired": false,
        "displayName": "Client",
        "serviceUrl": "http://www.myapp.com/backend/",
        "path": "client",
        "protocols": ["https"],
        "value": "http://www.myapp.com/swagger/myapiswagger.json",
        "format": "swagger-link-json"
      },
      "name": "[concat(parameters('ApimServiceName'), '/Client')]",
      "type": "Microsoft.ApiManagement/service/apis",
      "apiVersion": "2019-01-01",
      "dependsOn": []
    }
  ]
}

Now you can copy the resource definition to your master deployment template or link the generated template to it.

Note that this template will not create APIM Service - it should be already deployed or defined in another ARM template.

Using EXTRACTOR Tool

Using extractor is a straightforward exercise, although first, you need to define your API in APIM on Azure. Some people prefer that way of prototyping things. Moreover, it can be that you are integrating with some external service and want to use only part of its API. It could be easier to define and test a real thing first. Additionally, it is a valid scenario for a development environment to work with Azure Portal, and then provide the release version of API to the integration and production environments by creating a PR with a new ARM template.

As described in the Extractor Tool wiki, before starting you need to login to you az account and select proper subscription:

az login
az account set --subscription <subscription_id>

Then the simplest way of extracting single API is following call: dotnet run extract --sourceApimName <name_of_the_source_APIM_instance> --destinationApimName <name_of_the_destination_APIM_instance> --resourceGroup <name_of_resource_group> --fileFolder <path_to_output_folder> --apiName <api_name>

The last parameter is optional. Without it, Extractor will extract all APIs defined in the target APIM instance. Usually you just want the specific ones.

Here you can see a sample call done against default EchoApi:

dotnet run extract --sourceApimName apim-azure-service-name --destinationApimName test --resourceGroup apim-test --fileFolder {path} --apiName echo-api
API Management Template

Connecting to apim-azure-service-name API Management Service on apim-test Resource Group ...
Executing extraction for echo-api API ...
1 API's found ...
------------------------------------------
Extracting resources from echo-api API:
'create-resource' Operation found
 - Operation policy found for create-resource operation
'modify-resource' Operation found
'remove-resource' Operation found
'retrieve-header-only' Operation found
 - Operation policy found for retrieve-header-only operation
'retrieve-resource' Operation found
'retrieve-resource-cached' Operation found
 - Operation policy found for retrieve-resource-cached operation
'starter' Product association found
'unlimited' Product association found
------------------------------------------
Extracting API version sets from service
------------------------------------------
Extracting authorization servers from service
------------------------------------------
Extracting loggers from service
------------------------------------------
Extracting products from service
'starter' Product found
 - Product policy found for starter product
'unlimited' Product found
------------------------------------------
Extracting named values from service
------------------------------------------
Extracting backends from service
Templates written to output location
Press any key to exit process:

As the outcome, you will get set of ARM files describing different aspects of your API:

sample output

Conclusion

I hope you enjoyed the read. I have briefly described the different options of APIM deployment automation. All of them have their pros and cons and a valid usage scenario. But our goal is more strategic. We want to achieve a fully automated solution. Nowadays, in a cloud and DevOps world, you should not treat your resources as something precious. You should be able to delete, re-create, re-configure or duplicate any resource with a single mouse click and few minutes of waiting.

Only the last solution looks promising in that space. It will require some tweaks here and there, but at least we have our starting point.

But will it handle all the scenarios and security options I have highlighted in the previous article? What if we want to expose a subset of endpoints from OpenApi spec? What if we want to change the endpoint paths? I will address all those issues in next post, so please stay with me :)




Join us?

See what we have to offer - Careers