Exposing A Service Bus Topic Using Azure Api Management

Introduction

Microsoft released a new service to Azure, called API Management. This service was released on May the 12^th 2014.

Currently the Azure API Management is still in Preview, but already it enables us to easily create an API façade over a diverse set of currently available Azure services like Cloud Services, Mobile Services, Service bus as well as on premise web-services.

Instead of listing all the features and write an extensive introduction about Azure API Management I’d rather supply you with a few links which contains more information about the Azure API Management service:

Microsoft:

http://azure.microsoft.com/en-us/services/api-management/

http://azure.microsoft.com/en-us/documentation/services/api-management/

Blogs:

http://trinityordestiny.blogspot.in/2014/05/azure-api-management.html

http://blog.codit.eu/post/2014/05/14/Microsoft-Azure-API-Management-Getting-started.aspx

Some more background

As most of my day-to-day work involves around the Microsoft Integration space in which I am mainly focusing on BizTalk Server, BizTalk Services and Azure in general, I was looking to a find a scenario in which I, as an Integration person, could and would use Azure API management.

The first thing which popped in to my mind; wouldn’t it be great to virtualize my existing BizTalk Service Bridges using Azure API management. Well currently this is not possible, as the only authentication and authorization method on a BizTalk Service Bridge is ACS (Access Control Service) and this is not supported in the Azure API Management Service.

Luckily with the last feature release of BizTalk Services included support for Azure Service Bus Topics / queues as a source J and luckily for me Azure Service Bus supports SAS (Shared Access Signatures) and using such a signature I am able to generate a token and use this token in the HTTP Header – Authorization section of my http request.

Knowing the above, I should be able to define API’s which virtualize my Service bus Endpoints. Create a product combining the defined API’s and assign policies to my API operations.

Sounds easy? Doesn’t it. Well it actually is. So without further ado, let’s dive into a scenario which involves exposing an Azure Service bus Topic using Azure API Management.

Getting started

Before we actually start please note that the sample data (Topic names, user-accounts, topic subscriptions, topic subscription rules etc.) I use for this ‘Step by step’ guide is meant for a future blog post 😉 extending an article I posted a while back on the TechNet Wiki

Other points you should keep in mind are

• Messages send to a TOPIC may not exceed 256Kb in size, if a message is larger you will receive an error message from Service Bus telling you that the message is too large.
• Communication to service bus is asynchronous; thus we send a message and all we get back is an HTTP code telling us the status of the submission (200 OK, 210 Accepted etc.) or an error message indicating that something went wrong (401 Access Denied, etc.). So actually our scenario is using the Fire and Forget principle.
• You will have to create a SAS Token, which needs to be send in the header of the message in order to authenticate to Service bus.

Enough set, let’s get started.

For the benefit of the reader I’ve added hyperlinks below such that you can skip those sections involved which you might already know.

Sections Involved

• Create a new Azure Account
• Provision an Azure Service Bus entity
• Create a new Azure Service Bus TOPIC
• Add Subscriptions and filters to your newly created TOPIC
• Assign Shared Access Policies to your TOPIC
• Generate a SAS Token
• Create a new API Management Instance
• Azure API Management Configuration
• Create and configure a new API
• Create a new Product
• Create a new Policy for your defined API operations
• Setting up your Policy definition
• Create a Group, an User and Add subscribe the user to a product
• Test your API
• Takeaways
• Conclusion

Create a new Azure Account

If you don’t already have an Azure account, you can sign up for a free trial here

Provision an Azure Service Bus entity

Once you’ve signed up for an Azure account, login to the Azure Portal and create a new Azure Service Bus Topic by following the steps listed below.

1. If you have logged in to the preview portal, click on the ’tile’ Azure Portal. This will redirect you to an alternative portal which allows for a more complete management of your Azure Services.

   </li>

2. In the ‘traditional’ portal click on Service Bus

   </li>

3. Create a new Namespace for your Service bus Entity

   </li>

4. Enter a name for your Service bus namespace and select the region to which it should be deployed and select the checkmark which starts the provisioning.

   </li>

5. Once the provisioning has finished, select the Service Bus Entity and click on Connection Information

   </li>

6. A window will appear with Access Connection Information, in this screen copy the ACS Connection String to your clipboard. (We will need this connection string later on) and then click on the checkbox to close to window

   </li> </ol>

    

   Create a new Azure Service Bus TOPIC

   Now that a new Service Bus Entity has been provisioned, we can proceed with creating a TOPIC within the created Service Bus entity, for this we will use the Service Bus Explorer from Paolo Salvatori. Which you can download here. Once you’ve downloaded this tool, extract it and execute the ServiceBusExplorer.exe file and follow the below mentioned steps.

   1. Press CRTL+N which will open the “Connect to Service Bus Namespace” window
   2. In the Service Bus Namespaces box, select the following option from the dropdown: “Enter Connection String”
   3. Copy and paste the ACS Connection String (you copied earlier, see previous section step 6) and once done press “OK”

      </li>

   4. The Service Bus Explorer should now have made a connection to your Service Bus Entity

      </li>

   5. In the menu on your left, select the TOPIC node, right click and select “Create Topic”

      </li>

   6. In the window which now appears enter a TOPIC name “managementapi_requests” in the “Path” box and leave all other fields blank (we will use the defaults). Once done press the “Create button”

      </li>

   7. Your new Topic should now have been created

      </li> </ol>

      Add Subscriptions and filters to your newly created TOPIC

      Now that we have created a TOPIC it is time to add some subscriptions. The individual subscriptions we will create will contain a filter such that messages which are eventually posted to this TOPIC end up in a subscription based on values set in the HTTP header of the submitted messages. In order to set up some subscriptions follow the below mentioned steps:

      1. Go back to your newly created TOPIC in the Service Bus Explorer
      2. Right click on the TOPIC and select “Create Subscription”

         </li>

      3. The Create Subscription windows will now show in which you should execute the following steps
         • A) Subscription Name: BusinessActivityMonitoring
         • B) Filter: MessageAction=’ BusinessActivityMonitoring’
         • C) Click on the Create button

         </li>

      4. Now repeat Steps 2 and 3 in order to create the following subscription
         • Subscription Name: Archive
         • Filter: 1 = 1
      </ol>

      Assign Shared Access Policies to your TOPIC

      At this point we have set up our Topic and added some subscriptions le viagra est il efficace. The next step consists of adding a Shared Access Policy to our topic. This policy than allows us to generate a SAS token which later on will be used to authenticate against our newly created Topic. So first things first, let’s assign a Shared Access Policy first. The next steps will guide you through this.

      1. Login to the Azure Portal
      2. Go to the Service Bus menu item and select the Service Bus Service you created earlier by clicking on it.
      3. Now select TOPICS from the tab menu

         </li>

      4. Select the Connection Information Icon on the bottom

         </li>

      5. A new window will pop up, in this windows click on the link “Click here to configure”

         </li>

      6. Now un the shared access policies:
         • A) Create a new policy named ‘API_Send’
         • B) Assign a Send permission to this policy
         • C) Create a new policy named ‘API_Receive’
         • D) Assign the Listen permission to this policy
         • E) Create a new policy named ‘API_Manage’
         • F) Assign the Manage permission to this policy
         • G) Click on the SAVE icon on the bottom

         </li>

      7. At this point for each policy a Primary and Secondary key should be generated.

         </li> </ol>

         Generate a SAS Token

         Once we’ve added the policies to our Topic we can generate a token. In order to generate a token, I’ve build a small forms application which uses part of the code which was originally published by Santosh Chandwani. Click the following link to start downloading the application “Sas Token Generator“. Using the SAS Token Generator application we will now generate the token signatures.

         1. Download the Sas Token Generator and start it
         2. Fill out the following form data
            • A) Resource Uri = HTTPs Endpoint to your TOPIC
            • B) Policy Name = API_Send
            • C) Key = The Primary Key as previously generated
            • D) Expiry Date = Select the date you want the Sas token to expire
            • E) Click on generate

            </li>

         3. After you have clicked GENERATE by Default a file will be created on your desktop containing all generated Sas Tokens, the file is named SAS_tokens.txt. Once saved you will be asked if you want to copy the generated token to your Clipboard. Below 2 images depicting the message-prompt as well as the contents stored in the generated file.Perform step 2 for the other 2 policies as well (API_Listen and API_Manage)

            

            </li> </ol>

            Create a new API Management Instance

            At this point we have set up our Service Bus Topic, Subscriptions and have generated our Sas Tokens we are all set to start exposing the newly created service bus topic using Azure API Management, but before we can start with this we need to create a new API Management Instance. The steps below detail how to do this.

            1. Login to the Azure Portal
            2. Click on API Management, in the right menu-bar, and click on the link “Create an API Management Service”

               </li>

            3. A menu will pop up in which you need to select CREATE once more

               </li>

            4. At this point a new window will appear:
               • A) Url: Fill out an unique name
               • B) Pricing Tier: Select Developer
               • C) Subscription: Check your subscription Id (Currently it does not show the subscription name, which I expect to be fixed pretty soon)
               • D) Region: Select a region close to you
               • E) Click on the right arrow

               </li>

            5. You now will end up at step 2:
               • A) Organization Name: Fill out your organization name
               • B) Administration E-Mail: Enter your email address
               • C) Click on the ‘Checkmark’ icon

               </li>

            6. In about 15 minutes your API management Service will have been created and you will be able to login.
            </ol>

            Azure API Management Configuration

            Now that we have provisioned our Azure API management service it is time to create and configure an API which exposes the previously defined Azure Service Bus Topic such that we can send messages to it. The API which we are about to create will expose one operation pointing to the Azure Service Bus Topic and will accept both XML as JSON messages. Later on we will define a policy which will ensure that if a JSON message is received it is converted to XML and that the actual calls to the Service Bus Rest API are properly authenticated using our SAS token created earlier.

            So let’s get started with the creation of our API by Clicking the Manage Icon which should be visible in the menu bar at the bottom of your window.

            

            Once you’ve clicked the Manage icon, you should automatically be redirected to the API Management portal.

            

            Create and configure a new API

            Now that you are in the Azure API Management Administration Portal you can start with creating and configuring a new API, which will virtualize your Service Bus Topic Rest Endpoints. In order to do so follow the following steps.

            1. Click on the API’s menu item on your left

               </li>

            2. Click on ADD API

               </li>

            3. A new window will appear, fill out the following details
               • A) Web API title
                 • Public name of the API as it would appear on the developer and admin portals.
               • B) Web Service Uri
                 • This should point to your Azure Service Bus Rest Endpoint. Click on this link to get more information. The format would be: http{s}://{serviceNamespace}.servicebus.Windows.net/{topic path}/messages
               • C) Web API Uri suffix
                 • Last part of the API’s public URL. This URL will be used by API consumers for sending requests to the web service.
               • D) Once done, press Save

               </li>

            4. Once the API has been created you will end up at the API configuration page

               </li>

            5. Now click on the Settings Tab
               • A) Enter a description
               • B) Ensure to set authentication to None (we will use the SAS token later on, to authenticate)
               • C) Press Save

               </li>

            6. Now click on the Operations Tab, and click on ADD Operation.
               • Note: detailed information on how to configure an operation can be found here

               </li>

            7. A form will appear which will allow you to configure and add an operation to service. By default the Signature menu item is selected, so we start with configuring the signature of our operation.
               • A) HTTP verb: Choose POST as we will POST messages to our Service Bus Topic
               • B) URL Template: We will not use an URL template, so as a default enter a forward-slash ” / “
               • C) Display Name: Enter a name, which will be used to identify the operation
               • D) Description: Describe what your operation does

               </li>

            8. Now click on the Body item in the menu bar on the left underneath REQUESTS (we will skip caching as we don’t want to cache anything), and fill out the following fields
               • A) Description: add a description detailing how the request body should be represented

               </li>

            9. Now click on the ADD REPRESENTATION item just underneath the description part and enter Application/XML
            </ol> </ol>

            

            1. Once you’ve added the representation type, you can add a representation example.

            

            1. Now once again click on the ADD REPRESENTATION item just underneath the description part and enter Application/JSON

            

            1. Once you’ve added the representation type, you can add a representation example.

            

            1. Now click on the ADD item in the menu bar on the left underneath RESPONSES (We will skip Caching, Parameters and Body as we don’t need it)

               </li>

            2. Start typing and select a response code you which to return once the message has been send to the service operation.

               </li>

            3. Now you could add a description and add a REPRESENTATION, however in our case we will skip this as a response code 202 Accepted is all we will return.
            4. 
            5. Press save
            </ol>

            Create a new Product

            Now that we have defined our API we need to make it part of a Product. Within Azure API Management this concept has been introduced as a container containing one or more API definitions to which consumers (developers) can subscribe. In short if your API is not part of a product definition consumers can not subscribe to it and use it. More information regarding the ‘Product’ definition can be found here.

            In order to create a new product, we need to perform the following steps:

            1. Select the Products menu item from within the API Management Administration Portal and click on it.

               </li>

            2. A new window will appear listing all products currently available, in this window click on the ADD PRODUCT item

               </li>

            3. Fill out the following form items in order to create a product.
               • A) Title: Enter the name of the product
               • B) Description: Add a description of the product
               • C) Require subscription approval: Ensure to check this, as this will require any subscription requests to this product to be approved first
               • D) Press Save

               </li>

            4. Now that we have created our Product it is time to see if there are any other things we need to configure before we add policies to it and publish it. In order to check the settings by clicking on the newly created product

               </li>

            5. On the summary tab, click on the link ADD API TO PRODUCT

               </li>

            6. A new window will pop up, select the API you want to add to the product and once done click Save

               </li> </ol>

               Create a new Policy for your defined API operations

               At this point we have created a product but have not yet published it. We will publish it in a bit, but first we need to set up some policies for the API and the operation we’ve created earlier, In order to do this follow these steps

               1. From the menu bar on your left select the Policies item and in the main window in the policy scope section make the following selections
                  • A) For the API select the API you created earlier
                  • B) For the Operation select the operation you created earlier

               

               1. Now in the Policy definition section, click on ADD Policy

               

               1. At this point the Empty Policy definition is visible

               

               Setting up your Policy definition

               For our API operation to correctly function we are going to have to add a few policies. These policies should take care of the following functionality

               • Authenticate to our Service Bus Topic using our previously created SAS Token
               • Automatically convert potential JSON messages to their equivalent XML counterpart
               • Add some additional context information to the inbound message which are converted to brokered message properties when passed won to Azure Service Bus.

               General information on which policies are available to you within the Azure API Management Administration Portal and how to use them can be found here

               The next few steps will show you how we can add policy statements which will ensure the above mentioned functionality is added.

               1. In the Policy definition section, ensure to place your cursor after the <inbound> tag

                  </li>

               2. From the Policy statements, select and click on the “Set HTTP Header” statement.

                  </li>

               3. A “Set Header” Tag will be added to the Policy Definition area, which will leverage the Authorization Header containing our SAS token we created earlier. The steps required are listed below:
                  • A) Put in the value “Authorization” for the attribute “name”
                  • B) Put in the following value “skip” for the attribute “exists-action”
                  • C) Now get the SAS Token you created earlier, wrap the token string in between a CDATA element and put all of this in between the “value” element

                  

                  Textual example:

                  <set-header name=”Authorization“ exists-action=”skip“>

                   

                  <value><![CDATA[YOUR SAS TOKEN STRING]]></value>

                   

                  </set-header>

                   </li>

               4. Place your cursor just after the closing tag </set-header>
               5. From the Policy statements, select and click on the “Convert JSON to XML” statement.

                  </li>

               6. A “json-to-xml” Tag will be added to the Policy Definition area, which contains the instructions resulting in JSON messages to be converted to XML. Ensure that the tag is configured as mentioned below:
                  • A) Put in the value “content-type-json” for the attribute “apply”
                  • B) Put in the following value “false” for the attribute “consider-accept-header”

                  

                  Textual example:

                  <json-to-xml apply=”content-type-json“ consider-accept-header=”false“/>

                   

                   </li>

               7. Now add “set-header” policy statements adding the following headers

                  • A) Header name: MessageId

                     

                    • exists-action: “skip”
                    • value:“00000000-0000-0000-0000-000000000000” 

                  • B) Header name: MessageAction

                     

                    • exists-action: “skip”
                    • value=“Undefined” 

                  
                  textual example:

                  <set-header name=”MessageId“ exists-action=”skip“>

                   

                  <value>00000000-0000-0000-0000-000000000000</value>

                   

                  </set-header>

                   

                  <set-header name=”MessageAction“ exists-action=”skip“>

                   

                  <value>Undefined</value>

                   

                  </set-header>

                   

               8. Once you have added all the policy statements, press the save button
               9. 
               </ol>

               Create a Group, an User and Add subscribe the user to a product

               Now we have created a new product and assigned policies we need to perform some group/user related actions. This way we can set up a dedicated Group of users which is allowed to use our Product and it’s containing API.

               The steps below will guide you through this process

               1. Now select the Visibility tab, and click on the MANAGE GROUPS link

                  </li>

               2. You will be redirected to the GROUPS page, on this page click on the ADD GROUP Link

                  </li>

               3. A new windows will pop up, fill out the following fields
                  • A) Name: Unique name of the group
                  • B) Description: General description of the group and its purpose
                  • C) Click on Save

                  </li>

               4. After you’ve created the new Group, select the developers menu item and in the main window click on ADD User

                  </li>

               5. Once again a new window will pop up. In this window fill out the following fields:
                  • A) Email
                  • B) Password
                  • C) First and last name
                  • D) Press Save
               6. 
               7. Now that we have created a new user, we need to make it a member of our group. In order to do so follow the following steps
               8. • A) Ensure to select the new user
                  • B) Click on the ADD TO GROUP item and add the user to the earlier created Group

                  </li>

               9. Now go back to the PRODUCTS menu item and select the product you created earlier

                  </li>

               10. In the main window follow these steps
                   • A) Click on the Visibility Tab
                   • B) Allow the new group to subscribe to your product
                   • C) Click Save

                   </li>

               11. Now Click on the summary tab and click on the PUBLISH link

                   </li>

               12. Now select the Developers menu item and click on the user you created earlier

                   </li>

               13. The main window will now change, in this window click on ADD Subscription

                   </li>

               14. A window will pop up, in this window ensure to put a checkmark in front of the product you want the user to subscribe to. Once done press the subscribe button

                   </li> </ol>

                   Test your API

                   At this point you have set up your API and now you can proceed with testing it. In order to test we will use the Azure API Management Developer Portal and we will log-on to it using the user account we set up previously.

                   The steps below list the steps involved:

                   1. First log out of the Azure API Management Administration Portal
                   2. Now login using the email and password of the user you defined earlier

                      

                       </li>

                   3. In the top menu, select APIS

                      </li>

                   4. Click on the API you created earlier

                      </li>

                   5. Click on the button “Open Console”

                      </li>

                   6. A form will appear which allows you to send a message to the API. Follow the steps below to send a JSON formatted message to the API.
                      • A) From the dropdown select a subscription-key (used to authenticate to the API)
                      • B) Add two http headers
                        • Content-type: application/json [indicates that the message you are about to send is formatted as JSON]
                        • MessageAction: NonExisting [this will ensure that the message ends up in our Azure Service Bus subscription named Archive, as this subscription is our catch-all
                        • MessageId: 11111111-1111-1111-1111-111111111111
                        • MessageBatchId: SingleMessageID
                      • C) Add some sample JSON
                      • D) Press HTTP POST

                      </li>

                   7. Now open up the Service Bus Explorer and connect to your service bus instance
                   8. 
                   9. Right Click on the Archive subscription and select the option “Receive all messages”
                   10. 
                   11. One message should be received, which should meet the following test criteria:
                       • The message should be formatted in XML, although the original message we send was formatted as JSON
                         • The following custom message properties should be available
                         • MessageAction: NonExisting
                         • MessageId: 11111111-1111-1111-1111-111111111111
                         • MessageBatchId: SingleMessageID

                       

                       MY TEST RESULT: PASSED

                        

                   12. Now we will perform another test, but this time we will send an XML formatted message to the API.
                       • A) From the dropdown select a subscription-key (used to authenticate to the API)
                       • B) Add two http headers
                         • Content-type: application/xml [indicates that the message you are about to send is XML]
                         • MessageAction: BusinessActivityMonitoring [this will ensure that the message ends up in our Azure Service Bus subscription named BusinessActivityMonitoring and it will end up in our Archive subscription (as it is our catch-all)]
                       • C) Add some sample XML
                       • D) Press HTTP POST
                   13. 
                   14. Right Click on the BusinessActivityMonitoring subscription and select the option “Receive all messages”. One message should be received, which should meet the following test criteria:
                       • The message should be formatted in XML
                       • The following custom message properties should be available
                         • MessageAction: BusinessActivityMonitoring
                         • MessageId: 00000000-0000-0000-0000-00000000000
                         • MessageBatchId: SingleMessageID
                   15. 
                       

                       MY TEST RESULT: PASSED

                        

                   16. Now let’s receive all message from our Archive subscription (it should contain a copy of the previous message). Reason for this, is the fact that the archive subscription is our Catch-All subscription and thus all messages send to the topic end up in this subscription as well.

                        

                       

                       MY TEST RESULT: PASSED

                        

                       </li> </ol>

                       Takeaways

                       • Ensure to document your API well, this makes life easier for the consumers
                       • Using SAS Tokens you can fairly easy integrate fairly with Azure Service Bus Queues / Topics
                       • If using a policy to set Custom Headers (which you use for setting the Authorization header) ensure to enclose the SAS token within a <![CDATA[ …… ]]> tag
                       • The Azure API Management Key can be found on the developer page of the Azure API Developer Portal (f.e https://{namespace}.portal.azure-api.net/developer)
                       • Code examples on how to consume an API are available from the Azure API Developer Portal, by clicking on the menu item APIS and then clicking on the API you want to test
                       • Logging into the Azure API Management Administration Portal must be done using the Azure Management Portal
                       • Azure Management API, in my opinion, could easily be used to virtualize your on premise internet-faced web services (BizTalk generated web services f.e). This way you have one central place to govern and manage them.

                       Conclusion

                       I hope this walkthrough contributed in gaining a better understanding of how we as integrators can leverage the Azure API Management Service to expose Service Bus entities. Once you’ve grasped the concepts you could easily take it a step further and for example involve Azure BizTalk Services which would process messages from certain subscriptions, do some transformations and deliver it to for example another Azure Service Bus Topic, the topic endpoint could then be incorporated into a new API which would allow your API consumers to retrieve their processed messages.

                       Ah well you get the idea, the possibilities are almost endless as Azure delivers all these building-blocks (services) which enable you to create some amazing stuff for your customers.

                       I hope to publish a new post in the coming weeks; I’ve already worked out a scenario on paper which involves Azure API Management, Azure Service Bus, Azure BizTalk Services, Azure File Services, and an Azure Website; however implementing it and writing it down might take some time and currently my spare-time is a bit on the shallow side. Ah well, just stay tuned, check my Twitter and this blog.

                       Until next time!

                       Cheers

                       René