How to Obtain an Access Token for Azure Cost Management API Calls

In this article I will explain how to obtain an Access Token which will be used when calling Azure Cost Management APIs.

Following were the steps I used

Create an App Registration and Note the Tenant ID and Client ID

Next, generate a client secret and store it securely.

Our app registration is now successful. Next, we need to assign the service principal (app registration) the appropriate permissions to access cost data from our subscription.

Go to your Subscription, navigate to the Access Control (IAM) section, and click on 'Add role assignment.

We will assign the Cost Management Reader permission

Next, select the App registration we created and complete the role assignment process.

That's all we have to do within Azure portal. Let's construct our API request using Postman client

#URL
https://login.microsoftonline.com/{Tenant ID}/oauth2/v2.0/token

#METHOD
POST

#X-WWW-FORM-URLENCODED
#X-WWW-FORM-URLENCODED
grant_type=client_credentials
&client_id={Client ID}
&client_secret={Client Secret}
&scope=https://management.azure.com/.default

Following is the response I get with the access token

Optimizing Data Retention for Specific Tables in Azure Log Analytics Workspace

Application Insights plays a crucial role in applications hosted on Microsoft Azure. It is important to periodically review the associated Log Analytics Workspace instances to ensure they are properly governed and optimized.

The following areas should be examined for these resources in alignment with the Azure Well-Architected Framework.

• Reliability
• Security
• Cost Optimization
• Operational Excellence
• Performance Efficiency

The following analysis aligns with Cost Optimization, Operational Excellence, and Performance Efficiency pillars of the well architected framework.

Recently, I noticed an increase in app usage for a client, leading to higher log volumes. 

We can easily analyze this pattern by navigating to the Log Analytics Workspace and checking the Usage & Estimated Costs section.

It is evident that AppTraces have increased recently, which may negatively impact overall query performance.

My Log Analytics Workspace is set to the default 90-day data retention period, which I prefer to keep unchanged to achieve cost optimization.

Since the outlier is only in the AppTraces table, I want to reduce the retention period specifically for that table. The expectation is to maintain a lower data volume in that container while keeping the overall workspace retention unchanged.

Following was the approach I used.

Navigate to Tables in the Log Analytics Workspace and select the AppTraces table.

Click on Manage Table to access the table settings.

Change the retention period to 60 days, which overrides the default retention setting for this table only.

That's it! The AppTraces table will now retain data for 60 days, reducing log volume while keeping the overall workspace retention unchanged.

Let's consider the opposite approach: How can you increase data retention while still optimizing costs?

You can also use this approach to optimize costs. For example, if you need to increase the retention period from 90 to 120 days, instead of applying the change to all tables, you can selectively extend retention only for the tables that truly require it. This helps balance cost efficiency and data availability.

Additional retention will incur costs based on the amount of storage consumed.

By strategically planning data retention for specific tables, you can effectively optimize costs.

Using Azure Cost Management API to Analyze Resource Group Expenditures by Cost Unit

Azure Cost Management offers various tools to track expenses and optimize costs across our tenants. Following are some features provided by Azure Cost Management

• Cost Analysis
• Budgets
• Cost Alerts
• Cost Allocation
• Exports
• Recommendations
• Multi-Cloud Support
• Governance and Accountability
• Integration with Power BI
• API

In this short article, I'll explain how to view expenditures for a specific resource group, broken down by the various meters Azure uses for cost calculation.

By analyzing the output, we can gain insights into how Azure calculates costs for our resources, ensuring there are no surprises when the bill arrives.

One way to achieve above is by using Cost Management, specifically within the Cost Analysis section.

However, I sometimes prefer using the Cost Management API over the Cost Analysis section because it offers greater flexibility and allows me to customize parameters to better achieve my objectives.

Following is the API I used

Request

#URL
https://management.azure.com/subscriptions/{Tenant ID}/resourceGroups/{Resource Group}/providers/Microsoft.CostManagement/query?api-version=2024-08-01

#METHOD
POST

#AUTHORIZATION
Bearer Token

#BODY
{
  "type": "Usage",
  "timeframe": "MonthToDate",
  "dataset": {
    "granularity": "None",
    "aggregation": {
      "totalCost": {
        "name": "PreTaxCost",
        "function": "Sum"
      }
    },
    "grouping": [
      {
        "type": "Dimension",
        "name": "ResourceType"
      },
      {
        "type": "Dimension",
        "name": "MeterCategory"
      }
    ]
  }
}

You can use this approach to generate an access token

Response

{
    "id": "subscriptions/{Tenant ID}/resourcegroups/{Resource Group}/providers/Microsoft.CostManagement/query/5c25de9e-06aa-4839-81b4-64273e0bc86f",
    "name": "5c25de9e-06aa-4839-81b4-64273e0bc86f",
    "type": "Microsoft.CostManagement/query",
    "location": null,
    "sku": null,
    "eTag": null,
    "properties": {
        "nextLink": null,
        "columns": [
            {
                "name": "PreTaxCost",
                "type": "Number"
            },
            {
                "name": "ResourceType",
                "type": "String"
            },
            {
                "name": "MeterCategory",
                "type": "String"
            },
            {
                "name": "Currency",
                "type": "String"
            }
        ],
        "rows": [
            [
                34.957915,
                "microsoft.apimanagement/service",
                "API Management",
                "AUD"
            ],
            [
                0.0016342144,
                "microsoft.keyvault/vaults",
                "Key Vault",
                "AUD"
            ],
            [
                0.036166705555556,
                "microsoft.loadtestservice/loadtests",
                "Azure Load Testing",
                "AUD"
            ],
            [
                0.034666562722667,
                "microsoft.operationalinsights/workspaces",
                "Log Analytics",
                "AUD"
            ],
            [
                0.000007542612,
                "microsoft.storage/storageaccounts",
                "Bandwidth",
                "AUD"
            ],
            [
                0.0001935796,
                "microsoft.storage/storageaccounts",
                "Storage",
                "AUD"
            ],
            [
                0.0,
                "microsoft.web/sites",
                "Azure App Service",
                "AUD"
            ],
            [
                0.000000359172,
                "microsoft.web/sites",
                "Bandwidth",
                "AUD"
            ]
        ]
    }
}

With these insights, I can identify the various meters used for my resources and see the expenditure associated with each meter.

Mocking Custom Responses with Azure API Management – Custom Mock Response from External Files

This article is the third part of a multi-part series. In this section, I will explain how to render custom mock responses from JSON files stored in a storage container. Below are the different parts of this article series.

• Mocking Custom Responses with Azure API Management – Simple Mock Response
• Mocking Custom Responses with Azure API Management – Custom Mock Response
• Mocking Custom Responses with Azure API Management – Custom Mock Response from External Files

As discussed in the second part of this series, we can embed JSON responses directly within the policy. However, this approach can become messy and difficult to modify or maintain over time if you have many mock scenarios.

Custom mock responses from external files

As an improvement, we can store mock responses externally and retrieve them based on the request using APIM policies. An added advantage is that we can better organize our mock responses in external storage, where we can leverage features like versioning, enhanced availability, and more efficient management.

Here’s the approach I used:

I provisioned an Azure Storage Account and created a Blob Container. Then, I organized my mock responses within a dedicated folder inside the container.

Following is the content of a JSON file

{
    "id": 1,
    "name": "Introduction to Azure",
    "category": "Cloud Computing",
    "description": "A beginner-friendly course covering the fundamentals of Microsoft Azure.",
    "duration": "4 weeks",
    "instructor": "John Doe"
}

Next, I generated a Shared Access Signature (SAS) token with Read permissions and obtained the Blob SAS URL to securely access the mock responses.

Then, I navigated to the APIM instance, selected the Inbound Processing section, and opened the Policy Editor.

Following is the extract of the policy

<inbound>
        <base />
        <choose>
            <when condition="@(context.Request.Url.Query.GetValueOrDefault("id") == "1")">
                <send-request mode="new" response-variable-name="response" timeout="10" ignore-error="false">
                    <set-url>@($"https://rgstoragefedora001.blob.core.windows.net/data/Subject/1.json?sp=r&st=2024-03-08T12:01:42Z&se=2024-03-08T20:01:42Z&spr=https&sv=2022-11-02&sr=b&sig=DrqWZJxGP38PNJYFKCoMgIqn6AjNC71nw0x%2FjITu4aM%3D")</set-url>
                    <set-method>GET</set-method>
                </send-request>
                <return-response>
                    <set-status code="200" reason="OK" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>@(new JObject(((IResponse)context.Variables["response"]).Body.As<JObject>()).ToString())</set-body>
                </return-response>
            </when>
            <when condition="@(context.Request.Url.Query.GetValueOrDefault("id") == "2")">
                <send-request mode="new" response-variable-name="response" timeout="10" ignore-error="false">
                    <set-url>@($"https://rgstoragefedora001.blob.core.windows.net/data/Subject/2.json?sp=r&st=2024-03-08T12:34:15Z&se=2024-03-08T20:34:15Z&spr=https&sv=2022-11-02&sr=b&sig=OVhf3uGIASMHWHtg8F%2BypQzLatI9DeEzqoFns2iGOUk%3D")</set-url>
                    <set-method>GET</set-method>
                </send-request>
                <return-response>
                    <set-status code="200" reason="OK" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>@(new JObject(((IResponse)context.Variables["response"]).Body.As<JObject>()).ToString())</set-body>
                </return-response>
            </when>
            <otherwise>
                <return-response>
                    <set-status code="404" reason="Not Found" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>{
                            "error": "Student ID not found"
                        }</set-body>
                </return-response>
            </otherwise>
        </choose>
    </inbound>

That's all we need to configure within the policy. Next, I navigated to the Test console to verify the response. I have to specify the query string as below

Following is the response I got as expected

Mocking Custom Responses with Azure API Management – Custom Mock Response

This article is the second part of a three-part series. We will discuss how to render custom mock responses using APIM policy. Below are the different parts of this article series.

• Mocking Custom Responses with Azure API Management – Simple Mock Response
• Mocking Custom Responses with Azure API Management – Custom Mock Response
• Mocking Custom Responses with Azure API Management – Custom Mock Response from External Files

Custom mock responses

Often, we need more than just a 200 OK response without a body. Instead, we require comprehensive responses formatted as JSON messages. Adding to the complexity, the response often needs to vary based on specific query string parameters.

Here’s the approach I used to generate custom mock responses in Azure API Management based on varying query string parameters.

Navigate to your API Management instance and select the specific API operation you want to configure.

Click on the Inbound Processing section and open the Policy Code Editor.

We will modify the inbound section of the policy. We will add a policy segment to dynamically generate the response body based on a specified query string parameter.

Here is the code I used

<inbound>
        <base />
        <choose>
            <when condition="@(context.Request.Url.Query.GetValueOrDefault("id") == "1")">
                <return-response>
                    <set-status code="200" reason="OK" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>{
                        "studentId": "1",
                        "name": "Jane Smith",
                        "grade": "B"
                    }</set-body>
                </return-response>
            </when>
            <when condition="@(context.Request.Url.Query.GetValueOrDefault("id") == "2")">
                <return-response>
                    <set-status code="200" reason="OK" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>{
                        "studentId": "2",
                        "name": "Jane Smith",
                        "grade": "B",
                    }</set-body>
                </return-response>
            </when>
            <otherwise>
                <return-response>
                    <set-status code="404" reason="Not Found" />
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/json</value>
                    </set-header>
                    <set-body>{
                            "error": "Student ID not found"
                        }</set-body>
                </return-response>
            </otherwise>
        </choose>
    </inbound>

You can test the outcome by navigating to the Test console and specifying the appropriate query string parameter, as shown below.

Then, submit the request to verify that the appropriate response is returned based on the specified query string parameter.

Ensuring Static IP for Azure App Service When Accessing External APIs Over the Internet

Azure App Service assigns a range of outbound IPs when accessing external resources. However, if the external resource requires IP whitelisting, the default configuration may not be practical. This article outlines the steps to ensure that the external API is accessed using a static public IP.

Following is the default configuration. With this setting, the external API may be called with any of the IPs within the range

To achieve this objective, I chose to use NAT Gateway and related components. we need to complete the following tasks in order to implement the solution:

• Integrate the Web App with a Subnet in a Virtual Network
• Create a Public IP
• Create and Configure a NAT Gateway
• Associate the Web App with the NAT Gateway
• Test

If we had the external API or resource within a private network (e.g On-Premises) we could've used Hybrid Connections.

Let's discuss the implementation of each item

1. Integrate the Web App with a Subnet in a Virtual Network

Create a virtual network and a subnet or consume an existing virtual network

Next, navigate to your Web App, go to the Networking section, and enable Virtual Network Integration to connect it to the designated subnet.

2. Create a Public IP

3.Create and Configure a NAT Gateway

Specify Outbound IP. We can specify multiple public IP addresses if we want once NAT Gateway is configured

Specify the Subnet

4.Associate the Web App with the NAT Gateway

Since resources are within the same network, the NAT Gateway will be automatically configured with your Web App

5. Test

Let's test the solution. To validate the setup, I deployed a sample .NET API that calls an external service, which returns the calling IP address. 

    [ApiController]
    [Route("api/[controller]")]
    public class GatewayTestController : ControllerBase
    {
        private readonly HttpClient _httpClient;
        public GatewayTestController(HttpClient httpClient)
        {
            _httpClient = httpClient;
        }

        [HttpGet]
        public async Task GetOutboundIp()
        {
            // Call external API over internet
            var response = await _httpClient.GetAsync("https://httpbin.org/ip");
            if (!response.IsSuccessStatusCode)
            {
                return StatusCode((int)response.StatusCode, "Failed to get outbound IP");
            }

            var callerIP = await response.Content.ReadAsStringAsync();
            return Ok(callerIP);
        }
    }

Following is the response I get. This matches exactly with the Public IP I provisioned

Structured Approach to a Successful Azure Integration Services Implementation

Implementation of integration solutions using Azure Integration Services (AIS) components is quite common. It can harness the benefits of an Integration Platform as a Service (iPaaS). While the integration process may appear straightforward, it requires careful governance. Typically, the implementation timeline is tight, and the presence of multiple moving parts adds to the complexity of the process.

The following activities are recommended for a successful AIS engagement:

Planning & Discovery

• Project Plan Development: Establish the project’s scope, objectives, timelines, budget, and deliverables. This foundational task involves identifying stakeholder requirements and defining a roadmap for successful implementation.

• Discovery Workshop(s): Engage stakeholders in collaborative sessions to gather requirements, align expectations, and document business processes to ensure the integration aligns with organizational goals.

Deliverables: Project Plan Document, Discovery Workshop Artifacts, Open Communication Channel (e.g MS Teams)

Design

• Develop High-level Architecture: Define the overall architectural vision, including system components and their interactions, to serve as the blueprint for detailed designs.

• Develop Low-level Architecture: Provide a granular view of individual components, interfaces, and data flows to guide development and implementation teams.

• Design Documentation: Compile detailed design specifications, architectural diagrams, and decision logs to serve as reference material for all stakeholders.

• Design Walk-Through Workshop(s): Present the architecture and design plans to stakeholders for validation, feedback, and alignment before moving to implementation.

• Proof of Concept: Build a small-scale prototype to validate critical components, test assumptions, and mitigate risks before full-scale development begins.

Deliverables: High-level Architecture Document, Low-level Architecture Document, Consolidated Design Document, Workshop Notes, Proof of Concept

Implementation

• Integration Environment Readiness: Set up environments, such as staging and testing platforms, to support development and ensure readiness for system integration. It is essential to ensure that both source and target environments are prepared for implementation. If the environments are in different locations (e.g., cloud vs. on-premises), the team must exercise extra caution regarding security and access management.

• Landing Zone Configuration: Establish a secure and scalable Azure foundation, ensuring compliance with best practices for networking, identity, and governance.

• DevOps Environment Configuration: Set up CI/CD pipelines, repositories, and automated workflows to streamline development and deployment processes.

• Specific AIS Component Configuration: Configure Azure Integration Services components like Logic Apps, Service Bus, and API Management as per the design.

• Development of Components: Build custom integrations, workflows, and connectors required for the solution based on detailed designs. Most of the development will be conducted using Azure Functions and Logic App workflows. It is crucial to follow a well-structured solution design that incorporates appropriate cloud components. Additionally, adopting a "shift-left" approach to security is highly recommended to address potential risks early in the development lifecycle

• Implement Disaster Recovery & High Availability: Establish redundancy, failover mechanisms, and recovery processes to ensure resilience and minimal downtime.

• Data Migration: Plan, map, and execute the migration of legacy data to the new system, ensuring data integrity and minimal disruption.

Deliverables: Integration Environment Readiness Confirmation, Landing Zone Configuration Document, DevOps Environment Configuration Document, AIS Component Configuration in Infrastructure as Code (IaC) in repository , Developed Components in repository, Disaster Recovery & High Availability Plans, Data Migration Report

Testing

• Unit Testing: Validate individual components or modules to ensure they meet functional requirements and perform as expected.

• Integration Testing: Assess the system as a whole to verify that all components communicate and function seamlessly together.

• Vulnerability Assessment & Penetration Testing: Identify security vulnerabilities and test the system’s ability to withstand potential threats or breaches.

• User Acceptance Testing: Engage end-users to validate the system against business requirements and ensure it meets their needs.

Deliverables: Unit Testing Report, Integration Testing Report, Security Testing Report, User Acceptance Testing Deliverables

Deployment

• Configure CI/CD Pipelines: Finalize and optimize pipelines to automate deployment processes, enhancing efficiency and consistency.

• Deployment to Non-Prod Environments: Deploy the solution to development and staging environments for final validation and testing.

• Deployment to Prod Environments: Execute the live deployment of the solution in production, ensuring minimal downtime and proper configuration.

• Post-Deployment Verification: Validate the deployment success by verifying all functionalities, data, and integrations are working as intended.

Deliverables: CI/CD Pipeline Configuration Files, Non-Production Deployment Validation, Production Deployment Plan, Post-Deployment Verification Report

Closure

• Establish Monitoring & Logging: Configure monitoring tools and logging mechanisms to ensure ongoing visibility into system performance and health.

• Issue Resolution: Address any post-deployment issues or bugs, ensuring the system operates smoothly and meets expectations.

• Project Closure Documentation: Prepare final documentation, including lessons learned, handover details, and user manuals, to conclude the project formally.

Deliverables: Monitoring & Logging Configurations, Issue Resolution Logs, Project Closure Documentation