Azure Durable Functions with retry options and retry operations

This is the third post of a serious of articles on Azure Functions/Durable functions. This is the continuation of my previous post on Durable Functions.

• Posting data to Azure Function using HTTP trigger with .NET 6
• Azure Durable Functions to implement function chaining with custom data
• Azure Durable Functions with retry options and retry operations

In this post we will discuss how we handle failures, specifically retry operations with Durable Functions.

Ability to handle failures is a key benefit that Durable Functions offer us. It is implemented using features of Durable Task Framework.

In order to illustrate retry operations we use the same example we used in previous article. Let's say saving contact details in CRM system is a heavy process and it is not under our control. Let's say it is a SaaS platform provided by a vendor. It can fail due to various reasons like

• Network related failures
• Planned/unplanned downtimes of the SaaS platform
• etc..

Let's implement the retry pattern

Step 1: Instantiate RetryOptions object in our Orchestrator function. We have to specify retry interval and maximum retry attempts

We will filter out the exception and I will retry only when there is a network error. And we will retry three attempts

var retryOptions = new RetryOptions(System.TimeSpan.FromSeconds(5), 3)
{
    Handle = ex => ex.Message.Contains("Network Error")
};

Step 2: Instead of using CallActivityAsync function we will use CallActivityWithRetryAsync function. Following is the implementation of orchestration function after using retry options

[FunctionName("DurableCustomer")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
    Customer customer =  context.GetInput();


    int customerId = await context.CallActivityAsync("DurableCustomer_AddToSQL", customer);


    var retryOptions = new RetryOptions(System.TimeSpan.FromSeconds(5), 3)
    {
        Handle = ex => ex.Message.Contains("Network Error")
    };

    int returnId = await context.CallActivityWithRetryAsync("DurableCustomer_AddCRM",retryOptions,customerId);

    return returnId;
}

Step 3: Let's change the AddCRM function a little bit and generate a custom error

[FunctionName("DurableCustomer_AddCRM")]
public static int AddToCRM([ActivityTrigger] int customerId, ILogger log)
{
    //insert to CRM
    int returnId = customerId * 10; //return code from CRM insert
    throw new System.Exception("Network Error");
    //return returnId;
}

Step 4: Let's deploy and check in postman. In the same time we will navigate to Monitoring -> Log Stream section of the Function App in Azure portal.

Let's check log stream. Look, I got four error messages. Original error and three attempts just after that

Durable Functions have other error handling mechanisms as well. We have discussed only the retry pattern in this article.

Hope this helps :)

Update 16-02-2022 : I've uploaded the code to this repository