Part 15 - 429s, Throttling and How to Manage It

What is Throttling, a 429 Status code, how do we handle them, and create safe patterns for using OData Entities?

Throttling, in a general sense, is something acting upon another actor to actively hinder a process or request. Dynamics 365 For Finance and Operations recently added Priority-based Throttling. In the past, when hitting an OData endpoint, we only had to worry about non-affirmative timeouts; it took longer than 120 seconds to complete. It didn't specifically matter to me, the consumer of an endpoint, what else Finance and Ops may have going on. Now, with priority-based throttling, it is my problem as a consumer. I now get affirmative decline in the form of a 429 HTTP return code.

When does throttling occur?

I'd suggest you review the FAQ for Priority-Based Throttling. However, in my mind, the answers leave a lot to be desired and don't help me plan ahead to proactively prevent and manage throttling. At the time of publication, it has lots of great info but doesn't tell me what would cause a 429 to be sent. There is reference to "high resource utilization" but we're not given much more info on what that could mean and what part of the system may be under heavy load. 

What Do I do when Throttling / 429s Occur?

There are 2 pieces of a 429. They  are that you got a 429 error code that you have to respond to in some way and that if a 429 that gets repeated, it may infer an action needs to take place as a result. In simple terms, a 429 is a "please wait" request potentially an amount of time to wait. Additionally, if we're getting "please wait" requests repeatedly, maybe we should fail the request or thread and try again on the next integration run. A simple diagram of this process will look like this.

As you can see, there is a feedback loop we'll have to manage. We have to respond to the "please wait" 429 error but also prevent us from getting stuck in an infinite "please wait" loop.

Basic Pattern for How to Handle Throttling

In my mind, there are 2 basic patterns for handling throttling. They are very similar but have code in different places so we get either automatic behaviors or we have to code for explicit behaviors. I'm a fan of automatic behaviors using delegate method(s).

Automatic Behaviors using ReceivingResponseEvent

Using this idea, we add an event to an event handler on our connection to detect, trap and respond to a 429 error code. This is beneficial when we want all responses from an endpoint to have the same consistent logic. We simply have to declare the method then add it to our context. Below is an example:

        public static void ContextReceivingResponse(object sender, Microsoft.OData.Client.ReceivingResponseEventArgs e)
        {
            var response = e.ResponseMessage as HttpWebResponseMessage;
            var statusCode = response.StatusCode;
            var headers = response.Headers;

            Resources context = (Resources)sender;

            int retrySeconds = 5;

            if (statusCode == 429)
            {
                if (response.GetHeader("Retry-After") != "")
                {
                    var val = response.GetHeader("Retry-After");
                    retrySeconds = int.Parse(val);
                }

                Console.WriteLine("Caught 429, waiting {0}", retrySeconds);

                Thread.Sleep(TimeSpan.FromSeconds(retrySeconds));

            }

        }

To hook in this behavior, we'd simply need to add the method to the ReceivedResponse eventHandler as a delegate on our context, like so:

            context.BuildingRequest += (sender, e) =>
            {
                var uriBuilder = new UriBuilder(e.RequestUri);
                // Requires a reference to System.Web and .NET 4.6.1+
                var paramValues = HttpUtility.ParseQueryString(uriBuilder.Query);
                if (paramValues.GetValues("cross-company") == null)
                {
                    paramValues.Add("cross-company", "true");
                    uriBuilder.Query = paramValues.ToString();
                    e.RequestUri = uriBuilder.Uri;
                }
            };

            context.SendingRequest2 += new EventHandler<SendingRequest2EventArgs>(delegate (object sender, SendingRequest2EventArgs e)
            {
                var authenticationHeader = OAuthHelper.GetAuthenticationHeader(useWebAppAuthentication: true);
                e.RequestMessage.SetHeader(OAuthHelper.OAuthHeader, authenticationHeader);
            });

            context.ReceivingResponse += ContextReceivingResponse;

With this, we'll always detect, catch, and apply a 429 "please wait" message and apply Retry-After if specified from the AOS. Otherwise,  we'll wait 5 seconds; our minimum seconds to wait when we get a 429.

Specific Error Handling per Request

You can create specific error handling for requests. This would need to be implemented for each query but give you greater control over the error conditions you are watching.

        public static void runOneReadCustomThrottling(Resources context, string filePath, TestType testType, TestWorkload testWorkload, string SalesOrderNumber, string DataAreaId)
        {
            void ContextReceivingResponse(object sender, Microsoft.OData.Client.ReceivingResponseEventArgs e)
            {
                var response = e.ResponseMessage as HttpWebResponseMessage;
                var statusCode = response.StatusCode;
                var headers = response.Headers;

                int retrySeconds = 5;

                if (statusCode == 429)
                {
                    if (response.GetHeader("Retry-After") != "")
                    {
                        var val = response.GetHeader("Retry-After");
                        retrySeconds = int.Parse(val);
                    }

                    Console.WriteLine("Caught 429, waiting {0}", retrySeconds);

                    Thread.Sleep(TimeSpan.FromSeconds(retrySeconds));

                }

            }

            int tryCount = 0;
            context.ReceivingResponse += ContextReceivingResponse;

            while (true)
            {
                try
                {
                    tryCount++;
                    SalesOrderHeaderV2 SalesOrderHeaderV2 = context.SalesOrderHeadersV2.Where(x => x.SalesOrderNumber == SalesOrderNumber && x.dataAreaId == DataAreaId).First();

                    break;

                }
                catch (Exception e)
                {
                    if (tryCount >= 3)
                    {
                        throw (e);
                    }
                }
            }
        }

Read Pattern

Below is a code example of a pattern I've been using. Specifically, I decided to go with a while(true) pattern as I have 2 potential outcomes for this method and I want to reach either one of them. A While(tryCount < [some value] ) didn't feel right.

            int tryCount = 0;
            Stopwatch sw = new Stopwatch();

            sw.Start();

            while (true)
            {
                try
                {
                    tryCount++;
                    SalesOrderHeaderV2 SalesOrderHeaderV2 = context.SalesOrderHeadersV2.Where(x => x.SalesOrderNumber == SalesOrderNumber && x.dataAreaId == DataAreaId).First();

                    break;

                }
                catch (Exception e)
                {
                    if (tryCount >= 3)
                    {
                        throw (e);
                    }
                    sw.Reset();
                    sw.Start();
                }
            }

Insert Pattern

Similar to the Read Pattern, we're doing something very similar for an insert. Note we're declaring our DataServiceCollection outside the while loop.

            int tryCount = 0;
            DataServiceCollection<SalesOrderHeaderV2> SalesOrderCollection = new DataServiceCollection<SalesOrderHeaderV2>(context);
            SalesOrderHeaderV2 salesOrderHeaderV2 = new SalesOrderHeaderV2();

            while (true)
            {
                try
                {
                    tryCount++;
                    SalesOrderCollection = new DataServiceCollection<SalesOrderHeaderV2>(context);
                    salesOrderHeaderV2 = new SalesOrderHeaderV2();

                    SalesOrderCollection.Add(salesOrderHeaderV2);

                    // Required Fields
                    salesOrderHeaderV2.OrderingCustomerAccountNumber = customerAccount;
                    salesOrderHeaderV2.InvoiceCustomerAccountNumber = customerAccount;
                    salesOrderHeaderV2.dataAreaId = DataAreaId;
                    salesOrderHeaderV2.CurrencyCode = "USD";
                    salesOrderHeaderV2.LanguageId = "en-us";

                    context.SaveChanges(SaveChangesOptions.PostOnlySetProperties | SaveChangesOptions.BatchWithSingleChangeset);

                    break;

                }
                catch (Exception e)
                {
                    if (tryCount >= 3)
                    {
                        throw (e);
                    }
                }
            }

Update Pattern

The update pattern is a little different as we have to get our result before we can update it.

            int tryCount = 0;
            SalesOrderHeaderV2 SalesOrderHeaderV2;

            while (true)
            {
                try
                {
                    tryCount++;
                    context.SalesOrderHeadersV2.FirstOrDefault();

                    SalesOrderHeaderV2 = context.SalesOrderHeadersV2.Where(x => x.SalesOrderNumber == SalesOrderNumber && x.dataAreaId == DataAreaId).First();
                    break;

                }
                catch (Exception e)
                {
                    if (tryCount >= 3)
                    {
                        throw (e);
                    }
                }
            }

            tryCount = 0;

            if (SalesOrderHeaderV2.SalesOrderNumber != null)
            {
                DataServiceCollection<SalesOrderHeaderV2> SalesOrderCollection = new DataServiceCollection<SalesOrderHeaderV2>(context);

                while (true)
                {
                    try
                    {
                        tryCount++;
                        string salesURL = SalesOrderHeaderV2.URL;

                        // replace whatever is there with this trash
                        salesURL = "http://www." + Guid.NewGuid().ToString() + ".com";

                        sw.Start();
                        SalesOrderCollection.Add(SalesOrderHeaderV2);

                        SalesOrderHeaderV2.URL = salesURL;
                        sw.Start();
                        context.SaveChanges(SaveChangesOptions.PostOnlySetProperties | SaveChangesOptions.BatchWithSingleChangeset);

                        break;

                    }
                    catch (Exception e)
                    {
                        if (tryCount >= 3)
                        {
                            throw (e);
                        }
                    }
                }
            }

Delete Pattern

The delete pattern is similar to a read pattern with the addition of a delete. I didn't think it made sense to add the fetch in another While loop as we need to find and delete a specific record in a change set. I can imagine a scenario where we have cascading concerns but I think the likelihood of it ever happening in the real world is basically zero.

            int tryCount = 0;
            string SalesOrderNumber;
            SalesOrderHeaderV2EntityOnlySalesTable SalesOrderHeaderV2EntityOnlySalesTable;
            DataServiceCollection<SalesOrderHeaderV2EntityOnlySalesTable> SalesOrderCollection = new DataServiceCollection<SalesOrderHeaderV2EntityOnlySalesTable>(context);

            while(true)
            {
                try
                {
                    var SalesOrderHeadersV2EntityOnlySalesTable = context.SalesOrderHeadersV2EntityOnlySalesTable.Where(x => x.SalesOrderNumber == SalesOrderNumber && x.dataAreaId == DataAreaId).First();

                    context.DeleteObject(SalesOrderHeadersV2EntityOnlySalesTable);
                    context.SaveChanges(SaveChangesOptions.PostOnlySetProperties | SaveChangesOptions.BatchWithSingleChangeset);
                    break;
                }
                catch (Exception e)
                {
                    if (tryCount >= 3)
                    {
                        throw (e);
                    }
                }
            }

Testing

Brad Bateman wrote a great article on how to create and test for handling of 429 error codes.

All code for this article can be found at https://github.com/NathanClouseAX/AAXDataEntityPerfTest/tree/main/Projects/AAXDataEntityPerfTest/Part15.