BatchRetryable Top Picking Batch Job Example

06/29/24

BatchRetryable

I recently presented at DynamicsMinds on batch jobs and how to design them for specific types of workloads. There was interest in code samples so this in one of those code samples talking about Top Picking with a review. Top picking is a method of creating a list of work that is stored in a table then "picking" from the "top" of it. You can think of it as a queue with mutliple consumers popping stuff from the queue to process. We're not using a real queue but we're getting very similar results. In prior articles, I used an analogy of a highway. While other batch job design methods maybe make use of the different lanes on the highway in different ways by used some kind of metering system, top picking can take a lane on the highway and temporarily dedicate it to all "traffic" to a specific batch job. This means if you have a top picking batch job with 4 batch tasks, it will use 4 threads ( of your standard available 8 ) to processing all pending work items in your work items table. This can greatly increase the throughput for work items in a batch task but that dedicates specific batch capacity to that throughput until all of the pending work items in the table are processed. This is performance for one group of work at the potential expense of other batch tasks waiting longer. Because we have only 1 batch task that essentially repeats itself, we don't incur any extra batch framework overhead such as determining if the batch task has met its conditions for execution. Using the highway analogy again, top picking is taking x number of lanes on the highway and dedicating them to our tasks - but those tasks in the lanes dedicated to them can go as fast as possible. They have no specific constraints other than the amount of work they have to do. Those tasks have no speed limit. They can "put the pedal to the metal"

Data Contract

For Top Picking, there is no specific contract required. These will use 1 or more threads to process work from a specific table of pending work. In a way, the data contract is the table containing the work items to process. Before the batch tasks can start to execute, the data in this table needs to exist so you'll have to plan ahead for how this table get's its data. Look at tables CustVendGeneralTopPickingHeader and CustVendGeneralTopPickingItems for a more robust solution.

Controller

Next, we have the controller. This is nearly identical to any other controller. It links the execution controller to the service.

class AAXBatchTopPickingController extends SysOperationServiceController
{
public static SysOperationController construct(Args _args = null)
{
SysOperationController controller = new AAXBatchTopPickingController(
classStr(AAXBatchTopPickingService),
methodStr(AAXBatchTopPickingService, process));

controller.parmArgs(_args);

return controller;
}

public static void main(Args _args)
{
AAXBatchTopPickingController::construct(_args).startOperation();
}

/// <summary>
///
/// </summary>
/// <returns></returns>
public ClassDescription caption()
{
return "Top Picking Task Example";
}

}

Service

Lastly, we have the service. For this style of batch task, it has a few defining characteristics. first, we're using a do-while loop which is something I rarely see in x++ for data processing so the loop will execute at least once. Next, we're specifying on one of our buffers that the readPast() value be set to true. You can also do this as part of the select if you'd like. The effect this has is if a record is locked, this instructs the database abstraction layer to go past that locked record getting rid of any wait time that could exist or happen waiting for a locked record. Next, we're using a firstonly and pessimisticlock as part of the select. What this will do is select the first record and place a pessimistic lock on it - meaning the record is locked in the database for any CRUD operation from any other process until the transaction closes that placed the pessimistic lock. This allows 1 thread to lock a record for use then another thread to read past that lock to get the next record. Lastly, we're placing the entire contents of the do-while loop in a transaction. This is important because we could have multiple actors / threads trying to do the same thing so this will open a transaction, lock a record for use for 1 thread then the next thread that tries to select a record will read past the already locked record - making the magic of top picking work. Also, this implements BatchRetryable so we need to design the task to fail completely or be succesful complete. Again, if the task fails for any reason, why it failed gets wiped from the logs and the task is simply tried again. In this style of batch job, if we have 50 iterations be successful, then 1 fail and it retry, any logged values in InfoLog would be removed for the first 50 items. Each retry clears the InfoLog.

public class AAXBatchTopPickingService extends SysOperationServiceBase implements BatchRetryable
{
public void process()
{
AAXBatchTopPickJobWorkStagingTable AAXBatchTopPickJobWorkStagingTable;

AAXBatchTopPickJobWorkStagingTable.readPast(true); // skip locked records

do
{
ttsbegin;
select firstonly pessimisticlock Processed, RefRecId, RecId from AAXBatchTopPickJobWorkStagingTable
where AAXBatchTopPickJobWorkStagingTable.Processed == NoYes::No;

if (AAXBatchTopPickJobWorkStagingTable)
{

AAXBatchTopPickWorkItem::processWorkItem(AAXBatchTopPickJobWorkStagingTable.RefRecId);

AAXBatchTopPickJobWorkStagingTable.Processed = NoYes::Yes;
AAXBatchTopPickJobWorkStagingTable.update();

}
ttscommit;
}
while (AAXBatchTopPickJobWorkStagingTable);
}

public boolean isRetryable()
{
return true;
}

}

Work Item Processor

This is the class with the encapsulated logic to perform the actual process we've turned into top-picked batch task.

internal final class AAXBatchTopPickWorkItem
{
public static void processWorkItem(RecId _recId)
{
AAXBatchTopPickJobWorkItemTable AAXBatchTopPickJobWorkItemTable;

select forupdate Id, Updated, RecId from AAXBatchTopPickJobWorkItemTable
where AAXBatchTopPickJobWorkItemTable.RecId == _recId
&& AAXBatchTopPickJobWorkItemTable.Updated == NoYes::No;

if(AAXBatchTopPickJobWorkItemTable)
{
ttsbegin;
AAXBatchTopPickJobWorkItemTable.Updated = NoYes::Yes;
AAXBatchTopPickJobWorkItemTable.update();
ttscommit;
}
}

}