BatchRetryable Individual Task Modeled Batch Job Example

06/14/24

BatchRetryable With lots of Individual tasks from 1 batch job

I recently presented at DynamicsMinds on batch jobs and how to design them for specific types of workloads. There was interest in code samples so this in one of those code samples talking about Individual task modeling with a review. Individual task modeling is an approach in task management and scheduling where each work item or unit of work is represented as a separate, individual task. This means that instead of grouping multiple work items into a single task, each work item gets its own distinct task. What this means when modeling tasks, we will create a task for each work item, resulting in a 1:1 mapping between tasks and work items. This means if we have x tasks to complete, we create a batch task for for each task we want to complete. This method eliminates the need for pre-allocation meaning we don't have to set aside or allocate time, memory, CPU, or any other type of resource for the batch task execution. As batch capacity becomes available for a task, that task gets executed. You can think of batch capacity like a highway and our tasks are being allowed onto that highway to get to their destination. If there are no other cars on the highway, having lots of cars suddenly get on the highway isn't a big deal. If the highway is busy, some cars may have to wait for a bit while space / capacity on the highway comes available from cars moving ( or other batch tasks completing ). Batch capacity has several pressure metrics, switches, and levers that it uses to determine what should run next so it's important to think of batch in terms of capacity rather than threads because of several metrics around what should run next. Is something "critical"? Does it have a specific time window it can execute it? Those are factors in what tasks get executed next. So just think of it like a highway with some normal lanes, a carpool lane and the occasional emergency vehicle that gets priority above other cars. With each work item independently managed by a worker thread, workload distribution becomes more consistent because as capacity is available, it gets temporarily used by 1 of the tasks for the 1 of x tasks on our list of things to do. This strategy prevents the issue of large work items being grouped together and negatively impacting batch response time. However, if you're processing a high number of work items, you'll end up with an equally vast number of batch tasks. The overhead for the batch framework to manage this enormous volume of tasks will be substantial and should be a factor in selecting a batch job style for your specific scenario. The batch framework must check multiple conditions, dependencies, and constraints each time a task is considered for execution and also completed and a new item needs to be selected for execution from the ready state. You can read more on capacity here. We'll jump into code using the SysOperationService set of constructs. That means we will have a data contract, a service, and a controller. We'll review each below.

Data Contract

This is an example of the data contract class. We are passing in the RecId for the record we want to process as well and a NoYes value to simulate an unplanned error just to see what will happen. This simulates of the database becomes unavailable for a split second. Otherwise, include any data you want to be passed into the execution of your task.

[DataContractAttribute]
class AAXBatchIndividualTaskDataContract
{
int64 RecId;
int64 workItemCount;
NoYes throwError;

[DataMemberAttribute, SysOperationControlVisibilityAttribute(false)]
public int64 parmRecId(int64 _RecId = RecId)
{
RecId = _RecId;

return RecId;
}

[DataMemberAttribute, SysOperationControlVisibilityAttribute(false)]
public NoYes parmThrowError(NoYes _throwError = throwError)
{
throwError = _throwError;
return throwError;
}

[DataMemberAttribute, SysOperationControlVisibilityAttribute(false)]
public int64 parmWorkItemCount(int64 _workItemCount = workItemCount)
{
workItemCount = _workItemCount;
return workItemCount;
}
}

Controller

Next, we have the controller. This just links the execution controller to the service.

class AAXBatchIndividualTaskController extends SysOperationServiceController
{
public static SysOperationController construct(Args _args = null)
{
SysOperationController controller = new AAXBatchIndividualTaskController(
classStr(AAXBatchIndividualTaskService),
methodStr(AAXBatchIndividualTaskService, process));

controller.parmArgs(_args);

return controller;
}

public static void main(Args _args)
{
AAXBatchIndividualTaskController::construct(_args).startOperation();
}
}

Service

Finally, we have the service. This is what does the actual work. Based on the method specified in the controller, the process method below will called during batch execution. The example below also has a class that does the work, AAXBatchIndividualTaskModelWorkItem to encapsulate the workload for this example. You'll note the service is BatchRetryable so we'll want the class that does the actual work, AAXBatchIndividualTaskModelWorkItem to be built in such a way that it may try to update the same record twice, although this type of scenario would be very rare.

public class AAXBatchIndividualTaskService extends SysOperationServiceBase implements BatchRetryable
{

public void process(AAXBatchIndividualTaskDataContract _dataContract)
{
info(strFmt("Running batch job with RecId parameter: %1", _dataContract.parmRecId()));
Info(strfmt("ThrowError set to %1", _dataContract.parmThrowError()));

AAXBatchIndividualTaskModelWorkItem::doProcessWorkItem(_dataContract.parmRecId());

if(_dataContract.parmThrowError())
{
if(new RandomGenerate().randomInt(0,100) == 100)
throw Error("Encountered Error");
}
}
public boolean isRetryable()
{
return true;
}

}

Work Class

As mentioned above, we've encapsulated the logic for this batch job into a class.

internal final class AAXBatchIndividualTaskModelWorkItem
{
public static void doProcessWorkItem(RecId _recId)
{
AAXIndividualTaskWorkItemTable AAXIndividualTaskWorkItemTable;

select forupdate Id, Updated, RecId from AAXIndividualTaskWorkItemTable
where AAXIndividualTaskWorkItemTable.RecId == _recId
&& AAXIndividualTaskWorkItemTable.Updated == NoYes::No;

if(AAXIndividualTaskWorkItemTable)
{
ttsbegin;
AAXIndividualTaskWorkItemTable.Updated = NoYes::Yes;
AAXIndividualTaskWorkItemTable.update();
ttscommit;
}
}

}