A Generic Recursive Runtime Decision Making Batch Class

Previously you could only execute 5 batches jobs from any single context

But one of my ideas on IdeasExchange was included in a recent Salesforce Release now you execute up to 100 batches which become queued in the AsyncApexJob object

What I'd like to cover in this blog, a generic recursive runtime decision making batch class.

We will make a class that requires little change and can serve and batch processing for any batch.

There are situations whereby once a batch has fully executed you want to initiate another batch:

1.      The first batch executes as many operations as it can and then initiates a decision process that either executes the same batch process again or ends the executions

            Situations where this scenario can be used:

a.       A callout to a 3rd party system and you dont know how many records exist in the 3rd party system

2.      After the first batch executes, records are set into a state that now allows a different batch to execute. Of course the second batch could be scheduled for a certain time but there is no way of knowing when the 1st batch will complete and so you have to space batches apart. If it is important to complete the operations in a timely fashion you will want to execute the 2nd batch immediately when the 1st batch completes

            Situations where this scenario can be used:

a.       The 1st batch updates a field on the Account which fires a trigger and workflows. This sets conditions on say the Contact object by updating various fields. The 2nd batch now picks up records on the Contact where this field has been updated. So we need the 1st batch to complete for the 2nd to process.

Lets consider a situation where a batch makes a call to a 3rd party system requesting for a number of records, but due to payload limitations the 3rd party can only return a certain number of records and the 3rd party doesn't provide a means of identifying how many records there are in the 3rd party because such a call drains system resources on the 3rd party.

So we need to setup a batch class that makes a call to the 3rd party and retrieves X number of records. When the batch falls into the Finish() we call a decision method which identifies how many records were processed which tells us if we have processed the last batch or not.

public with sharing class Constants {             public static final string CONST_DOWNLOAD = 'DOWNLOAD 3rd PARTY'; }

global class batchProcess implements Database.Batchable<sObject>, Database.Stateful, Database.AllowsCallouts{             global integer mx;                               //number of records to process             global String batchType;                     //identifies which batch processing to call             global String soql;                               //the soql query if the batch is to make a query to feed records into the Execute()             global Map<String,String> vars;         //this holds any arguments that are to be passed to the batch function in the Execute()             global Boolean success;                      //determines if the last batch execution was successful, if it wasnt we might decide to stop any further batch processing since there is a possibly a fault has been encountered             global batchDownloadCurrentGlobals(String thisbatchType){                         batchType = thisbatchType;             }                         global batchProcessing (String thisbatchType, Map<String,String> thisvars, String thissoql){                         batchType = thisbatchType;                         vars = thisvars;                         soql = thissoql;             }                         global Database.QueryLocator start(Database.BatchableContext bc) {                         if (soql == null || soql == '')                                     return Database.getQueryLocator('Select id From User limit 1');                        else                                     return Database.getQueryLocator(soql);             }                         global void execute(Database.BatchableContext BC, List<sObject> glbs){                                     if (batchType == Constants.CONST_DOWNLOAD){//identifies the batch type we are calling, for a different batch you simply introduce another if statement                                                                            if (vars.containskey('Max') && vars.get('Max') != '0'){                                                             List<ApexClass> apxCls = (List<ApexClass>)glbs;                                                             String maxCls = vars.get('Max');                                                             mx = integer.valueof(maxCls)-1;                                            //call method that retrieves "mx" number of records from the 3rd party, if the callout can be made and is successful this returns true to "success". You could introduce a for loop here to make the callout a maximum of 10 times to reduce on the number of batch operations                                     success = Utils.retrieveData(mx);                                     }                         }             }             global void finish(Database.BatchableContext BC){                         if (batchType == Constants.CONST_DOWNLOAD ){                                     if (success)//identifies the last callout was successul                                                 Utils.decideToRunAgain(mx);                                     else                                                 //do something when last batch didnt process and encountered an issue                         }             }

This is the decision method:

public static void decideToRunAgain (integer mx){                         //This custom setting is set in retrieveData() for the number of records retrieved from the 3rd party in the last callout made in the batch Execute() if this number is less than "mx" the last callout was the last callout required                         Configurations__c latestCall = Configurations__c.getinstance(Constants.CONST_MAX);         integer newlatestCallInt = (latestCall != null) ? integer.valueof(latestCall.Value__c) : 0;                         if (newlatestCallInt == mx){                                     //the last callout retrieved the same number of records as was requested so this cannot be the last callout to make so a new batch can be created                                                                         //we also need to check that the number of queued batches is less than 100 otherwise the maximum in the queue has been reached                                     //unfortunately we cannot halt execution for a time or even continually check AsyncApexJob in a for loop waiting for the queue to drop because that would hit governor limits //Note:JobType ='Batch Apex' identifies a batch being processed, the JobType ='Batch Apex Worker' identifies the latest record being processed in the batch and so is constantly changing                                     if ([Select id From AsyncApexJob where JobType ='Batch Apex' and Status = 'Holding'].size() < 100){                                                 batchProcessing batch = new batchProcessing (Constants.CONST_DOWNLOAD, <<specify the other parameters>>);                     Database.executeBatch( batch, 1 );                                     }         }                          }

           

The are various different themes you can employ on this concept, such as all the logic could be pulled completely outside of the batch into separate classes, keeping the batch class lightweight and actually never needs to change

Further information

http://releasenotes.docs.salesforce.com/en-us/spring15/release-notes/rn_apex_flex_queue_ga.htm?edition=&impact=