scheduler job execute more than one time

Question

sugarcrm version: SugarCRM Version 11.0.3 (Build 292 P) (Q2 2021) (professional) 
 I have a monthly invoice renewal cron job. It run on every 1 day of each month and will create around 4000 invoices. The problem is I found some invoice is created more than 1 or 2 randomly and usually no more than 10 duplicates each month. I have checked the job_queue table, all job status is done and 0 requeue and none of it have retry_count. I have no clue where to check and why these scheduler job run more than once. Any suggestion would be appreciated.

Manson ho · Answer

Hi everyone, For those of you who are encountering this problem. Here are some observations and how we are avoiding this scenario. This did happen in version 12.0.2 as well. 1. When we added logs in the custom job script, we could confirm that a single job was picked up multiple times. 2. Then we added a simple file-locking mechanism. It checks if a job has a corresponding file, if not, create a new file, execute the job and delete the file. If the file is found, exit the script. This brought down the number of duplicates drastically. Some jobs were executed simultaneously at the same time. 3. To avoid the parallel run scenario, we introduced a random sleep between 0 and 1 second at the start of the script. We had 0 duplicates. 
 4. We had a max_cron_jobs value close to 200 in production. Upon bringing it down to 50 (the default is 25 I think), the duplicate did not occur. I do not fully understand the significance of max_cron_jobs values. I guess a very high value makes the processing of job queues unreliable. 
 So, we have brought down the value of max_cron_jobs and also based on the criticality of the job and to ensure the same job is not run again, we will still keep the locking and delay mechanisms in our script. 
 Hope this is of use to someone. 
 Ramprasath Karunakaran