Error while executing BULK api "in-code"

Hi there !!

In the purpose of enhancing an existing process within Sugar, I'm passing to asynchronous target creation into a campaign. 

The current state is that an external program is calling sugar once for the campaign creation, then calling X times the target creation API to attach targets. The problem is that the process was very badly designed, and for each target creation the token is retrieved, there is no cache... you can imagine how bad could be the performances if there are 50.000 targets.

To improve that, I created a custom API that will create the campaign based on the arguments passed to the API, then create X batchs of 1000 target creation to the job queue, this with the bulk api, here is an example of the created payload : 

{
	"requests": [{
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"111111\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"222222\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"333333\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"44444444\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"5555555\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"66666\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"77777773\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}, {
		"url": "\/v11_12\/part_c",
		"method": "POST",
		"data": "{\"accounts_part_c_1accounts_ida\":\"8888888\",\"camp_c_part_c_1camp_c_ida\":\"e5891fdc-868d-11ed-9ca5-00505698e5aa\"}"
	}]
}

Here is the part that is pushing an entry into the job_queue : 

$job = new SchedulersJob();
$job->name = "Send CreateTarget Request - ";
$job->data = json_encode($targets);
$job->target = "function::process_create_target_campaign";
$job->assigned_user_id = $current_user->id;

$jq = new SugarJobQueue();
$jq->submitJob($job);

Here is my scheduled task that is consuming the json payload : 

$data = json_decode($job->data, true);

$serviceBase = new RestService();
$bulkAPI = new BulkApi();
$bulkAPI->bulkCall($serviceBase, $data);

return true;

Everything is working fine, except the fact that I'm getting an error when the job is executed : 

[13:11] Enes Saridogan
Wed Dec 28 12:05:03 2022 [24340][1][FATAL] Job 728639b8-869f-11ed-af94-00505698e5aa (Send CreateTarget Request - ) failed in CRON run, The given security subject is not active

The job falls in error (status done, resolution failure) but the targets are created ... 

Any idea why I have this strange behaviour? 

Did you ever used the Bulk API like I did, by instanciating it directly within the code without real rest call ?

Thanks in advance for your answers and have a nice afternoon.

Best regards,

Enes

Parents
  • Hi ,

    Happy New Year...

    it seems to be that your "$serviceBase = new RestService();" is the problem as it wouldn't be authenticated and it needs to be to run the BulkApi.

    Try to change that to your job's instance or an object that has the variable "$user" to it, that's all it needs.

    With that, it should be able to complete the request properly and active/deactivate and complete your job request.

    Another thing is, 1000 creations per request? bit too much, try to create less but in an async + multi-threaded way.

    Pretty good performance material right here.

    SugarCRM | Principal Developer Advocate

Reply
  • Hi ,

    Happy New Year...

    it seems to be that your "$serviceBase = new RestService();" is the problem as it wouldn't be authenticated and it needs to be to run the BulkApi.

    Try to change that to your job's instance or an object that has the variable "$user" to it, that's all it needs.

    With that, it should be able to complete the request properly and active/deactivate and complete your job request.

    Another thing is, 1000 creations per request? bit too much, try to create less but in an async + multi-threaded way.

    Pretty good performance material right here.

    SugarCRM | Principal Developer Advocate

Children
No Data