When the the fts_queue is being processed at full capacity, there are issues to keep in mind considering performance. For the most part, these can be controlled by the $sugar_config['search_engine']['max_bulk_query_threshold'] setting.
Often times, when fts_queue cycles fail (shown in the job_queue as name = “Elasticsearch Queue Consumer”) it’s because there are too many records being processed in one scheduler cycle (aka “Elasticsearch Queue Consumer” cycle). But why is this the case? The short answer is when a “Elasticsearch Queue Consumer” job is run, all of the module data that is to be processed will be loaded into the server's RAM so as not to keep making hundreds of calls to the database for each job. Bulk operations are more efficient that single operations.
There two things to keep in mind with the “Elasticsearch Queue Consumer”: How many records are being processed at once and how big each record for that module can be. Here is why: the process will select a matrix or array of max_bulk_query_threshold records (or rows), each with a field (or column) for each searchable field in the module. So, if max_bulk_query_threshold is set to 15,000 (which is default) there will be 15,000 records selected from the module and if there are 50 fields that are searchable, there are 50 columns in the array. That is three quarters of a million fields to save in memory. Keep in mind, a lot of times a description field or TextArea field is marked as searchable and these can be very long (multiple kilobytes). Frequently, you will know the max_bulk_query_threshold is set too high because you’ll see memory errors in the PHP log.
Of course, since fts_queue processing is done asynchronously, but is supposed to look like it's instant, one would want to allow as many records as possible to be processed at once so the search will seem to be updated in realtime. However, as explained earlier, this volume can overwhelm the host running the application; therefore, a compromise must be made. Usually the first adjustment we try in the config_override.php file is this:
$sugar_config['search_engine']['max_bulk_delete_threshold'] = 1000;$sugar_config['search_engine']['max_bulk_query_threshold'] = 5000;$sugar_config['search_engine']['search_engine.max_bulk_threshold'] = 1000;
In our adjusted settings, the max_bulk_query_threshold is set to about 1/3 the size of its default. The other two settings, 'max_bulk_delete_threshold and search_engine.max_bulk_threshol, are usually kept at about 1/3 of the value of max_bulk_query_threshold. All three of these values are used to control bulk operations. Bulk operations are more efficient and cause less network traffic but also require more memory and are more prone to fail if the values are set too high.
So, how will lowering these values going to affect performance? To be perfectly pragmatic, if the settings are not working properly on a higher setting, that leads to zero performance. A lower setting that works is better than zero. Even at this slower speed, realistically, it gets better. Most of the time, there are only a few records per module in an “Elasticsearch Queue Consumer” run at any one time – only those records that have been edited or updated. Of course, with a mass update, an import or a re-indexing, that speed is more of the essence, but that is not something that happens often and can usually be controlled. In most cases under normal user use the number of records being processed during each “Elasticsearch Queue Consumer” cycle is far lower than the max_bulk_query_threshold value.
If your instance is hosted on the SugarCloud, you will need to have a Technical Support representative change these settings in your config_override.php file.
Related Articles:
Monitoring Elastic Search Indexing With job_queue Queries
Checking The Status Of Global Search Indexing
Core Settings - search_engine.max_bulk_threshold