Improving the security of files uploaded to Sugar

When users can upload arbitrary files to an application, it poses a considerable security risk when those files are stored on the file system. For instance, a user might unknowingly upload a file containing an executable script by a malicious source, in addition to legitimate files such as images or CSVs from other sources. This presents a security threat that we are addressing in Sugar solutions.

We introduced a file encoding mechanism in the Sugar 12.2 release which significantly improves security by preventing such exploits from happening. With this mechanism in place, any uploaded file is automatically encoded to reduce the likelihood of malicious files being executed on the host system. This enhancement is an essential step towards ensuring the safety and integrity of our application, and we encourage all users to take advantage of this added security feature.

We’ve been utilizing this feature in SugarCloud for a few months now and we would like to share some valuable lessons learned with you. Especially for those of you running on-premise environments like Sugar Enterprise.

When upgrading to versions 12.2, 11.0.6, 12.0.3 and 13.0, we noticed that the SugarUpgradeEncodeUploadFiles upgrade script could take several hours to complete. For some customers, it couldn’t be completed successfully. After some investigation, we found out that upload directories larger than 500GBs were being affected. upload directories using less than 500GBs were fine. Here’s why:

The approach we took on Sugar Upgrader script was to scan all the files in the uploads directory and run the encoding all-at-once regardless of the size. That proved to work for most of our SugarCloud customers, but those 500GB+ (even TB+) instances have led us to come up with an alternative that I’m happy to share with our community in case you need it. 

If you have a Sugar on-premise instance with a 500GB+ filesystem, follow those instructions before running the upgrade process, and as always, it’s a good practice to perform such upgrades on Cloned/Sandboxes if available:

• Rename the upload directory to upload_tmp (mv upload upload_tmp)
• Create a new upload directory (mkdir upload)
• Run the upgrade process as you normally would (note it will still “encode” the upload folder, but now in an empty folder).
• Post-upgrade, rename back upload_tmp to upload (rm -rf upload & mv upload_tmp upload)
• Run this standalone script for encoding the upload dir files at your convenience (over the weekend, during the night, or any time that works best for you).
  • This is a CLI PHP script, which means you need to create it, give exec permission, and trigger it from the Sugar instance.
    • vi Encode_uploads_with_progress.php (paste the contents of the script down below)
    • chmod +x Encode_uploads_with_progress.php
    • php Encode_uploads_with_progress.php
  • You will notice that this script provides some feedback on how many files it has processed until it finishes.

Script:

<?php

use Sugarcrm\Sugarcrm\Util\Files\FilePhpEntriesConverter;
define('sugarEntry', 'upgrade');

require_once('include/entryPoint.php');

if(!in_array("upload", stream_get_wrappers())) {
    UploadStream::register();
}
$iterator = new RecursiveIteratorIterator(new RecursiveDirectoryIterator(UploadStream::getDir()));
$fileConverter = new FilePhpEntriesConverter();

/** @var SplFileInfo $fileInfo */
echo "Starting conversion", PHP_EOL;
$filesProcessed = 0;
foreach ($iterator as $fileInfo) {
    if ($fileInfo->isDir()) {
        continue;
    }

    $path = $fileInfo->getPathname();
    $encPath = $fileConverter->convert($path);
    rename($encPath, $path);
    $filesProcessed++;
    if ($filesProcessed % 1000 === 0) {
        echo 'Processed ' . $filesProcessed . ' files.', PHP_EOL;
    }
}
echo 'Done', PHP_EOL;