We have been in business using Sugar since at least 2008 and this has never been an issue until the last month. Something must have caused this to happen. I don't know if there was an update where misc data is being retained that was never retained before but they need to figure this out. Now that I'm seeing another user with the same issue as of recent, it makes me believe it's not us. 

Bud Hartley file system allocation increases as email attachments are imported into the CRM as well as Documents/Notes attachments.

Some of our customers faced issues regarding filesystem getting inflated pretty fast.

In order to reduce the storage allocation we designed a strategy to identify and prune emails and their attachments not relevant to the Customer relationship and follow-up.

I mean, several companies do not care about being imported into the CRM emails not relevant to the Customer relationship and follow-up, like:

• automatic emails from google, apple etc
• COVID test results (these use to be pretty big ones)
• personal emails/attachments
• etc
• etc
• etc

By identifying those target emails we look for emails messages whose neither sender or recipient are not User, Lead, Contact, Account, Prospect, custom modules from person/company templates.

For our customers those emails and attachments accomplish around 80% of the total allocation!!!

We compile a sample of emails and attachments and share with the customers so they can validate and decide whether those data are relevant or not.

Based on customer decision we compile a huge sql file which prunes all records (emails, emails_text, emails_beans, emails_email_addr_rel, notes, notes_audi) and a bash script which deletes all associated attachments, then we file a ticket to SugarCRM Support to proceed the clean up.

Perhaps it helps you

Thanks Andre!

We have a firewall program that filters almost all of the junk mail.  You have me thinking that email attachments may be the issue (PDF purchase orders from customers mostly, and possibly email signatures that have images attached).

I appreciate the insight and I'll dig into that today.

Thanks Dawn!  Please let me know what you find.  The recent increase in the file size surprised me.  Our clean-up project has reduced the database size (down to 61 from about 73), and I;'m looking for a way to reduce the File System 

This brings to mind my request to segment the upload directory by related module, at least, which would allow me to distinguish which modules the Documents are linked to so I can tell the difference between Uploads that are copies of documents we need to retain (linked to the Custom Contracts Module for example), vs the spurious images from signatures or Case attachments that could be removed after x-years etc... which could allow my Uploads to be regularly cleaned up.

Yes!  That would help!  I've been locating and deleting Documents that are ancient history which came from our prior CRM 11 years ago.  I haven't seen a reduction in the File System size yet.

I've also been rolling through text field sizes that seem uselessly large - Like changing 255 to 50 for things that are never larger than 20 and 255 to 2 for fields that we don't use.

I received the warning from Sugar that we're about to incur a charge for the storage use if we don't reduce the size.

Hello, don't know if this will help, but we had users uploading huge files, there is a setting in admin > System Settings> Maximum upload size and you can set the file size limit. I have it set to 1000000.
They will get an error if the file is larger and then they use an online PDF program to shrink the size of the PDF. I believe there is a scheduler to prune the database to actually remove deleted records, maybe verify that is running.

Thanks Edward.  Yes, I reduced the maximum size of the uploads many years ago. I have the "standard" pruner running on the 1st of the month, and I have a second one that I trigger after we do clean-up projects.

The Database size has been reduced, but it's the File System that seems to keep growing.  The last successful reduction was when we cleaned out old Documents and Knowledge Base files.

I'm thinking we may have to increase our allocation.

Hi Bud Hartley,

I can't really explain it perfectly, because I'm not a developer myself, but we regularly do a kind of normalization in the upload directory (we are on-prem, I have to add): We identify file duplicates (which often happens through e.g. images in email signatures and stuff like that) with rdfind and hard-link all notes/emails that basically refer to the same file to one single copy of that file and delete the rest.

root@XXXXXXXX:/var/www/html/sugarcrm/upload [22:42] $ rdfind -removeidentinode false -deleteduplicates true -makehardlinks true . 
Now scanning ".", found 1113514 files. 
Now have 1113514 files in total. 
Total size is 139555040280 bytes or 130 GiB 
Removed 53148 files due to unique sizes from list.1060366 files left. 
Now eliminating candidates based on first bytes:removed 87369 files from list.972997 files left. 
Now eliminating candidates based on last bytes:removed 32170 files from list.940827 files left. 
Now eliminating candidates based on sha1 checksum:removed 13379 files from list.927448 files left. 
It seems like you have 927448 files that are not unique 
Totally, 71 GiB can be reduced. 
Now making results file results.txt 
Now making hard links. 
Making 829566 links.

Maybe it helps someone else as well

BR

Julia Weinhold

Many Thanks to Andre for pointing me to possible redundant files in the Notes Module!  Success!  There were hundreds of copies on PDF files that were emailed to many contacts - things like catalogs, and marketing flyers AND there were near countless copies of email signature files (from contacts and some outbound from users).

Deleting the "older" duplicates (from a year or more ago) reduced the File System size dramatically!