What file to customize to change the way sugar does login?

You can create a custom user authentication. Have a look on modules/Users/authentication/, here you find different authentication methods. You can write your own files in custom/modules/Users/authentication/CustomSugarAuthenticate/ and custom/modules/Users/authentication/SugarAuthenticate/ to create an own SSO mechanism together with an own view in custom/modules/Users/views/view.authenticate.php

For this you need some entry like $sugar_config['authenticationClass'] = 'CustomSugarAuthenticate'; 
in config.php (not in config_override.php!)

It's not simple, but doable.

See here for current implementation: https://support.sugarcrm.com/Knowledge_Base/Password_Management/Security_Layers_for_User_Authentication_in_Sugar/

Can you please elaborate/detail how this should happen? It is very promising, but I never touched this area, so I am unfamiliar. I customized quotes and accounts usually

You need a clear specification how the SSO should work.

One possible scenario could be that Sugar Authentication has to call an external tool which creates a session and returns the session_id back to Sugar, perhaps by another service call from sugar to this tool. 

So, the first thing is to know the details  of the used SSO tool, some kind of information flow.

Oooh.. you mean it is possible to DIRECTLY implement SSO? We use Open ID connect, SAML is not yet available.

support.sugarcrm.com/.../

Ah, but here is the problem. We are "google" from your example. Meaning we are the identity provider. And we are on premise. It seems to me that sugarCRM has a couple of hard coded options (azure, google etc). We need to connect to our own

Yes, when you are your own Google, you have to implement you own SSO. I did that some time ago for CAS, which calls Sugar with a ticket. The Sugar has to ask for the ticket details, check it against its credentials and gives access or not.

All that can be done by implementing an own custom authentication class. 

What about plugins like outlook. We had an attempt way back and were stopped because outlook plugins are hardcoded.

Outlook plugins are a piece of code running under control of Outlook. You do not have a chance to implement an SSO on client side.

On the server side they call webservices to authenticate. So, you could change these webservices also, redirecting to the SSO provider. But again, it is your own custom code. And there is no access to any auth token by the outlook to sugar connection, so you have to enter the credentials in the plugin anyways.

Outlook plugins are a piece of code running under control of Outlook. You do not have a chance to implement an SSO on client side.

On the server side they call webservices to authenticate. So, you could change these webservices also, redirecting to the SSO provider. But again, it is your own custom code. And there is no access to any auth token by the outlook to sugar connection, so you have to enter the credentials in the plugin anyways.