Introducing ExternalResourceClient your friendly Transport Client

29 Jun 2022

10 minute read time

In our Q3 2022 (12.1) developer webinar, we announced many cool features we’re adding in this release, and one that stands out and is most impactful to you as a developer is the cURL replacement by Sugar's own ExternalResourceClient.

We will explore this in much more detail in this article.

Who is affected by this change?

If you've built an app or integration for Sugar that uses a Module Loadable Package (MLP) that includes any of the PHP curl_*, socket_* or stream_* functions then you will need to change your code to use this new ExternalResourceClient lib. This client will be added in 12.1 and backported as patches to Sugar 11.0 and 12.0 releases. You will also need to update your module manifest to indicate appropriate compatibility, for example, with Sugar 11.0.4 and greater.

What action do I need to take?

At the very least, you should scan your code for curl_, socket_, stream_, or look for any of the following blocked functions before the upgrade, and if you find any, it is a good idea to plan to replace them with ExternalResourceClient sooner rather than later.

If you have access to Sugar Release Previews, we encourage you to check our webinar recording and slides for Sugar 12.1 (SugarCloud release), since we did dig deep into the new ExternalResourceClient and several different scenarios, it will definitely help you to proactively identify and resolve cURL issues prior to GA.

If you are not a member of the Sugar Release Preview program and would like to be added, please contact developers@sugarcrm.com.

Continue reading below to see if you think you can get a head start on any of the changes we are highlighting.

What will happen during the next Sugar upgrade?

As I communicated in the webinar, we have the following plan in place to ease the transition and give time to all of our community to adapt. We have temporarily introduced a Sugar Config parameter that decides if checks for blocked PHP cURL and etc. functions are enforced.

For the Sugar_12.1.0 release, the temporary config parameter will be disabled by default allowing MLPs with HTTP clients like curl, socket, and streams to be updated or installed.
For Sugar_12.2.0 and Sugar_12.3.0 we will enable the temporary config by default so that MLPs with the above functions will be blocked, Sugar Support has the ability to turn this off.
For Sugar_13.0.0 we will be removing the temporary config setting, by this time we will have given everyone 4 releases time to adapt.
Let MLP's developers safely communicate with external APIs to stay in place and will be the main source of communication.
ExternalResourceClient will be installed into all supported Sugar versions (ex. 11.0.4+) as part of your next patch or feature release upgrade.

Self-Signed Certificates

ExternalResourceClient has been designed to be as secure as possible. We will not support disabling the check for authenticity of the peer's certificate as you could in cURL with using the CURLOPT_SSL_VERIFYPEER option. This is not a security best practice therefore not allowed in the new client.

If your target URL has an invalid or self-signed certificate, you can benefit from using Let’s Encrypt, a free, automated, and open Certificate Authority (CA) that makes it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention. You can get started here.

IP vs Domain/Hostnames via DNS

As mentioned during the webinar, ExternalResourceClient does not support using an IP address in your URL. Editing /etc/hosts on your server is a very easy and fast way of accomplishing this but if that isn't an option and you have a proper DNS server configured, you can use one of the following articles to help you understand what DNS is and how you can configure them in different OSs and infrastructures.

Local Endpoints

If you don’t know what SSRF (server-side request forgery) is, please refer to this detailed explanation and examples of how an attacker can reach your local network using malicious HTTP requests.

ExternalResourceClient prevents this type of attack by taking the resolved IP from your URL and checking against a Sugar Config ($sugar_config['security']['private_ips’]) array list.

This list contains a wide range of internal IPs blocked by default ([ '10.0.0.0|10.255.255.255', '172.16.0.0|172.31.255.255', '192.168.0.0|192.168.255.255', '169.254.0.0|169.254.255.255', '127.0.0.0|127.255.255.255')].

For example, if your URL is http://an-internal-url/api/get and that “an-internal-url” resolves to an IP 10.0.0.87, it will throw an exception because it is within the range of your config.

Add your critical IP ranges to this config so no attacker will get to sensitive IPs through ExternalResourceClient’s HTTP Request.

Sugar Proxy

To get proxy settings ExternalResourceClient relies on Administration::getSettings('proxy'). This utility queries Sugar's infrastructure (cache or DB) to retrieve those settings, therefore it needs to be bootstrapped into SugarCRM.

In the case of MLPs that use standard Modules via the web, you don’t need to manually bootstrap SugarCRM, so you may use ExternalResourceClient in your methods without requiring entryPoint.php

So in case, you are writing a CLI script you need to include SugarCRM entryPoint.php:

<?php
if (!defined('sugarEntry')) {
   define('sugarEntry', true);
}

define('ENTRY_POINT_TYPE', 'api');
require_once 'include/entryPoint.php';

?>

Test your customization

You can easily test your package by enabling PackageScanner checks in the supported versions for now ($sugar_config['moduleInstaller']['enableEnhancedModuleChecks'] = true), remember, it comes disabled by default in the 12.1.

If your code doesn’t go through PackageScanner, it will not be enforced, that’s the premise we’re working with.

If you are an on-premise install, ExternalResourceClient relies on file_get_contents and requires allow_url_fopen to be ON in your php.ini.

In our SugarCloud, that setting is ON

Replace cURL by Examples

We've put together a series of examples below that can be used as a guideline on how to replace your current cURL code with ExternalResourceClient.

You must import this client and its exceptions (if you need them) from:

<?php

use Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient;
use Sugarcrm\Sugarcrm\Security\HttpClient\RequestException;

HTTP GET

The following code snippet provides you with a code replacement from cURL GET to ExternalResourceClient.

Replace

With

// cURL GET
$ch = curl_init();

$url = 'https://httpbin.org/get';
curl_setopt_array($ch, array(
    CURLOPT_URL => $url,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => "",
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 60,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_2TLS,
));

$response = curl_exec($ch);

$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$errorNo = curl_errno($ch);
$error = curl_error($ch);

if ($errorNo !== 0) {
    $GLOBALS['log']->log("fatal", "curl error ($errorNo): $error");
    throw new \SugarApiExceptionError("curl error ($errorNo): $error");
}

curl_close($ch);

if ($httpCode === 0 || $httpCode >= 400) {
    $GLOBALS['log']->log("fatal", "Error message: $error");
    throw new \SugarApiException($error, null, null, $httpCode ? $httpCode : 500);
}

echo $response;

// ExternalResourceClient GET
try {
    // Set timeout to 60 seconds and 10 max redirects
    $response = (new ExternalResourceClient(60, 10))->get($url);
} catch (RequestException $e) {
    $GLOBALS['log']->log('fatal', 'Error: ' . $e->getMessage());
    throw new \SugarApiExceptionError($e->getMessage());
}
$httpCode = $response->getStatusCode();
if ($httpCode >= 400) {
    $GLOBALS['log']->log("fatal", "Request failed with status: " . $httpCode);
    throw new \SugarApiException("Request failed with status: " . $httpCode, null, null, $httpCode);
}
echo $response->getBody()->getContents();

HTTP POST

The following code snippet provides you with a code replacement from cURL POST to ExternalResourceClient.

Replace

With

// cURL POST JSON
$url = 'https://httpbin.org/post';

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array ("Content-Type: application/json"));
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(['foo' => 'bar']));

$response = curl_exec($ch);
curl_close($ch);
$parsed = !empty($response) ? json_decode($response, true) : null;
var_dump($parsed);

// ExternalResourceClient POST JSON
try {
    $response = (new ExternalResourceClient())->post($url, json_encode(['foo' => 'bar']), ['Content-Type' => "application/json"]);
} catch (RequestException $e) {
    throw new \SugarApiExceptionError($e->getMessage());
}

$parsed = !empty($response) ? json_decode($response->getBody()->getContents(), true) : null;
var_dump($parsed);

Authenticated Endpoints (CURLOPT_USERPWD)

The following code snippet provides you with a code replacement from cURL to ExternalResourceClient passing authentication params.

Replace

With

$username = 'foo';
$password = 'bar';
$url = 'https://httpbin.org/basic-auth/{$username}/{$password}';

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_USERPWD, $username . $password);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(['foo' => 'bar']));

$response = curl_exec($ch);
curl_close($ch);
$parsed = !empty($response) ? json_decode($response, true) : null;
var_dump($parsed);

$username = 'foo';
$password = 'bar';
$auth     = base64_encode( "{$username}:{$password}" );
 //Passing custom 'Authorization' header
// try + catch is omitted for brevity
$response = (new ExternalResourceClient())->get("https://httpbin.org/basic-auth/{$username}/{$password}", [
    'Authorization' => 'Basic ' . $auth,
]);
echo $response->getBody()->getContents();
 
// OR the same by using user:password@hostname.tld URL format
 // try + catch is omitted for brevity
$response = (new ExternalResourceClient())->get("https://{$username}:{$password}@httpbin.org/basic-auth/{$username}/{$password}");
echo $response->getBody()->getContents();

Stream

The following code snippet provides you with a code replacement from stream to ExternalResourceClient.

Replace

With

<?php
// Sending GET
if ($stream = fopen('https://httpbin.org/get', 'r')) {
    echo stream_get_contents($stream);
}
 
// Sending POST
$options = [
    'http' => [
        'method' => 'POST',
        'header' => ["Content-Type: application/x-www-form-urlencoded"],
        'content' => http_build_query(['fieldName' => 'value', 'fieldName2' => 'another value']),
        'ignore_errors' => true,
    ],
];
$context = stream_context_create($options);
if ($stream = fopen('https://httpbin.org/post', 'r', false, $context)) {
    echo stream_get_contents($stream);
}

// Using ExtenalResourceClient
// GET
echo (new ExternalResourceClient())
->get(‘https://httpbin.org/get’)
->getBody()
->getContents();
// POST
echo (new ExternalResourceClient())
->post(
‘https://httpbin.org/post’, 
['fieldName' => 'value', 'fieldName2' => 'another value']
)
->getBody()
->getContents();

Socket

The following code snippet provides you with a code replacement from socket to ExternalResourceClient.

Replace

With

<?php 

// Sending GET
$fp = fsockopen("httpbin.org", 80, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)\n";
} else {
    $out = "GET /get HTTP/1.1\r\n";
    $out .= "Host: httpbin.org\r\n";
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}
// Sending POST
 
$fp = fsockopen("httpbin.org", 80, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)\n";
} else {
    $postData = http_build_query(['fieldName' => 'value', 'fieldName2' => 'another value']);
    $out = "POST /post HTTP/1.1\r\n";
    $out .= "Host: httpbin.org\r\n";
    $out .= "Content-type: application/x-www-form-urlencoded\r\n";
    $out .= "Content-length: " . strlen($postData) . "\r\n";
    $out .= "Connection: Close\r\n\r\n";
    $out .= "{$postData}\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}

// Using ExtenalResourceClient
// GET
echo (new ExternalResourceClient())
->get(‘http://httpbin.org/get’)
->getBody()
->getContents();
// POST
echo (new ExternalResourceClient())
->post(
‘http://httpbin.org/post’, 
['fieldName' => 'value', 'fieldName2' => 'another value']
)
->getBody()
->getContents();

Upload a file

The following code snippet provides you with a code example using ExternalResourceClient to upload files from Sugar.

// Uploading files from `upload` directory
// File data will be accessible on the target server as $_FILES['file_field_name']
echo 'File upload', PHP_EOL, PHP_EOL;
$file = new UploadFile('file_field_name');
// relative to `upload` directory
$filename = 'f68/2cad6f68-e016-11ec-9c9c-0242ac140007';
$file->temp_file_location = $file->get_upload_path($filename);
// try + catch is omitted for brevity
echo (new ExternalResourceClient())->upload(
    $file,
    'https://httpbin.org/post',
    'POST',
    ['foo' => 'bar'], // Optional, additional form data
    ['custom-header-name' => 'custom-header-value'] // Optional, additional headers
)->getBody()->getContents();

Parents

Dmitrii Mikhalchenko over 2 years ago

Hi Rafael,

We are rebuilding Customer integration to use ExternalRessourceClient, but its not working.

This is how we use it:

use Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient;
use Sugarcrm\Sugarcrm\Security\HttpClient\RequestException;

$url = 'https://connect.eventpro.net/ApiAU/DataService.svc/leads?'. urlencode('filter=(Modified >= "2022-06-27T07:37:00.000" AND Modified <= "2022-11-28T09:19:17.743") AND [Contact.ContactCategory2List][ItemCode="Venue Client"].Exists()');
try {
    // Set timeout to 60 seconds and 10 max redirects
    $response = (new ExternalResourceClient(60, 10))->get($url,  ['authorization' =>  'WEDrzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%247893R21iTjBMq6dw%3d%3d']);
} catch (RequestException $e) {
    $GLOBALS['log']->log('fatal', 'Error: ' . $e->getMessage());
    throw new \SugarApiExceptionError($e->getMessage());
}

$httpCode = $response->getStatusCode();

if ($httpCode >= 400) {
    $GLOBALS['log']->log("fatal", "Request failed with status: " . $httpCode);
    throw new \SugarApiException("Request failed with status: " . $httpCode, null, null, $httpCode);
}
echo json_decode($response->getBody()->getContents(), true); // will give response in array

and this is response we get:

Sugarcrm\Sugarcrm\Security\HttpClient\RequestException Object
(
    [message:protected] => file_get_contents(https://104.153.193.29/ApiAU/DataService.svc/leads?filter%3D%28Modified+%3E%3D+%222022-06-27T07%3A37%3A00.000%22+AND+Modified+%3C%3D+%222022-11-28T09%3A19%3A17.743%22%29+AND+%5BContact.ContactCategory2List%5D%5BItemCode%3D%22Venue+Client%22%5D.Exists%28%29)
        : failed to open stream: no suitable wrapper could be found
    [string:Exception:private] => 
    [code:protected] => 0
    [file:protected] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/src/Security/HttpClient/ExternalResourceClient.php
    [line:protected] => 199
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                    [file] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/src/Security/HttpClient/ExternalResourceClient.php
                    [line] => 84
                    [function] => request
                    [class] => Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient
                    [type] => ->
                )

            [1] => Array
                (
                    [file] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/RunScheduler.php
                    [line] => 26
                    [function] => get
                    [class] => Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient
                    [type] => ->
                )

        )

    [previous:Exception:private] => 
)

2 options we found:

1. set php_admin_flag allow_url_fopen "On" but this is sugarondemand Cloud, not sure if it would be prohibited;

2. There is no issue if we use GuzzleHttp\Client and no PHP_Flag required to set up:

use GuzzleHttp\Psr7\Request;
$client = new GuzzleHttp\Client(['headers' => ['authorization' => 'WEFdzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akGHTE4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']]);
$url = 'https://connect.eventpro.net/ApiAU/DataService.svc/leads?'. urlencode('filter=(Modified >= "2022-06-27T07:37:00.000" AND Modified <= "2022-11-28T09:19:17.743") AND [Contact.ContactCategory2List][ItemCode="Venue Client"].Exists()');
try {
    $request = new Request('GET', $url, ['authorization' => 'WEJbzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']);
    $res =  $client->send($request, ['headers' => ['authorization' => 'WEJbzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']]);
    $output = json_decode($res->getBody()->getContents(), true);
} catch (\GuzzleHttp\Exception\GuzzleException $e) {
    $errorMessage = json_decode($e->getResponse()->getBody(true));
    echo "<pre>"; print_r($errorMessage);
}

What would be your recommendation?

Regards,
Dmitrii

Comment

Dmitrii Mikhalchenko over 2 years ago

Hi Rafael,

We are rebuilding Customer integration to use ExternalRessourceClient, but its not working.

This is how we use it:

use Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient;
use Sugarcrm\Sugarcrm\Security\HttpClient\RequestException;

$url = 'https://connect.eventpro.net/ApiAU/DataService.svc/leads?'. urlencode('filter=(Modified >= "2022-06-27T07:37:00.000" AND Modified <= "2022-11-28T09:19:17.743") AND [Contact.ContactCategory2List][ItemCode="Venue Client"].Exists()');
try {
    // Set timeout to 60 seconds and 10 max redirects
    $response = (new ExternalResourceClient(60, 10))->get($url,  ['authorization' =>  'WEDrzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%247893R21iTjBMq6dw%3d%3d']);
} catch (RequestException $e) {
    $GLOBALS['log']->log('fatal', 'Error: ' . $e->getMessage());
    throw new \SugarApiExceptionError($e->getMessage());
}

$httpCode = $response->getStatusCode();

if ($httpCode >= 400) {
    $GLOBALS['log']->log("fatal", "Request failed with status: " . $httpCode);
    throw new \SugarApiException("Request failed with status: " . $httpCode, null, null, $httpCode);
}
echo json_decode($response->getBody()->getContents(), true); // will give response in array

and this is response we get:

Sugarcrm\Sugarcrm\Security\HttpClient\RequestException Object
(
    [message:protected] => file_get_contents(https://104.153.193.29/ApiAU/DataService.svc/leads?filter%3D%28Modified+%3E%3D+%222022-06-27T07%3A37%3A00.000%22+AND+Modified+%3C%3D+%222022-11-28T09%3A19%3A17.743%22%29+AND+%5BContact.ContactCategory2List%5D%5BItemCode%3D%22Venue+Client%22%5D.Exists%28%29)
        : failed to open stream: no suitable wrapper could be found
    [string:Exception:private] => 
    [code:protected] => 0
    [file:protected] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/src/Security/HttpClient/ExternalResourceClient.php
    [line:protected] => 199
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                    [file] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/src/Security/HttpClient/ExternalResourceClient.php
                    [line] => 84
                    [function] => request
                    [class] => Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient
                    [type] => ->
                )

            [1] => Array
                (
                    [file] => /home/websites/agnsw.crm.onlocal.com/agnsw-sugarcrm/wwwroot/RunScheduler.php
                    [line] => 26
                    [function] => get
                    [class] => Sugarcrm\Sugarcrm\Security\HttpClient\ExternalResourceClient
                    [type] => ->
                )

        )

    [previous:Exception:private] => 
)

2 options we found:

1. set php_admin_flag allow_url_fopen "On" but this is sugarondemand Cloud, not sure if it would be prohibited;

2. There is no issue if we use GuzzleHttp\Client and no PHP_Flag required to set up:

use GuzzleHttp\Psr7\Request;
$client = new GuzzleHttp\Client(['headers' => ['authorization' => 'WEFdzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akGHTE4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']]);
$url = 'https://connect.eventpro.net/ApiAU/DataService.svc/leads?'. urlencode('filter=(Modified >= "2022-06-27T07:37:00.000" AND Modified <= "2022-11-28T09:19:17.743") AND [Contact.ContactCategory2List][ItemCode="Venue Client"].Exists()');
try {
    $request = new Request('GET', $url, ['authorization' => 'WEJbzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']);
    $res =  $client->send($request, ['headers' => ['authorization' => 'WEJbzt%2b22mHC7uyQGXH3qFYf7za0jDSSSHipe%2b2akFTBR4y1hJWKRR258E5yjVjaXQuY2nT%2f7yR21iTjBMq6dw%3d%3d']]);
    $output = json_decode($res->getBody()->getContents(), true);
} catch (\GuzzleHttp\Exception\GuzzleException $e) {
    $errorMessage = json_decode($e->getResponse()->getBody(true));
    echo "<pre>"; print_r($errorMessage);
}

What would be your recommendation?

Regards,
Dmitrii

Children

Rafael Fernandes over 2 years ago in reply to Dmitrii Mikhalchenko

Hi Dmitrii Mikhalchenko,

You should use ExternalResourceClient as it is our preferred and part of Sugar's best practice from now on.

It does rely on file_get_contents and requires allow_url_fopen to be ON as you mentioned.

I did confirm with our DevOps and Support that allow_url_fopen is ON in our Cloud.
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More Actions
- Cancel