SSL Connection Error with Azure Hybrid Connection

When we added a hybrid connection to our Azure App Service using an Azure Resource Manager (ARM) template, we were greeted by a very difficult to solve error message, “The SSL connection could not be established.”

The architecture of the application is very similar to the diagram depicted by Microsoft’s documentation on Azure App Service Hybrid Connections. There is a web API hosted in Azure App Service that uses a relay hybrid connection to call an on-premise API.

The full stack trace returned to our application after adding the hybrid connection association to our ARM template is as follows:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at SolveProblems.WebApi.Controllers.V1.ResourceController.GetAsync() in D:\DEV-2017-01A\959\s\src\SolveProblems.WebApi\Controllers\V1\ResourceController.cs:line 42

The following three error messages stood out to us.

  • The SSL connection could not be established
  • System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host
  • System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host

The fact that some of the on-premise APIs use X.509 certificates that are signed by a private certificate authority led us to believe that the SSL connection might not be getting established due to an untrusted certificate.

After many hours of debugging, we determined that the sendKeyName property of the hybrid connection in the ARM template was not being set correctly. Below is the snippet where we attach the hybrid connection to our Azure App Service using the Microsoft.Web sites/hybridConnectionNamespaces/relays template reference. The snippet below shows the sendKeyName that was being set incorrectly.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Web/sites",
      "apiVersion": "2018-11-01",
      "resources": [
        {
          "condition": "[equals(parameters('ASPNETCORE_ENVIRONMENT'), 'development')]",
          "apiVersion": "2019-08-01",
          "dependsOn": [
            "[resourceId('Microsoft.Web/sites', variables('appName'))]"
          ],
          "name": "[concat(variables('relayName'), '/', parameters('hybridConnectionName'))]",
          "properties": {
            "hostname": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[0]]",
            "port": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[1]]",
            "relayArmUri": "[variables('hybridConnectionResourceId')]",
            "relayName": "[parameters('hybridConnectionName')]",
            "sendKeyName": "[parameters('THIS_WAS_USING_THE_WRONG_PARAMETER')]",
            "sendKeyValue": "[listkeys(concat(variables('hybridConnectionResourceId'), '/authorizationRules/defaultSender'), '2017-04-01').primaryKey]",
            "serviceBusNamespace": "[variables('relayName')]"
          },
          "tags": {
            "displayName": "hc-name-of-our-hybrid-connection"
          },
          "type": "hybridConnectionNamespaces/relays"
        }
      ]
    }
  ]
}

Once we set the sendKeyName correctly, the hybrid connection worked perfectly. 💯

In summary, if you receive an SSL connection error when using hybrid connections to contact on-premise APIs from an Azure App Service, then be sure to check the settings in the ARM template before spending hours or days checking every potential issue that can cause SSL communications to fail. One way to determine if the settings in the ARM template are the culprit is to deploy your application, manually remove the hybrid connection from your Azure App Service, then manually add the hybrid connection back. If it works after that, then something is amiss with your ARM template.