Trace HTTPS Requests and Responses in D365FO

Recently, we were faced with the challenge of checking what our D365FO environment was sending and receiving over HTTPS protocol on the Internet and why the integration with external services such as Microsoft Office or SharePoint Online was not working. In order to solve the problem at hand, we decided to intercept the HTTP traffic and check the URLs, sent and received headers, and the body (contents). The same technique can be used when exploring how to create your own customizations by calling the existing services that are already present in the D365FO and cannot be extended using the standard APIs.

In this article we describe in detail how to achieve such a scenario on the D365FO OneBox development machine.

The easiest way to do this is to set up a web debugging proxy tool, such as Telerik Fiddler, that sits between your computer and the Internet and captures incoming and outgoing HTTP traffic between the two. You can use the tool to examine the traffic and perform modifications, if necessary. However, the main problem here is that the traffic is encrypted, and you need to decrypt it, capture it, and then resend it to the destination service.

In this article, we will first show you how to configure Fiddler and then, on the example, examine the traffic and implement your own solution. The same procedure can be applied to detect problems in the standard D365FO functionality.

Install & Configure Telerik Fiddler on your D365FO

There are several versions of the Telerik Fiddler debugging tool, and we used Fiddler Classic for the purpose of this article. You can download and install Fiddler on your D365FO OneBox computer here.

After the installation is complete, open Fiddler and follow the steps described below to decrypt, inspect, and encrypt the traffic sent to and received back from your computer to the Internet:

  1. Open the Tools > Options menu
  2. Click on the tab HTTPS
  3. Enable the Capture HTTPS CONNECTs option
  4. Enable Decrypt HTTPS traffic option
  5. Confirm the Windows reconfiguration of the Trusted CA list
  6. Make sure you have accepted to install the Fiddlers certificate, as it acts as a man-in-the-middle to capture HTTPS traffic.
  7. Confirm all the next dialogs
  8. Click the OK button and Fiddler is now ready to decrypt, intercept and encrypt all the HTTPS traffic.
  9. Start the tracing by pressing F12 function key.

Now, that Fiddler is properly configured and running, we can enable redirection of D365FO OneBox traffic to the Fiddler’s proxy server, which by default runs at http://localhost:8888.

The easiest way is to go to the D365FO web.config file, which by default is located at c:\AOSService\webroot\web.config, and add the following configuration to the <system.net> XML node:

After you stop using Fiddler, don’t forget to remove or comment out the configuration you have added to the web.config file, as the Fiddler proxy only lives when you are capturing traffic.

If you set everything up as described above, you should now see quite a bit of traffic in the Fiddler, and it’s difficult to detect just the requests from the D365FO. You need to change the capturing filter from "All Processes" to "Non-Browser". The capturing filter "Non-Browser" only captures traffic from the background services, and in this case, those called from your D365FO environment. After changing the capturing filter, press "Ctrl+X" to clear all the captured traffic and start testing the calls in the D365FO.

The Example

In the following example, we will try to figure out how the conversion of the generated document from Electronic Reporting to PDF works, and what services are called to accomplish this scenario. Once we get the answer, we will try to implement our solution to convert any Microsoft Word or Microsoft Excel document to PDF format.

In our example, we use the Sales Invoice document. Here you can download all the configurations and import them into your environment. If you did everything as expected, you should have the following ER configuration set up in your environment, as shown in the following image.

Once you have imported the configurations, you need to define all the possible Electronic Reporting destination testing scenarios as follows:

  • Test Scenario #1 » Show the Sales Invoice on-screen print destination for Microsoft Word conversion to PDF
  • Test Scenario #2 » Show the Sales Invoice on-screen print destination for Microsoft Excel with conversion to portrait PDF layout
  • Test Scenario #3 » Show the Sales Invoice on-screen print destination for Microsoft Excel with conversion to landscape PDF layout

 

You can find out more about Electronic Reporting and Business Documents on our YouTube channel here.

Intercepting D365FO Calls

Now, that our environment is properly set up, we can start intercepting the calls from the D365FO environment. After we have printed the Sales Invoices for all the before mentioned possible output scenarios, we have detected the following HTTP API calls:

  1. Request authentication OAuth2 token for resource https://configure.global.dynamics.com from Azure AD (https://login.windows.net/docentric.com/oauth2/token).
  2. Request authentication API key from https://apikeys.configure.global.dynamics.com for Microsoft Office Online Service.
  3. Convert Word file to PDF by using Microsoft Word Online Service (https://wordcs.officeapps.live.com/document/export/pdf?input=docx).
  4. Convert Excel file to PDF by using Microsoft Excel Online Service with Portrait page orientation (https://excelcs.officeapps.live.com/document/export/pdf?input=xlsx&printOption=1;-1;2;0).
  5. Convert Excel file to PDF by using Microsoft Excel Online Service with Landscape page orientation (https://excelcs.officeapps.live.com/document/export/pdf?input=xlsx&printOption=2;-1;2;0).

If we dig deeper into the Fiddler’s results, we can see, what kind of requests and responses were made and find out how we could implement these principles into our solution (Word or Excel file to PDF conversion by using Microsoft Office Online Conversion Service). We can skip authorization calls and focus on the conversion.

All the requests and responses used in this article can be downloaded here as SAZ file (Session Archive file) and loaded back to Fiddler.

Test Scenario #1 » Convert Word to PDF

In Fiddler, we will first open the request and response shown on the next picture of the Microsoft Word to a PDF file conversion. The request is highlighted in the following picture.

In order to understand the request and response contents, we need to use the Fiddler’s Interceptors tab. Let’s now analyze and try to explain the relevant information, which can help us to understand, what is being called and returned from the Microsoft Office Conversion API.

We will explain the request and response contents in the following table:

Request
URL https://wordcs.officeapps.live.com/document/export/pdf?input=docx
 
The API call requires query string parameter input with the docx value.
HTTP Method POST
Headers
X-ClientCorrelationId The value represents a GUID value used for an easier log tracing.
The sample value used: e50fabfa-955e-4371-b24e-423ea8c97c14
X-PassThroughDownloadHeaders Sent but always empty.
ClientName The name of the client who made the request.
The sample value used: DynamicsAxElectronicReporting
Ocp-Apim-Subscription-Key The authorization and authentication key fetched from Dynamics globalization service.
The sample value used: 581f71160c5e429b8829d08dda2eeb01
Content-Type File sent to the server as binary in the request body is in format Microsoft Word Open XML.
The sample value: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Body Binary content of the Microsoft Word file to be converted to PDF.
Response
Headers
Content-Type The file returned from the server received as binary in the response body is in the format PDF.
The sample value used: application/pdf
X-ErrorCode The conversion error code. It can be used for a display to the end-user or to write to the log file in order to resolve the problem during the conversion process.
The sample value used: None
X-ClientCorrelationId The value represents a GUID value used for an easier log tracing.
The sample value used: e50fabfa-955e-4371-b24e-423ea8c97c14
X-ErrorCategory The error category. It seems that it’s the same as the HTTP status code.
The sample value: Success
X-IsRetriable We can ignore the headers as the service always return the entire document, and the value from the sample calls, was always the same.
The sample value: False
X-ConversionTimeInMs The time it took on the server-side to convert the source file to a PDF format.
The sample value: 890.6283
Body Represents the binary contents of the converted PDF file.

Test Scenario #2 » Convert Excel File to PDF in Portrait Layout

Let us now analyze the request and response used to convert Microsoft Excel to a PDF file in a portrait orientation (see on next picture). As we can see immediately from the request URL, the API requires information about the page orientation used to convert to a PDF format.

A detailed analysis of the request and response gives us the following relevant information, displayed in the following table:

Request
URL https://excelcs.officeapps.live.com/document/export/pdf?input=xlsx&printOption=1;-1;2;0
 
The API call requires the following query string parameters:

  • input=xlsx » source document in format XLSX.
  • printOrientation=1;-1;2;0 » Print orientation used for PDF is Portrait (the first integer value in the array of the specified values).
HTTP Method POST
Headers
X-ClientCorrelationId The value represents a GUID value used for an easier log tracing.
The sample value used: e50fabfa-955e-4371-b24e-423ea8c97c14
X-PassThroughDownloadHeaders Sent but always empty.
ClientName The name of the client who made the request.
The sample value used: DynamicsAxElectronicReporting
Ocp-Apim-Subscription-Key The authorization and authentication key fetched from Dynamics globalization service.
Sample value used: 581f71160c5e429b8829d08dda2eeb01
Content-Type The file sent to the server as binary in the request body is in the format Microsoft Word Open XML.
The sample value: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Body The binary contents of the Microsoft Excel file to be converted to a PDF format.
Response
Headers
Content-Type The server is returning after the conversion has been successful wrong Content-Type. We got text/html, but application/pdf is expected. In our sample solution, we will ignore this returned header value.
The sample value used: text/html
X-ErrorCode The conversion error code. It can be used for a display to the end-user or to write to the log file in order to resolve the problem during the conversion process.
The sample value used: None
X-ClientCorrelationId The value represents a GUID value used for an easier log tracing.
The sample value used: e50fabfa-955e-4371-b24e-423ea8c97c14
X-ErrorCategory The error category. It seems that it’s the same as the HTTP status code.
The sample value: Success
X-IsRetriable We can ignore the headers as the service always return the entire document, and the value from the sample calls, was always the same.
The sample value: False
X-ConversionTimeInMs The time it took on the server-side to convert the source file to a PDF format.
The sample value: 593.7451
Body Represents the binary contents of the converted PDF file.

Test Scenario #3 » Convert Excel File to PDF in Landscape Layout

And finally, for the last scenario, we converted Microsoft Excel to a PDF using landscape page layout.

The request and response are the same as in test scenario #2. The only difference is in the URL query string parameters. The first value in the query string parameter printOption contains the value 2 and represents the landscape page orientation (https://excelcs.officeapps.live.com/document/export/pdf?input=xlsx&printOption=2;-1;2;0).

The Reverse-Engineered Custom Solution

Now it’s time to write our sample solution in X++,call the Microsoft Office conversion service and apply what we have learned from the HTTP traffic analysis.

For simplicity purpose, we created a runnable class called DocOfficeConversionService_POC, that can be executed with the following URL calls from your OneBox D365FO environment:

  • Convert Microsoft Word to a PDF file » https://usnconeboxax1aos.cloud.onebox.dynamics.com/?cmp=usmf&mi=SysClassRunner&cls=DocOfficeConversionService_POC&args=docx
  • Convert Microsoft Excel to a PDF file with page orientation Portrait » https://usnconeboxax1aos.cloud.onebox.dynamics.com/?cmp=usmf&mi=SysClassRunner&cls=DocOfficeConversionService_POC&args=xlsx;Portrait
  • Convert Microsoft Excel to a PDF file with page orientation Landscape » https://usnconeboxax1aos.cloud.onebox.dynamics.com/?cmp=usmf&mi=SysClassRunner&cls=DocOfficeConversionService_POC&args=xlsx;Landscape
The sample solution uses some parts of the Docentric AX API. Please download and install Docentric AX Free Edition first, and then install the solution provided in this article.

The solution consists of two parts:

  • Part 1 » Request Service 2 Service Azure AD authorization token
  • Part 2 » Prepare and send the request to convert Word or Excel to PDF

Part 1 » Request Service 2 Service Azure AD authorization token

In order to use the Office Conversion Service, we must first authenticate our request. For this purpose, we use the internal D365FO Azure AD Service 2 Service authentication method rather than the authorization key used in the standard Electronic Reporting Framework implementation. The key is retrieved from Dynamics Globalization Service.

Part 2 » Prepare and send the request to convert Word or Excel to PDF

Now that we know how to authenticate in the Office Conversion Service, we can prepare the request with the appropriate query parameters, headers, and body with the binary source file to be converted to a PDF. For simplicity purpose, we will use two sample resource files (Word and Excel) for in the following example.

Final Thoughts and Ideas for Improvement

Life can get a lot easier for an X++ developer if you can figure out what’s going on in the background and what D365FO is doing. If you use Fiddler as a man-in-the-middle, you can find out many exciting things and how to use what is already there in D365FO to achieve scenarios you may not even know you can do and use the functionality in your solutions. In this article the solution is just an example, but you can still use it for conversion and improve it with the following upgrades:

  • Make it a generic solution to convert any Word or Excel file to PDF
  • Cache the authentication token for a short period of time, or at least until it expires
  • Implement more robust exception handling

If you have any new ideas or discoveries, please share them as well.

Resources

Download Electronic Reporting configurations for Sales Invoice
Download Fiddler Session Archive file
Download solution project

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Docentric respects your privacy. Learn how your comment data is processed >>

Docentric respects your privacy. Learn how your comment data is processed >>