Import/Export Container Data Fields Using D365FO Data Management

If you are reading this article, it is probably because you have faced the import/export issue with table fields of type container when using the D365FO Data Management framework. As you have probably noticed, a D365FO data entity container field is by default simply skipped during the export procedure!

Microsoft did not do the best job of documenting the correct approach for solving this issue. It is written only that if an entity has container fields and these fields need to be exported, the entity must implement the getFieldsToBeConvertedToFile() method (see Tips and tricks in the Build and consume data entities tutorial). But other important pieces of information are missing, which is why we decided to explore the import/export logic around D365FO containers by debugging the DMFEntityBase class. We will describe our findings in this article.

Let us set the stage by specifying what D365FO container fields are: they are used for storing binary content like images and documents, and for storing packed objects, like packed queries. An example of binary content is a worker or product image, and an example of a packed object are packed Print management settings.

The two container content types (binary content and packed objects) require different steps for a successful export. We will describe the D365FO data entity container field export rules by using two different examples for these two content types.

Container field with binary content

We will demonstrate how to export the D365FO container field with binary content on a custom DocElectronicSignatureAppearance data entity (Docentric Electronic Signature Appearance). Its main data source is a table named DocElectronicSignatureAppearance that has a SignatureImage field of type container:

Learn more about the Docentric Electronic Signature Appearance data entity >>

To export/import this container field:

  1. Add a new string field to the staging table.
  2. Give this field the same name as the container field name, with the ‘FileName‘ suffix.
  3. Use the FileName EDT for this field.

That’s it! You don’t need anything else for the containers with binary content. When the Data Management framework finds a field with the FileName EDT and suffix during the export, it will take the matching container field and save its content to a separate file. If such a field is found during the import, the framework will know how to identify the matching file and will then save its content to a matching container field. See the Import/Export the package chapter below for instructions on the export/import process.

Container fields with packed objects

Container fields with packed objects (e.g. a packed class or packed query) are a bit more complex. For successful export, this kind of content must first be converted into a base64 string. Then, the export procedure must know which base64 string fields to consider for the export. And finally, as in the case of a container with binary content, a special field (FileName EDT and suffix) must be added to the staging table.

We will illustrate this with the Docentric Print Management Settings Package data entity. Its main data source is a standard D365FO Print management settings table (PrintMgmtSettings) that contains 2 container fields: QueryPacked – a print management condition stored as a packed query, and PrintJobSettings – print destination settings stored as a packed object.

Here are the steps for each of these 2 fields:

  1. Add one unmapped Memo field to the data entity (unmapped string field, for example use the Notes EDT to force the Memo type). There is no special rule for naming this field. We will use it to store the base64 string representation of the container field, so we simply used the name of the container field with the Str suffix. See the fields PrintJobSettingsStr and PrintSettingsQueryPackedStr below:
  2. Implement the logic for converting the container field content into the base64 string during the data export. This code snippet is added to the postLoad() data entity method:
  3. Implement the logic for converting the base64 string into the container field content during the import. We added the code snippet below to the defaultRow() data entity method:
  4. As in the case of a container with binary content, add one string field to the staging table, for exporting the Memo field from step 1. The field will have the FileName EDT and the suffix ‘FileName‘ added to the name of the Memo field. As a result, we see the PrintJobSettingsStrFileName and PrintSettingsQueryPackedStrFileName fields below:

  5. Implement the getFieldsToBeConvertedToFile() method on the staging table, where you need to list all memo fields added in step 1. Here is our implementation:

Import/Export the package

Now when we have prepared our data entities to support the container fields export/import, we are ready to perform the export/import procedure. We will use the Docentric Electronic Signature Appearance data entity in this example.

Export procedure

The contents of the container fields are exported as separate files, one file for each record and each container field equipped for export. Therefore, we want to export into a data package that will contain all these additional files.

After adding the data entity and selecting the preferred export format (we recommend using XML-Element), make sure that the Generate data package option is set to “Yes” (see the last image above).

After the export is finished, proceed with the download of the package:

The downloaded data package is a .zip file. When you unpack it, you will find the exported XML-Element file (see Docentric Electronic Signature Appearance.xml below) and a Resource folder with all container contents exported as separate files.

The exported XML-Element file contains an XML element with the SignatureImageFileName tag that contains a GUID:

This GUID points to a file in the Resources folder that contains the exported container content:

Import procedure

The images below illustrate the process of importing the package:

After clicking the Upload and add button, we will point to the .zip file created during the export. Once we select the package file (.zip), a data entity is automatically populated based on the information found in the Manifest.xml file from the .zip file.

Note that the import/export procedure for the other data entity used in previous examples, Docentric Print Management Settings Package, is the same.

There is more: Overcome the 32 KB limitation in Data entity Memo field size

When you export the content of a string field with more than 32,768 characters and import that content to a string field, the imported string will be truncated. This is a known limitation according to the Microsoft documentation. If you need to import strings larger than 32,768 characters, Microsoft suggests (without any explanation) that you use container entity fields. Which of the two cases described earlier would be a better fit for this situation? A container with binary content, or a container with packed object? Actually, it will be something in between: we will need an additional field in the staging table, plus we will need to implement the getFieldsToBeConvertedToFile() method on a staging table, but we do not need additional data entity fields and conversions.

We will demonstrate the necessary steps by using the example of the DocSysEmailMessageTablePackageEntity (Docentric Organization Email Messages Package) data entity. Docentric implemented this data entity to support the import/export of the Organization email templates that are stored in the SysEmailMessageTable table. This table contains two Memo fields: Mail and XSLTMail, which carry the content of the email body.

To support the transfer of the email bodies larger than 32KB, we did the following:

  1. We added 2 string fields to the related staging table (DocSysEmailMessageTablePackageStaging), one for the Mail and one for the XSLTMail field. They are of the FileName EDT and have names suffixed with ‘FileName‘:
  2. We implemented the getFieldsToBeConvertedToFile() method on the staging table where we listed these two fields. Here is the code snippet with our implementation:

That is all you need to do to export content of Memo fields larger than 32 KB.

Summary

When it comes to exporting and importing the D365FO container fields, we recognize 3 different cases. See our cheat sheet below and remember to export as a package, where each container content will be saved as a separate file.

Staging:
Add string field with ‘FileName’ suffix and FileName EDT
Staging:
Implement the getFieldsToBeConvertedToFile()
method
Data entity:
1. Add Memo field for the base64 string
2. Implement the conversion between base64 string and container during export/import
Containers with binary content YES
Containers with packed object YES YES YES
Strings longer than 32K characters YES YES

3 thoughts on “Import/Export Container Data Fields Using D365FO Data Management

  1. hi ,

    I see you are overwritting the postload method on the data entity. Are you still able to skip the staging on the data entity for export?

    I am just thinking of the performance for the export.

    Thank you for the documentation!

    1. Hi Guillaume,

      PrintMgmtSettings.PrintJobSettings field is a container that we need to transform into an unmapped string field for the export purpose. The only way to populate an unmapped field during export is in the postLoad() method, so we didn’t have much choice there.

      It is true, as you pointed out, that postLoad() requires staging (it can’t be skipped in this case) and for sure influences the performance. I think that it all depends on the use-case. Our example is based on the Docentric Print Management settings data entity, where companies usually don’t have large amount of data, and on the other hand don’t have any OOTB way of migrating the settings among different environments.

      Kind regards,
      Sanja

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Docentric respects your privacy. Learn how your comment data is processed >>

Docentric respects your privacy. Learn how your comment data is processed >>