Data integration file storage (Amazon S3)

caution

  Prerequisites:
Access to the Configuration > Configuration > System (CM006) menu.

File storage service

When the Data integration file storage option is enabled in the Configuration > Configuration > System (CM006) menu, this indicates that the system is set up to use the Amazon Simple Storage Service (Amazon S3) as storage location for electronic files (objects) associated with the functions of the DIINTERFACE table.

Additionally, in order to support the storage of these files in Amazon S3, we use AWS Transfer Family. This service provides managed support for business-to-business (B2B) file transfer through SFTP, allowing objects to be directly moved to an Amazon S3 bucket in a highly secure manner by customer and by contracted environment.

info

For customers that use SoftExpert Hosting and want to use this feature, we suggest contacting SoftExpert's sales team for details on our SFTP service.

Object metadata guidelines

When sending files through Secure File Transfer Protocol (SFTP), it is imperative to observe specific guidelines related to the metadata of the objects being transferred. See an example of these guidelines below:

• When creating a name for your object, you can use Unicode characters with UTF-8 encoding.
• Object key names are case-sensitive.
• Object key names can be up to 1,024 characters long.
• An object prefix cannot start with a slash ("/").
• Add a slash ("/") to the object prefix after the first character is interpreted as a directory. These directory segments are limited to 255 characters.

The Amazon S3 data template is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of sub-buckets or sub-folders. At the However, it is possible to assume a logical hierarchy using prefixes and key name delimiters, just as the Amazon S3 console does. The Amazon S3 console supports the concept of folders.

Let's assume that your SFTP (admin-created) has four objects with the following object keys:

• Development/Projects.xls
• Finance/statement1.pdf
• Private/taxdocument.pdf
• s3-dg.pdf

The console uses key name prefixes (Development/, Finance/ and Private/) and the delimiter ("/") to display a folder structure. The s3-dg.pdf key does not have a prefix, so its object appears directly at the root level of the bucket. When the Development/ folder is opened, the Projects.xlsx object is shown.

info

• For naming convention details, go to Creating object key names (AWS portal).
• Make sure the object key name also supports protocols other than S3. Specifically, avoid special characters that may not be supported in other protocols.

Examples of characters to avoid

Avoid the following characters in an object key name, due to significant special handling to ensure consistency across all applications.

• Backslash \

• Left curly bracket {

• Non-printable ASCII characters (128-255 decimal characters)

• Circumflex ^

• Right curly bracket }

• Percent sign %

• Grave accent `

• Right square bracket ]

• Quotation marks

• Greater-than sign >

• Left square bracket [

• Tilde ~

• Less-than sign <

• Hashtag character #

• Vertical bar / pipe |

For example, when importing electronic files through the DCDOC interface, into DSFIELD03, it is crucial to avoid using these characters. The DSFIELD03 is the field in which you must specify the path of the electronic file in the SFTP, which consequently influences the file within the Amazon S3 bucket.

Valid example inserted in DSFIELD03	Invalid example inserted in DSFIELD03
Images/Image.png	{User/Document.doc}
Document-2023.doc	File^Name.txt
File123.pdf	{"Special/File.xlsx"}
User123/File.doc	Percentage%Report.pdf

caution

Be sure to avoid the characters listed in the "Invalid example" table when naming your object keys. These characters can cause consistency issues and inappropriate handling in the applications and systems involved. Use only permitted alphanumeric characters and symbols to ensure the integrity and correct functioning of the system.