Data integration file storage (Amazon S3)

caution

  Prerequisites:
Access to the Configuration > Configuration > System (CM006) menu

File storage service

When the DATA INTEGRATION FILE STORAGE option is enabled in the Configuration > Configuration > System (CM006) menu, this indicates that the system has been set up to use the Amazon Simple Storage Service (Amazon S3) as a storage location for electronic files (objects) associated with the features of the DIINTERFACE table.

Additionally, to support the storage of these files in Amazon S3, we use AWS Transfer Family. This service provides managed support for business-to-business (B2B) file transfer using the SFTP, allowing direct movement of objects to an Amazon S3 bucket in a highly secure manner, per client and per contracted environment.

info

For customers that use SoftExpert Hosting and that wish to use this resource, we suggest contacting the SoftExpert Sales team to request details on our SFTP service.

danger

It is important to stress that the SFTP must not be used as a file repository. In order to ensure security and optimize the use of resources, we have adopted a 15-day retention policy for all files stored in the SFTP. After this period, the files are automatically deleted.

Object metadata guidelines

When sending files via Secure File Transfer Protocol (SFTP), it is crucial to observe specific guidelines regarding the metadata of the objects being transferred. Here is an example of these guidelines:

• When creating a name for your object, you can use Unicode characters with UTF-8 encoding.
• Object key names are case sensitive;
• Object key names can be up to 1,024 characters long;
• An object prefix cannot begin with a slash ("/");
• Add a slash ("/") to the object prefix after the first character is interpreted as a directory. These directory segments are limited to 255 characters.

The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of sub-buckets or subfolders. However, you can assign a logical hierarchy by using key name prefixes and delimiters, as the Amazon S3 console does. The Amazon S3 console supports the concept of folders.

Let's assume that your SFTP (admin-created) has four objects with the following object keys:

• Development/Projects.xls
• Finance/statement1.pdf
• Private/taxdocument.pdf
• s3-dg.pdf

The console uses key name prefixes (Development/, Finance/ and Private/) and the delimiter ("/") to display a folder structure. The s3-dg.pdf key does not have a prefix, so its object appears directly at the root level of the bucket. When the Development/ folder is opened, the Projects.xlsx object is shown.

info

• For naming convention details, go to Creating object key names (AWS portal).
• Make sure the object key name also supports protocols other than S3. Specifically, avoid special characters that may not be supported in other protocols.

Examples of characters to avoid

Avoid the following characters in an object key name, due to significant special handling to ensure consistency across all applications.

• Backslash \

• Left curly bracket {

• Non-printable ASCII characters (128-255 decimal characters)

• Circumflex ^

• Right curly bracket }

• Percent sign %

• Grave accent `

• Right square bracket ]

• Quotation marks

• Greater-than sign >

• Left square bracket [

• Tilde ~

• Less-than sign <

• Hashtag character #

• Vertical bar / pipe |

For example, when importing electronic files through the DCDOC interface, into DSFIELD03, it is crucial to avoid using these characters. The DSFIELD03 is the field in which you must specify the path of the electronic file in the SFTP, which consequently influences the file within the Amazon S3 bucket.

Valid example inserted in DSFIELD03	Invalid example inserted in DSFIELD03
Images/Image.png	{User/Document.doc}
Document-2023.doc	File^Name.txt
File123.pdf	{"Special/File.xlsx"}
User123/File.doc	Percentage%Report.pdf

caution

Be sure to avoid the characters listed in the "Invalid example" table when naming your object keys. These characters can cause consistency issues and inappropriate handling in the applications and systems involved. Use only permitted alphanumeric characters and symbols to ensure the integrity and correct functioning of the system.