Guide to Data Aggregation Write Operations

Scenario and Value

In integration scenarios involving orders, inventory, expense details, or other line-based business data, a source system may return documents containing multiple nested arrays. Transmitting this data directly can prevent records from being written as intended because of message structures, grouping boundaries, or target-document line limits. Multidimensional data therefore needs to be flattened into a two-dimensional structure before grouped aggregation.

The Qeasy iPaaS platform supports these operations through visual configuration. Users can define header groups, body groups, calculated fields, and the target body name so that multiple detail lines can be organized into fewer target documents. This approach is suitable for reducing detail transmission volume, combining records with the same dimensions, and handling target-document line limits.

Integration Architecture and Data Flow

The process consists of three stages.

The first stage is data flattening. A multidimensional source document may contain a header, several detail arrays, or even deeper structures. Multidimensional data flattening rules are configured in the source platform, where nested arrays are expanded into a unified two-dimensional detail structure to prepare the data for subsequent grouping.

The second stage is grouped aggregation. Users identify header fields that represent business-document boundaries and specify the body fields that should participate in grouping. Header group fields determine the aggregation scope, while body group fields determine which detail lines belong to the same group.

The third stage is target writing. After aggregation, the platform writes the result to the target platform. Target-field references must conform to the new data structure: header fields are mapped directly, while the merged body is referenced through the configured array key.

Implementation Considerations

The core aggregation configuration includes the following parameters:

• headerGroup: Defines the header grouping fields used to identify the document aggregation scope.
• bodyGroup: Defines the body grouping fields used to determine which detail lines are aggregated into the same group.
• bodyName: Specifies the array key used to store the merged body data in the aggregation result.
• targetBodyName: Specifies the detail-body name in the target document.
• bodyMaxLine: Sets the maximum number of lines in a target detail body. If aggregated data exceeds the limit, the platform splits it according to the target body.
• calculate: Defines the fields and expressions used for aggregation. Supported expressions include $sum, $avg, $min, $max, $push, $addToSet, $first, and $last.

$sum calculates a total, while $avg calculates an average. $min and $max select the minimum and maximum values. $push adds values to an array without actively removing duplicates, whereas $addToSet excludes duplicate values. $first and $last select the first or last document data according to the source-document order. The appropriate expression depends on the business meaning of each field and cannot be determined from its data type alone.

After configuration, test samples should include records with the same grouping dimensions, different detail values, and a line count above the target limit. Verify grouping boundaries, accumulated results, body array keys, and field mapping after splitting to prevent unrelated business data from being combined because of incomplete dimensions.

Best Practices

First, grouping fields should be stable identifiers of business documents or classification dimensions. Confirm that source values have consistent formats and account for null values and precision differences. Second, before using $sum, standardize units, precision, and rounding rules for amounts and quantities. Define the required ordering basis for average and first- or last-value calculations.

Third, use $push and $addToSet carefully. Use $push when every value must be retained and $addToSet when a deduplicated set is required. Finally, correctly reference the array key defined by bodyName in target-field mapping and design splitting logic based on bodyMaxLine.

Before production, validate boundary samples for maximum-line handling, empty body data, and duplicate-value rules. If a header must map to only one target document, ensure that the header grouping conditions uniquely identify that document. If document splitting is allowed, define how header fields are copied or inherited across the resulting documents.