Improved metadata concatenation
Assignee
Reporter
Pavel Salamon
Pavel SalamonSprint
None
Description
Steps to reproduce
None
Activity
Show:
Pavel Salamon August 2, 2016 at 4:02 PMEdited
CrossJoin and SalesforceBulkWriter use the new algorithm.
If all input metadata are fixlen, result is fixlen and field sizes are kept.
Fixed
Details
Priority
Major
Fix versions
Branch
QA Testing
JUnit test
QA Test Identification
testMetadataConcat_delim;testMetadataConcat_fixlen;testMetadataConcat_mixed
Components
Created August 2, 2016 at 1:06 PM
Updated September 12, 2023 at 9:46 AM
Resolved August 2, 2016 at 4:02 PM
New components often need to propagate concatenation of several different metadata. The concatenation algorithm should be in shared utility method.
Currently, CrossJoin and SalesforceBulkWriter use the concatenation method, but the result is not robust enough (fixlen metadata are not handled well, delimiters are not normalized etc.).
The result of new concatenation:
delimited metadata with field delimiter set (delimiter taken from first input metadata)
result record properties same as record properties from first input metadata
result field properties same as input fields (except size and delimiter)
no size attribute on any field
no explicit delimiter on any field
This change is backwards incompatible because CrossJoin already propagates using the original algorithm (copy first input metada, then add a copy of all fields from following input metadata). However, the new algorithm only produces different metadata in situations where the original algorithm's result is unusable in practice, so it should be ok.