Uploaded image for project: 'CloverDX'
  1. CloverDX
  2. CLO-2054

XMLExtract, JSONExtract to support list types

    XMLWordPrintable

    Details

    • Branch:
    • QA Testing:
      Graph automated test
    • QA Test Identification:
      XMLExtract_list_support_CLO-2054
    • Additional information:
      Hide

      Documentation: http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/part-component-reference.html
      Video: https://intranet.javlin.eu/video/xmlextract-jsonextract-lists/
      Motivation: Working with lists in XMLExtract and JSONExtract used to be complicated and not directly supported. The goal of this feature is to simplify this process.
      This is what a list looks like in xml and json (identifier types, a part of a slightly modified response from google geocoding api):

      Before this functionality, one had to use 2 additional components to map a list to an output field. This worked by attaching an id to each item of the list an then using a denormalizer to create a list using the equality of the ids. If mapping together with another field, a joiner had to be used to join the list and the other field - using ids again.
      This is a graph that was necessary to map lists:

      XMLExtract required 2 sets of metadata on its output ports (screenshot from View data dialog):

      With this new functionality, a list can be mapped directly in the mapping dialog - the same way as non-list fields. Other functionality of the mapping dialog was adapted to lists - e.g. - dragging an xml field to the metadata viewer creates a new metadata field with "list" container type. Also, if a list field is mapped to a non-list metadata field, a warning i shown - the graph will still work, but the metadata field will only contain the last item of the list.
      A graph needed to map lists doesn't need any additional components now:

      With list support, XMLExtract requires only the simplest metadata:

      The attached listTypesProject.zip shows both possible ways of mapping lists.

      Show
      Documentation: http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/part-component-reference.html Video: https://intranet.javlin.eu/video/xmlextract-jsonextract-lists/ Motivation: Working with lists in XMLExtract and JSONExtract used to be complicated and not directly supported. The goal of this feature is to simplify this process. This is what a list looks like in xml and json (identifier types , a part of a slightly modified response from google geocoding api): Before this functionality, one had to use 2 additional components to map a list to an output field. This worked by attaching an id to each item of the list an then using a denormalizer to create a list using the equality of the ids. If mapping together with another field, a joiner had to be used to join the list and the other field - using ids again. This is a graph that was necessary to map lists: XMLExtract required 2 sets of metadata on its output ports (screenshot from View data dialog): With this new functionality, a list can be mapped directly in the mapping dialog - the same way as non-list fields. Other functionality of the mapping dialog was adapted to lists - e.g. - dragging an xml field to the metadata viewer creates a new metadata field with "list" container type. Also, if a list field is mapped to a non-list metadata field, a warning i shown - the graph will still work, but the metadata field will only contain the last item of the list. A graph needed to map lists doesn't need any additional components now: With list support, XMLExtract requires only the simplest metadata: The attached listTypesProject.zip shows both possible ways of mapping lists.

      Description

      It would be nice to have support for list and maps in XMLExtract and JSONExtract for reading lists/maps of primitive types.

      Desired metadata:

      <Metadata id="Metadata5" previewAttachmentCharset="ISO-8859-1">
      	<Record fieldDelimiter="|" name="AddressWithLocation" recordDelimiter="\r\n" type="delimited">
      		<Field name="long_name" type="string"/>
      		<Field name="short_name" type="string"/>
      		<Field name="typesValue" type="string" containerType="list"/>
      	</Record>
      </Metadata>
      

      Sample mappings:

      JSON:

      "address_components" :
      	{
      		"long_name" : "Radnor - Fort Myer Heights",
      		"short_name" : "Radnor - Fort Myer Heights",
      		"types" : [ "neighborhood", "political" ]
      	}
      ]
      

      should be mapped by code

      <Mapping element="address_components" outPort="1" 
      	xmlFields="{}long_name;{}short_name;{}types"
      	cloverFields="long_name;short_name;typesValue">
      </Mapping>
      

      XML:

      <address_components>
      	<long_name>Providence</long_name>
      	<types>administrative_area_level_3</types>
      	<types>political</types>
      	<short_name>Providence</short_name>
      </address_components>
      

      should be mapped by code

      <Mapping element="address_components" outPort="1" 
      	xmlFields="{}long_name;{}short_name;{}types"
      	cloverFields="long_name;short_name;typesValue">
      </Mapping>
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              musilj Jiri Musil (Inactive)
              Reporter:
              ulrychj Jan Ulrych (Inactive)
              Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: