Uploaded image for project: 'CloverDX'
  1. CloverDX
  2. CLO-6335

CustomJavaComponent improvements

    XMLWordPrintable

    Details

    • QA Testing:
      UNDECIDED
    • Additional information:
      Hide

      Motivation:
      There are times when you need certain functionality that none of the clover components or their combinations provide. You need to develop your own component. Until now developing your own component was too difficult. You had to mess with the eclipse extension system and provide various descriptors in various xml files. Most of the time was spent spent studying documentation and fixing extension points instead of developing the component's algorithm.
      Now you can use CustomJavaComponent and skip all the annoying stuff. All you need to do is add a component to the graph and directly write the algorithm.

      Example:
      Let's see how a simple component developed using CustomJavaComponent looks like. Our custom component is called Zipper and it reads file URLs from input port and zips all those files into a .zip file using standard Java API.

      We've put CustomJavaComponent in a subgraph and connected an input edge.

      The input edge defines the metadata of the subgraph. Metadata have a single field that is supposed to contain file URLs.

      Configuration of CustomJavaComponent is very simple. We have created a custom property called fileURL and exported it as a subgraph parameter. That allows users of the subgraph to specify an output .zip file.
      We also created our class called Zipper that defines component's algorithm.

      The Zipper class looks like this:

      Zipper.java
      import java.io.IOException;
      import java.io.InputStream;
      import java.util.zip.ZipEntry;
      import java.util.zip.ZipOutputStream;
      
      import org.jetel.component.AbstractGenericTransform;
      import org.jetel.data.DataRecord;
      import org.jetel.exception.ComponentNotReadyException;
      import org.jetel.exception.ConfigurationStatus;
      import org.jetel.exception.ConfigurationStatus.Priority;
      import org.jetel.exception.ConfigurationStatus.Severity;
      
      /**
       * Reads all input records, extracts file urls from them and compresses those files into a zip.
       * 
       * @author salamonp ([email protected]) (c) Javlin, a.s. (www.cloveretl.com)
       *
       */
      public class Zipper extends AbstractGenericTransform {
      	
      	// input metadata field
      	private static final String INPUT_URL_FIELD_NAME = "url";
      	
      	// component attribute
      	private final static String OUTPUT_FILE_URL = "fileURL";
      	
      	// internal buffer size
      	private final static int BUFFER_SIZE = 2048;
      	
      	// variables for output
      	private String outputFileURL;
      	private ZipOutputStream zipOutput;
      
      	@Override
      	public void execute() throws IOException {
      		DataRecord record;
      
      		// read all input records
      		while ((record = readRecordFromPort(0)) != null) {
      			String inputFile = record.getField(INPUT_URL_FIELD_NAME).getValue().toString();
      			
      			// create zip entry inside the zip for each input file
      			ZipEntry entry = new ZipEntry(getFile(inputFile).getName());
      			zipOutput.putNextEntry(entry);
      			
      			// copy all bytes from input file to output zip
      			try (InputStream is = getInputStream(inputFile)) {
      				int count;
      				byte data[] = new byte[BUFFER_SIZE];
      				while ((count = is.read(data, 0, BUFFER_SIZE)) != -1) {
      					zipOutput.write(data, 0, count);
      				}
      			}
      		}
      		
      	}
      
      	@Override
      	public ConfigurationStatus checkConfig(ConfigurationStatus status) {
      		super.checkConfig(status);
      		
      		outputFileURL = getProperties().getStringProperty(OUTPUT_FILE_URL);
      		if (outputFileURL == null || outputFileURL.length() == 0) {
      			status.add("File URL property is missing", Severity.ERROR, getComponent(), Priority.NORMAL, OUTPUT_FILE_URL);
      		}
      
      		return status;
      	}
      
      	@Override
      	public void preExecute() throws ComponentNotReadyException {
      		super.preExecute();
      		
      		// prepare output stream
      		outputFileURL = getProperties().getStringProperty(OUTPUT_FILE_URL);
      		try {
      			zipOutput = new ZipOutputStream(getOutputStream(outputFileURL, false));
      		} catch (IOException e) {
      			throw new ComponentNotReadyException(e);
      		}
      	}
      
      	@Override
      	public void postExecute() throws ComponentNotReadyException {
      		super.postExecute();
      		
      		// close the output stream
      		if (zipOutput != null) {
      			try {
      				zipOutput.close();
      			} catch (IOException e) {
      				throw new ComponentNotReadyException(e);
      			}
      		}
      	}
      }
      

      Notice that the execute() method which does all the work is not very long yet it does everything we need it to do.
      And that's it, the subgraph is ready to be used as any other component now. Using CustomJavaComponent developing a component is now a matter of hours, not days.

      The component could be further extended:

      • Robustness to null values and non-existing files
      • Support for directories (only files work now)

      Project with this executable subgraph is available in this archive: cjc_highlight.zip

      Show
      Motivation : There are times when you need certain functionality that none of the clover components or their combinations provide. You need to develop your own component. Until now developing your own component was too difficult. You had to mess with the eclipse extension system and provide various descriptors in various xml files. Most of the time was spent spent studying documentation and fixing extension points instead of developing the component's algorithm. Now you can use CustomJavaComponent and skip all the annoying stuff. All you need to do is add a component to the graph and directly write the algorithm. Example : Let's see how a simple component developed using CustomJavaComponent looks like. Our custom component is called Zipper and it reads file URLs from input port and zips all those files into a .zip file using standard Java API. We've put CustomJavaComponent in a subgraph and connected an input edge. The input edge defines the metadata of the subgraph. Metadata have a single field that is supposed to contain file URLs. Configuration of CustomJavaComponent is very simple. We have created a custom property called fileURL and exported it as a subgraph parameter. That allows users of the subgraph to specify an output .zip file. We also created our class called Zipper that defines component's algorithm. The Zipper class looks like this: Zipper.java import java.io.IOException; import java.io.InputStream; import java.util.zip.ZipEntry; import java.util.zip.ZipOutputStream; import org.jetel.component.AbstractGenericTransform; import org.jetel.data.DataRecord; import org.jetel.exception.ComponentNotReadyException; import org.jetel.exception.ConfigurationStatus; import org.jetel.exception.ConfigurationStatus.Priority; import org.jetel.exception.ConfigurationStatus.Severity; /** * Reads all input records, extracts file urls from them and compresses those files into a zip. * * @author salamonp ([email protected]) (c) Javlin, a.s. (www.cloveretl.com) * */ public class Zipper extends AbstractGenericTransform { // input metadata field private static final String INPUT_URL_FIELD_NAME = "url" ; // component attribute private final static String OUTPUT_FILE_URL = "fileURL" ; // internal buffer size private final static int BUFFER_SIZE = 2048; // variables for output private String outputFileURL; private ZipOutputStream zipOutput; @Override public void execute() throws IOException { DataRecord record; // read all input records while ((record = readRecordFromPort(0)) != null ) { String inputFile = record.getField(INPUT_URL_FIELD_NAME).getValue().toString(); // create zip entry inside the zip for each input file ZipEntry entry = new ZipEntry(getFile(inputFile).getName()); zipOutput.putNextEntry(entry); // copy all bytes from input file to output zip try (InputStream is = getInputStream(inputFile)) { int count; byte data[] = new byte [BUFFER_SIZE]; while ((count = is.read(data, 0, BUFFER_SIZE)) != -1) { zipOutput.write(data, 0, count); } } } } @Override public ConfigurationStatus checkConfig(ConfigurationStatus status) { super .checkConfig(status); outputFileURL = getProperties().getStringProperty(OUTPUT_FILE_URL); if (outputFileURL == null || outputFileURL.length() == 0) { status.add( "File URL property is missing" , Severity.ERROR, getComponent(), Priority.NORMAL, OUTPUT_FILE_URL); } return status; } @Override public void preExecute() throws ComponentNotReadyException { super .preExecute(); // prepare output stream outputFileURL = getProperties().getStringProperty(OUTPUT_FILE_URL); try { zipOutput = new ZipOutputStream(getOutputStream(outputFileURL, false )); } catch (IOException e) { throw new ComponentNotReadyException(e); } } @Override public void postExecute() throws ComponentNotReadyException { super .postExecute(); // close the output stream if (zipOutput != null ) { try { zipOutput.close(); } catch (IOException e) { throw new ComponentNotReadyException(e); } } } } Notice that the execute() method which does all the work is not very long yet it does everything we need it to do. And that's it, the subgraph is ready to be used as any other component now. Using CustomJavaComponent developing a component is now a matter of hours, not days. The component could be further extended: Robustness to null values and non-existing files Support for directories (only files work now) Project with this executable subgraph is available in this archive: cjc_highlight.zip

      Description

      Additional improvements of the CustomJavaComponent. Themes:

      • deployment on Server
      • running Hercules in other sandbox (even in local runtime)

        Attachments

        1. cjc_highlight.zip
          9 kB
        2. cjc_highlight1.png
          cjc_highlight1.png
          17 kB
        3. cjc_highlight2.png
          cjc_highlight2.png
          23 kB
        4. cjc_highlight3.png
          cjc_highlight3.png
          26 kB

          Issue Links

            Activity

              People

              Assignee:
              salamonp Pavel Salamon
              Reporter:
              urbanj Jaroslav Urban (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: