Uploaded image for project: 'CloverDX'
  1. CloverDX
  2. CLO-17571

ListFiles does not stream results

    XMLWordPrintable

    Details

    • Story Points:
      13
    • QA Testing:
      UNDECIDED
    • Sprint:
      Green Sprint 100, Green Sprint 101, Green Sprint 102

      Description

      Some protocols, for example S3, use paging to return the directory listing. But IOperationHandler.list() returns a List, so it needs to read all the files first. It would be possible to return java.nio.file.DirectoryStream<T> instead.

      This would save memory required to store huge directory listings and improve performance.

      We shouldn't break backward compatibility. We could probably add a new interface to handlers that allow streaming in directory listing.

      • S3: com.amazonaws.services.s3.AmazonS3.listNextBatchOfObjects(ObjectListing)
      • local files: java.nio.file.Files.newDirectoryStream(Path)
      • SMB2: com.hierynomus.smbj.share.Directory.iterator(Class<F>, String)

      We should rewrite both directory listing and wildcard resolution to support streaming. For wildcard resolution, it should be sufficient to test if the result is empty or not, we don't need to know the exact size.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              krivanekm Milan Krivanek
              Reporter:
              krivanekm Milan Krivanek
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 weeks, 2 days, 3 hours
                  2w 2d 3h