Filter step breaks previews when it fails at runtime
Assignee

Reporter

Sprint
None
Description
Steps to reproduce
None
Attachments
1
Activity
Show:

Tomas Horsky June 15, 2023 at 8:37 AM
Filter by formula step now does not fail the job, but marks column as rejected.
When an error is created in FILTER at runtime while evaluating condition, the preview is not available. Users cannot see the data with which their step failed so it's difficult to figure out what's wrong.
I have a filter step with this condition: $Integer > 0
When I save the step, I see a global error instead of preview. The step on right side is not marked as invalid.
The step failed because some rows had value "No value" (null) in column $Integer. So the condition failed to be evaluated.
Improve this situation:
The filter step should not fail anymore. Instead it should create REJECT_ERROR on all columns used in the condition.
The error must be REJECT_ERROR, not RUNTIME_ERROR. This is because we want to keep values of all the columns - filter step should not modify the data.
Similar marking of 'all used input fields' is done in step 'validate formula'.
In a similar manner, revisit the input field validity checking. All input fields used in condition should probably be marked with REJECT_ERROR when any of them is invalid.