Passthrough step loss

[umkaline]

In a specific scenario, passthrough can lose one of the previous steps data.

STR:
Flow with two consequent steps with new sailor (components deployed on 30th of January) components and then Transformation step in any place in the flow after them. Such a scenario will cause that after the Transformation step passthrough will lose data from the first step of the new sailor steps pair.

The easiest way to reproduce is to build flow with the next configuration:

• Webhook
• NodeJS component
• Transformation
• Replay

And if you specify $ in the reply body, you will see that there is no step_1 in the passthrough left.

In response you will see:

{
	"1": 1,
	"elasticio": {
		"step_2": {
			"body": {
				"result": "Hello world!"
			}
		},
		"step_3": {
			"id": "87f54d4f-4154-4d89-9f77-fe73ff6f6345",
			"attachments": {},
			"body": {
				"1": 1
			},
			"headers": {},
			"metadata": {}
		}
	}
}

[khanzadyan]

@umkaline and @juhrmacher we need some status on this to reply to the customer.

[kirill-levitskiy]

It looks, like the problem of passing data through steps with components with different versions of the sailor.

As a workaround was found the next case:

1. Add jsonata-component step right after the 1st step;
2. Define a new configuration for next steps based on mapped body from the jsonata-component step (to use passthrough.step_2 instead of passthrough.step_1).

Required additional investigation with detailed logs.

[jhorbulyk]

In version 20.02 of the platform, a feature was added:

A step must not put his own output message into the passthrough object.

New sailor version has introduced 2.6.0

This is likely a contributing factor to this issue.

Additionally, the change released in 20.02 is backwards incompatible. It was previously considered a best practice to reference the previous step via passthrough as opposed to directly.

*e.g. If going into step_4 you needed the foo property from step_3 it is better to reference elasticio.step_3.body.foo instead of simply foo.

The benefits were that:

• If you add a new step between step_3 and step_4 the mapping would not break.
• When you had a sequence of order agnostic steps, you did not have to remember the order.

However, now with the above change, flows which previously used elasticio.step_3.body.foo will no longer reference as expected.

[luzhok]

We've found the same problem, I think, and here the result of our investigation as well
the problem is actual for the next case: Step1_newSailor->Step2_newSailor->Step3_oldSailor
(newSailor >= 2.6.0)
In this case envar NO_SELF_PASSTHROUGH for our steps:
Step1_newSailor (TRUE)->Step2_newSailor (FALSE)->Step3_oldSailor (FALSE)

In this case data from Step1 is lost, because Step1 didn't add his own data to passthrough, as a component with "NO_SELF_PASSTHROUGH": true AND Step2 didn't add data from Step1 to passthrough, as component with "NO_SELF_PASSTHROUGH": false (Step2 added his own data to passthrough). As a result, data from Step1 is lost for all further steps

[luzhok]

After fixing the issue to check the next case:

1. Create a flow Webhook -Email (or any component with mapper and old sailor!!!)-Reply
2. For Webhook-step set sample with parameter: "foo":"somevalue"
3. Choose this Webhook parameter "foo" as the input value for Email-step (all required fields)
4. Choose this Webhook parameter "foo" as the input value for Reply-step
5. Run the flow
6. Hit Webhook with URL-parameter: "foo":"oksana@elastic.io"

AR: we received empty object as an answer for Webhook from Reply-step. Data from Step_1 is lost

ER: everything works properly. And we received "oksana@elastic.io" as an answer for Webhook from Reply-step