Trigger VS Debug

Azure Data Factory (ADF) offers 2 ways of executing a pipeline:

1. Trigger  (on-demand, or Manual, and available trigger types (scheduled, tumbling, storage/custom event))
   
2. Debug: As you author using the pipeline canvas, you can test your activities using the Debug capability. When you do test runs, you don’t have to publish your changes to the service before you select Debug. This feature is helpful in scenarios where you want to make sure that the changes work as expected before you update the workflow. Source

Selecting Debug actually runs the pipeline. For example, if the pipeline contains copy activity, the test run copies data from source to destination. As a result, we recommend that you use test folders in your copy activities and other activities when debugging. After you’ve debugged the pipeline, switch to the actual folders that you want to use in normal operations. Source

We will focus on these 2 execution types. Let us start with an empty pipeline and a SetVariable activity:

This activity has been created to assign the value ‘placeholder’ to executionType variable:

Note that ADF is connected to git.

Also, there’s one pending change to publish:

Let’s assign a default value to the pipeline variable and proceed with publishing these changes.

We can now imagine that we want to introduce a new feature, for the sake of simplicity, we will change the value of the SetVariable activity to “ValueX”:

And we will only save this change (not publish it):

Question: What will be the output value of “Set variable1” if we manually trigger the pipeline?

The answer is “placeholder”.

Why? Let’s go over the steps we took so far:

1. Assigned the value “placeholder” to the variable. Published pending changes.
2. Introduced a new feature (assigning the value “ValueX” to the variable)
3. Saved the change in our repository
4. Trigger a manual pipeline execution using “Trigger now”

Let us know introduce a new concept, the live mode.

I invite you to explore this and try it in your own resource. What are you expecting to find in here?

After changing (from Git Mode) to live mode, we see that the value of “Set variable 1” is still the “placeholder”.

What does this mean?

1. All published code will be present in the live mode, and this will be the code executed in Trigger executions – in the first publish operation, the SetVariable values was “placeholder” – we are now in a position of better understanding the answer to the question above.
   
2. Changes performed/applied directly in/to the canvas, which are not saved nor published, will not be persisted, and will be lost upon session closure (or just refreshing the Azure Data Factory studio). If one performs a debug execution, the code executed will be the one from the canvas, even if it is different from the one in the live mode.

When in git mode, you will not be able to perform changes directly in the live mode.

To illustrate, let’s perform another test. Let us change the SetVariable activity value to “Canvas”.

This is the value saved in the repository – even after changing the value directly in the canvas:

The value in the live mode is still the same (“placeholder”) as we have not performed the publishing operation.

After a debug execution, what is the expected output value for the SetVariable activity?

The answer is “Canvas”.

Summarizing:

Trigger/manual/on-demand executions will take the code we have in live mode — run of the last published pipeline (ADF Studio’s Trigger now button hover message)

Debug executions will take what we have in front of us, in the canvas, even if we have a different value in git (or live mode) — test runs of the current pipeline without publishing your changes to the service (ADF Studio’s Debug button hover message)

For illustration purposes, in this example I worked with just the main branch (which is my collaboration branch). Usually, new features start in a feature branch which are then merged into the main/collaboration branch. More info: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery#cicd-lifecycle