Configuring pipeline nodes

Copy assets

Export assets

Export selected assets from the scope, for example, a project or deployment space. The operation exports all the assets by default. You can limit asset selection by building a list of resources to export.

Import assets

Import assets from a ZIP file that contains exported assets.

Create AutoAI experiment

Use this node to train an AutoAI classification or regression experiment and generate model-candidate pipelines.

Create AutoAI time series experiment

Use this node to train an AutoAI time series experiment and generate model-candidate pipelines.

Create batch deployment

Use this node to create a batch deployment for a machine learning model.

Create data asset

Use this node to create a data asset.

Create deployment space

Use this node to create and configure a space that you can use to organize and create deployments.

Create online deployment

Use this node to create an online deployment where you can submit test data directly to a web service REST API endpoint.

Wait for all results

Use this node to wait until any result from the previous nodes in the pipeline is available so the pipeline can continue. Run the downstream nodes as soon as any of the upstream conditions are met.

This node takes no inputs and produces no output. When any results are available, the pipeline continues automatically.

Wait for any result

Wait for an asset to be created or updated in the location that is specified in the path from a job or process earlier in the pipeline. Specify a timeout length to wait for the condition to be met. If 00:00:00 is the specified timeout length, the flow waits indefinitely.

Wait for file

Loops in parallel

Add a parallel looping construct to the pipeline. A parallel loop runs the iterating nodes independently and possibly simultaneously.

For example, to train a machine learning model with a set of hyperparameters to find the best performer, you can use a loop to iterate over a list of hyperparameters to train the notebook variations in parallel. The results can be compared later in the flow to find the best notebook. To see limits on the number of loops you can run simultaneously, see Limitations.

In the following example, a Run Bash script node searches for and retrieves notebooks that match specified criteria. A Run DataStage job node retrieves data from a Git repository. When input from each node is available, the loop process begins, running each notebook retrieved by the search and processing the data retrieved from the Git repository.

Click Expand to add nodes or the outgoing icon on the node to view the full loop process. As the notebooks run, any errors in the notebook are captured in a condition called Poor quality. The condition triggers a Bash script to increment a user variable that is named Increase error count. When the value of the Increase error count variable meets the specified threshold, the loop is terminated.

Since the flow is executed in parallel for each notebook, it returns results faster than a sequential loop.

Loops in sequence

Add a sequential loop construct to the pipeline. Loops can iterate over a numeric range, a list, or text with a delimiter.

A use case for sequential loops is if you want to try an operation 3 time before you determine whether an operation failed.

Terminate loop

In a parallel or sequential loop process flow, you can add a Terminate pipeline node to end the loop process anytime. You must customize your own conditions for terminating. In the nodes, you can change the status of the loop when it terminates as either Complete or Failure. This ensures that you can complete the loop and still meet the conditions of the loop node failing and choose to either continue with the pipeline or do other actions.

Set user variables

Configure a user variable with a key/value pair, then add the list of dynamic variables for this node.

For more information on how to create a user variable, see Configuring global objects.

Terminate pipeline

You can initiate and control the termination of a pipeline with a Terminate pipeline node from the Control category. When the error flow runs, you can optionally specify how to handle notebook or training jobs that were initiated by nodes in the pipeline. You must specify whether to wait for jobs to finish, cancel the jobs then stop the pipeline, or stop everything without canceling. Specify the options for the Terminate pipeline node.

Update AutoAI experiment

Update batch deployment

Use these parameters to update a batch deployment.

Update deployment space

Update the details for a space.

Update online deployment

Use these parameters to update an online deployment (web service).

Delete

You can delete:

• AutoAI experiment
• Batch deployment
• Deployment space
• Online deployment

For each item, choose the asset for deletion.

Run AutoAI experiment

Trains and stores AutoAI experiment pipelines and models.

Run Bash script

Run an inline Bash script to automate a function or process for the pipeline. You can enter the Bash script code manually, or you can import the bash script from a resource, pipeline parameter, or the output of another node.

You can also use a Bash script to process large output files. For example, you can generate a large, comma-separated list that you can then iterate over using a loop.

In the following example, the user entered the inline script code manually. The script uses the cpdctl tool to search all notebooks with a set variable tag and aggregates the results in a JSON list. The list can then be used in another node, such as running the notebooks returned from the search.

Run batch deployment

Configure this node to run selected deployment jobs.

Run Data Refinery job

This node runs a specified Data Refinery job.

Run notebook job

Use these configuration options to specify how to run a Jupyter Notebook in a pipeline.

Run Pipelines component

Run a reusable pipeline component that is created by using a Python script. For more information, see Creating a custom component.

• If a pipeline component is available, configuring the node presents a list of available components.
• The component that you choose specifies the input and output for the node.
• Once you assign a component to a node, you cannot delete or change the component. You must delete the node and create a new one.

Run Pipelines job

Add a pipeline to run a nested pipeline job as part of a containing pipeline. This is a way of adding reusable processes to multiple pipelines. You can use the output from a nested pipeline that is run as input for a node in the containing pipeline.

Run SPSS Modeler job

Use these configuration options to specify how to run an SPSS Modeler in a pipeline.