Using the AzureML API V2 for python I want to create an AzureML job without running it. The job is supposed to start only when triggered in the AzureML Web UI. How can I achieve that?

When I create the job as follows it runs right away:

from azure.ai.ml import MLClient, command
from azure.identity import DefaultAzureCredential


@dsl.pipeline
def my_job:
    command(command="echo 'Hello'") 

ml_client = MLClient(
    subscription_id=subscription_id,
    resource_group_name=resource_group,
    credential=DefaultAzureCredential()
)
ml_client.jobs.create_or_update(my_job())

Using the AzureML API V2 for python I want to create an AzureML job without running it. The job is supposed to start only when triggered in the AzureML Web UI. How can I achieve that?

When I create the job as follows it runs right away:

from azure.ai.ml import MLClient, command
from azure.identity import DefaultAzureCredential


@dsl.pipeline
def my_job:
    command(command="echo 'Hello'") 

ml_client = MLClient(
    subscription_id=subscription_id,
    resource_group_name=resource_group,
    credential=DefaultAzureCredential()
)
ml_client.jobs.create_or_update(my_job())

• python
• azure-machine-learning-service

1 Answer 1

Write pipeline to yaml, and register it. Go to the AzureML Studio Web UI.Navigate to Pipelines > Pipeline Endpoints or Designer. Use the "Create Pipeline" or "Import" option to upload the my_manual_pipeline.yaml file. Save it without submitting it. It will now appear as a registered pipeline.

from azure.ai.ml import MLClient, dsl, command
from azure.identity import DefaultAzureCredential
import yaml

@dsl.pipeline(
    name="my_manual_pipeline",
    description="A pipeline that runs only when triggered manually",
)
def my_job():
    echo_job = command(
        command="echo 'Hello'",
        compute="compute-target",
    )
    return {"output": echo_job.outputs}

ml_client = MLClient(
    subscription_id="subscription-id",
    resource_group_name="resource-group",
    workspace_name="workspace-name",
    credential=DefaultAzureCredential(),
)

pipeline_instance = my_job()
pipeline_dict = pipeline_instance._to_dict()

with open("my_manual_pipeline.yaml", "w") as f:
    yaml.dump(pipeline_dict, f)

print("Pipeline definition saved to 'my_manual_pipeline.yaml'.")

EDIT

The PipelineComponent class lets you register a pipeline as a reusable asset in your workspace without running it. You can reuse it in other pipelines or manually trigger it later.

from azure.ai.ml import MLClient, dsl, command
from azure.identity import DefaultAzureCredential
from azure.ai.ml.entities import PipelineComponent

@dsl.pipeline(
    name="my_manual_pipeline",
    description="A pipeline that runs only when triggered manually",
)
def my_job():
    echo_job = command(
        command="echo 'Hello'",
        compute="compute-target",
    )
    return {"output": echo_job.outputs}

ml_client = MLClient(
    subscription_id="subscription-id",
    resource_group_name="resource-group",
    workspace_name="workspace-name",
    credential=DefaultAzureCredential(),
)

pipeline_instance = my_job()

pipeline_component = PipelineComponent(
    name="my_manual_pipeline",
    version="1",
    description="A manually triggered pipeline",
    pipeline=pipeline_instance,
)
registered_component = ml_clientponents.create_or_update(pipeline_component)

print(f"Pipeline registered as a component with name: {registered_component.name} and version: {registered_component.version}")