I have the standard (simplified) pattern, in a step machine:

• start a glue job
• get job satus
• if status is running:
  • wait
  • go back to get job status

My issue is with the connection between the wait state and the get job status state. I cannot forge an output which mimics the output of the starting job.

This is my current setup:

{
  "StartAt": "Parallel",
  "QueryLanguage": "JSONPath",
  "States": {
    "Parallel": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "Start Glue Job",
          "States": {
            "Start Glue Job": {
              "Type": "Task",
              "Resource": "arn:aws:states:::glue:startJobRun",
              "Parameters": {
                "JobName": "job_name"
              },
              "Next": "Get status"
            },
            "Get status": {
              "Type": "Task",
              "Parameters": {
                "JobName.$": "$.JobName",
                "RunId.$": "$.JobRunId"
              },
              "Resource": "arn:aws:states:::aws-sdk:glue:getJobRun",
              "Next": "Choice"
            },
            "Choice": {
              "Type": "Choice",
              "Choices": [
                {
                  "Variable": "$.JobRun.JobRunState",
                  "StringEquals": "RUNNING",
                  "Next": "Wait"
                }
              ],
              "Default": "Complete"
            },
            "Wait": {
              "Type": "Wait",
              "Seconds": 1,
              "QueryLanguage": "JSONata",
              "Next": "Get status",
              "Output": {
                "JobName": "{% $.JobRun.JobName %}", // THIS WHERE I HAVE AN ISSUE!
                "JobRunId": "{% $.JobRun.Id %}"
              }
            },
            "Complete": {
              "Type": "Pass",
              "End": true
            }
          }
        },
        {
          "StartAt": "Placeholder Pass",
          "States": {
            "Placeholder Pass": {
              "Type": "Pass",
              "End": true
            }
          }
        }
      ],
      "Next": "Success",
      "Catch": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "Next": "Fail"
        }
      ]
    },
    "Fail": {
      "Type": "Fail"
    },
    "Success": {
      "Type": "Succeed"
    }
  }
}

An execution tells me:

{
  "Error": "Glue.EntityNotFoundException",
  "Cause": "Job run not found for the given job name and runId. (Service: Glue, Status Code: 400, Request ID: c8ab4d3e-16ce-4fcd-8fe3-96a1aecb6bce)"
}

How can I pass output from wait in the expected format?

For the context, I am trying to have parallel glue jobs, which will complete even if one fails. If there is another direction I should look at, links are welcome.