I have AWS Stepfunction that starts with a Lambda function to prepare the execution of an AWS Batch Job, of which the Job Definition specifies to use Fargate (ecsProperties Job). This stepfunction fails at the submit-batch-job step:

{
  "Comment": "AWS Step Functions for processing batch jobs and updating Athena",
  "StartAt": "Prepare Batch Job",
  "States": {
    "Prepare Batch Job": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:<region>:<account_number>:function:prepare-batch-job",
      "Next": "Run Batch Job"
    },
    "Run Batch Job": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName.$": "$.jobName",
        "JobQueue.$": "$.jobQueue",
        "JobDefinition.$": "$.jobDefinition",
        "ArrayProperties": {
          "Size.$": "$.number_of_batches"
        },
        "Parameters": {
          "table_id.$": "$.table_id",
          "run_timestamp.$": "$.run_timestamp",
          "table_path_s3.$": "$.table_path_s3",
          "batches_s3_path.$": "$.batches_s3_path",
          "is_training_run.$": "$.is_training_run"
        }
      },
      "Next": "Prepare Athena Query"
    },
...

Upon execution, the Run Batch Job step fails with the following message:

Container overrides should not be set for ecsProperties jobs. (Service: AWSBatch; Status Code: 400; Error Code: ClientException; Request ID: ffewfwe96-c869-4106-bc4d-3cfd6c7c34a0; Proxy: null)

One very important thing to note is that, if I move the submit-job request to the first step (lambda) using the boto3 api, the job gets submitted and starts running without issues. However, when I submit the job from the Run Batch Job step within the stepfunction, the aforementioned error appears.

This question has already been posted here, wherein the author notes that AWS Stepfunctions automatically adds the following to the definition, which appears to be the root of the error:

  "ContainerOverrides":{
    "Environment": [
      {
        "Name": "MANAGED_BY_AWS",
        "Value": "STARTED_BY_STEP_FUNCTIONS"
      }
    ]
  }

The answer provided in the aforementioned post however seems unclear to me as someone who has only started using AWS Batch a short while ago. If anyone would care to elaborate and assist, I would be very grateful.

I should state that the only reason I need to use the Run Batch Job step approach, is that I need my workflow to wait for the batch job to complete before attempting to insert the results as a new partition into an Athena results table. This is not feasible from within the Lambda function using boto3, as Lambdas timeout after 15 minutes, and the boto3 submit_job method does not wait for the execution to complete.

Thanks in advance.