I am trying to generate some keys (according to the documentation) via a docker command. I want to use these keys in my terraform configuration. However, terraform does not seem to read the output of the command which causes it to complain that the returned map of results is empty. If I manually copy the output of the command and put it in echo statements it does work. What am I doing wrong?

data "external" "generated_secrets" {
    program = ["bash", "/REDACTED_ABS_PATH/test.sh"]
}

#!/bin/bash

# DOES NOT WORK
X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq .
{ $X }
EOF

# WORKS
# This is the exact output of the command above. Each line of output has it's own echo statement
echo "{"
echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
echo "}"

│ Error: Missing map element
│
│   on apprunner.tf line 232, in resource "aws_secretsmanager_secret_version" "vault_private_key":
│  232:     secret_string               = data.external.generated_secrets.result.PRIVATE_KEY
│     ├────────────────
│     │ data.external.generated_secrets.result is empty map of string
│
│ This map does not have an element with the key "PRIVATE_KEY".

EDIT 1:

It seems that the output of the docker command is not executed/evaluated at all, as it also does not lead to errors (which are present with other non-json-output commands)

The following command executed the following:

echo $(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | grep PRIVATE)
PRIVATE_KEY: 'd0b2850aaed4e6e0b62646fa62daedae820fef2f24d91fd750fe2d3d0716575f'

If I copy this output and place it at the top of the test.sh file nothing changes (empty map error):

#!/bin/bash

echo $(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | grep PRIVATE)
# OR (tested both)
docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys

X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq . -c -M | tr -d '\n'
{ $X }
EOF

# echo "{"
# echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
# echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
# echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
# echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
# echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
# echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
# echo "}"

However, if I directly copy the output of the command I get a different error:

#!/bin/bash

echo "PRIVATE_KEY: 'd0b2850aaed4e6e0b62646fa62daedae820fef2f24d91fd750fe2d3d0716575f'"

X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq . -c -M | tr -d '\n'
{ $X }
EOF

# echo "{"
# echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
# echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
# echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
# echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
# echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
# echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
# echo "}"

│ Error: Unexpected External Program Results
│
│   with data.external.generated_secrets,
│   on apprunner.tf line 149, in data "external" "generated_secrets":
│  149:     program = ["bash", "/REDACTED_ABS_PATH/test.sh"]
│
│ The data source received unexpected results after executing the program.
│
│ Program output must be a JSON encoded map of string keys and string values.
│
│ If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level. Terraform documentation on logging:
│ 
│
│ Program: /usr/bin/bash
│ Result Error: invalid character 'P' looking for beginning of value

This indicates that the Docker output is not evaluated at all. Why is this? A timing issue?

I am trying to generate some keys (according to the documentation) via a docker command. I want to use these keys in my terraform configuration. However, terraform does not seem to read the output of the command which causes it to complain that the returned map of results is empty. If I manually copy the output of the command and put it in echo statements it does work. What am I doing wrong?

data "external" "generated_secrets" {
    program = ["bash", "/REDACTED_ABS_PATH/test.sh"]
}

#!/bin/bash

# DOES NOT WORK
X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq .
{ $X }
EOF

# WORKS
# This is the exact output of the command above. Each line of output has it's own echo statement
echo "{"
echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
echo "}"

│ Error: Missing map element
│
│   on apprunner.tf line 232, in resource "aws_secretsmanager_secret_version" "vault_private_key":
│  232:     secret_string               = data.external.generated_secrets.result.PRIVATE_KEY
│     ├────────────────
│     │ data.external.generated_secrets.result is empty map of string
│
│ This map does not have an element with the key "PRIVATE_KEY".

EDIT 1:

It seems that the output of the docker command is not executed/evaluated at all, as it also does not lead to errors (which are present with other non-json-output commands)

The following command executed the following:

echo $(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | grep PRIVATE)
PRIVATE_KEY: 'd0b2850aaed4e6e0b62646fa62daedae820fef2f24d91fd750fe2d3d0716575f'

If I copy this output and place it at the top of the test.sh file nothing changes (empty map error):

#!/bin/bash

echo $(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | grep PRIVATE)
# OR (tested both)
docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys

X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq . -c -M | tr -d '\n'
{ $X }
EOF

# echo "{"
# echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
# echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
# echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
# echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
# echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
# echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
# echo "}"

However, if I directly copy the output of the command I get a different error:

#!/bin/bash

echo "PRIVATE_KEY: 'd0b2850aaed4e6e0b62646fa62daedae820fef2f24d91fd750fe2d3d0716575f'"

X=$(docker run --rm -ti psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^\(\s\|#\)/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq . -c -M | tr -d '\n'
{ $X }
EOF

# echo "{"
# echo '"SECRET_KEY": "nAqfuMv3tjTeHAhS8741BiEMSZvyFCQXh1mYxhXf4cRNuMe0paEqLhpXbf9ZqhB",'
# echo '"ACTIVATION_LINK_SECRET": "1tWi9nbVwfdT451Xq3dWfzEA8JSYPko7W32ZqZNBmvqhWtl3lyDRvnwaRQQ4rmu",'
# echo '"DB_SECRET": "DVnfwL90CInhaK5IBVBkKVvWkVHtOQr6MSYzZEe79oIOp8rrjfyIErjQapWL6BS",'
# echo '"EMAIL_SECRET_SALT": "$2b$12$.dEOcnPsb5YlYz5iPUeq3",'
# echo '"PRIVATE_KEY": "329bc28282978f5c16ffa21c1213ea292644aa32f0883b572f4a053151b52cb",'
# echo '"PUBLIC_KEY": "d7a8987ab9dabfcff306d0dcc5781eb3795253ab607443b359c11bca20e71b3d"'
# echo "}"

│ Error: Unexpected External Program Results
│
│   with data.external.generated_secrets,
│   on apprunner.tf line 149, in data "external" "generated_secrets":
│  149:     program = ["bash", "/REDACTED_ABS_PATH/test.sh"]
│
│ The data source received unexpected results after executing the program.
│
│ Program output must be a JSON encoded map of string keys and string values.
│
│ If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level. Terraform documentation on logging:
│ https://www.terraform.io/internals/debugging
│
│ Program: /usr/bin/bash
│ Result Error: invalid character 'P' looking for beginning of value

This indicates that the Docker output is not evaluated at all. Why is this? A timing issue?

• bash
• terraform

• What's the content of X ? – Paolo Commented 16 hours ago
• @Paolo The content is identical to the echo statements without the parenthesis. – Tom Stock Commented 16 hours ago

1 Answer 1

After digging in why only the docker command output was not evaluated I found that the -it flags caused the problem. This redirects the stdout, which if of course needed by terraform. The solution was to remove said flags:

#!/bin/bash

X=$(docker run --rm psono/psono-server:latest python3 ./psono/manage.py generateserverkeys | \
sed -e "/^[\s#\r\t]/d" \
    -e "/^$/d" | \
sed -e "$ ! s/[^: \s]'/',/" \
    -e "s/^/'/" \
    -e "s/:/':/" \
    -e "s/'/\"/g")

cat << EOF | jq . -c -M | tee output
{ $X }
EOF