Opensearch Ingestion Pipeline reads from S3 bucket with a file:
{"key1": "value1" , "value1": "54", "key3": "value10"}
{"key1": "value3" , "value3": "65", "key3": "value9"}
Pipeline configuration
version: "2"
log-pipeline:
source:
s3:
codec:
newline: null
compression: "none"
aws:
region: "eu-west-2"
sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
acknowledgments: true
scan:
buckets:
- bucket:
name: "tempesdata"
delete_s3_objects_on_read: false
processor:
- parse_json:
- delete_entries:
with_keys: ["s3", "message"]
sink:
- opensearch:
hosts: [ "{host}" ]
aws:
sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
region: "eu-west-2"
serverless: false
index: "test"
document_id: ${/key3} # UPDATE THIS
Problem is i want document id to be concat of key1's value's value + ${/key3}
,
Something like $/{${/key1}}-${/key3}
.
But this does not work.
In the given example, I would want documents to be indexed with 54-value10, 65-value9