I am developing a lambda architecture where we in the batch layer we send the events to s3->EMR->ETL->s3 and the idea is to then serve it through redshift, and in the other side the speed layer receive the same events through lambda->dynamoDB.

In the end we need to merge the data from both layers, and in our case we will need to do some bunch of where clauses based in the aggregation of both ends, we first thought of using athena to do the queries and read from the s3 and dynamo and merge then. However it is taking a while to process the data from both sides and return a result.

So, since the real time isnt supposed to have great differences for the last hour(which is the timeframe we are using between both layers), we thought of joining the data at application level, but I couldnt find any reference out there at doing it..

Do you know any libraries or good practice to doing so at application level? Do you have any other suggestion that may be easier or more efficient to merge this two layers?