I have two pySpark DataFrames, need to compare those two DataFrames column wise and append the result next to it.

DF1:

I have two pySpark DataFrames, need to compare those two DataFrames column wise and append the result next to it.

DF1:

Df2:

Expected Result in pySpark:

DF3:

• pyspark
• apache-spark-sql
• pyspark-pandas

• What is your actual question? What have you tried? – Andrew Commented Nov 20, 2024 at 15:57
• I want to compare two dataframe and if the column value matches it should populate True and if it not matches it should populate False next to the column – Srinivasan Commented Nov 20, 2024 at 15:59
• That's not a question, that's asking SO to write your code for you. – Andrew Commented Nov 20, 2024 at 16:06
• Sorry i dont understand your question... – Srinivasan Commented Nov 20, 2024 at 16:08
• 1 Unlike Pandas dataframes PySpark dataframes are not ordered. So the task is not doable unless a criterium is provided which rows of each dataframe should be compared. Simply saying "take the third row from df1 and compare it with the third row from df2" does not work unfortunately. There is no "third row", at least not when using large datasets with multiple partitions. – werner Commented Nov 20, 2024 at 18:24

1 Answer 1

DF(s) are nor ordered but distributed in different places so this is an invalid ask.

However what you can do instead is following -

• You can assume DF1 is master DF and join it with DF2 using Claim_number and if DF2 has no claim number then depending on join type, you can choose to ignore (inner join) or produce then as null(left outer join)

If that is what yous ask is, then here is the solution.

final_Df = df1.join(df2, Claim_number, "inner").distinct()