I have a file I am trying to load to postgresql through a python script. After researching I found that just running the copy command seems to be the standard route, so this is what I am doing.
COPY file__exportedData(dataID,dataName,dataDate)
FROM 'D:\\workspace\\dataApp\\ETL\\testFile\\_exportedData.txt' WITH (FORMAT text, DELIMITER ',')
The first file I ran this against worked successfully, however the second file (in the example above) is having an issue. The delimiter on the text file is a comma, however the field "dataName" is a varchar field with double quotes around the varchar. records in the data name CAN have a comma in the name itself, ex:
12345,Michigan,2024-01-01
67890,"Detroit,Michigan",2024-01-01
Both data records can be expected in this data file, but the copy command thinks the comma after Detroit is the end of that field and it breaks the code. Is there a different way of handling this type of scenario? I couldn't find much in my google search.