During a certain File type conversion from The Raw Tab file That we fetch to convert to Parquet Type, one of the rows is always missing \ in the columns values,

so the expected value for example is :'GH\334' But the Parquet File only has 'GH334' not sure why its skipping the '' especially since we are not doing any sort of data transform, just running a simple Copy Activity.

Design is: Source: CSV file type Sink: Parquet File type Mapping: @json(item().Mapping) (This mapping is provided from a lookup activity which is fetched from the SQL server) no other settings or such, Has anyone dealt with such issues before? would appreciate your inputs on this; Thank you.

Also to mention, this row is being converted to string type for further data transformation...I was wondering if that had any relation with it, But I would need some Ideas on How I can move forward with this Again, the source CSV/.tab file does indeed have the row with '' Character, its just during the copy Activity, that its getting removed

Frankly its such a simple copy Activity, I am sort of out of ideas on what to try, I would appreciate it, if someone could give me an idea on what Can I try

Sample Input Data: GH/DFC\3456

Sample Expected Output Data: GH/DFC\3456

Sample Actual Output Data: GH/DFC3456

During a certain File type conversion from The Raw Tab file That we fetch to convert to Parquet Type, one of the rows is always missing \ in the columns values,

so the expected value for example is :'GH\334' But the Parquet File only has 'GH334' not sure why its skipping the '' especially since we are not doing any sort of data transform, just running a simple Copy Activity.

Design is: Source: CSV file type Sink: Parquet File type Mapping: @json(item().Mapping) (This mapping is provided from a lookup activity which is fetched from the SQL server) no other settings or such, Has anyone dealt with such issues before? would appreciate your inputs on this; Thank you.

Also to mention, this row is being converted to string type for further data transformation...I was wondering if that had any relation with it, But I would need some Ideas on How I can move forward with this Again, the source CSV/.tab file does indeed have the row with '' Character, its just during the copy Activity, that its getting removed

Frankly its such a simple copy Activity, I am sort of out of ideas on what to try, I would appreciate it, if someone could give me an idea on what Can I try

Sample Input Data: GH/DFC\3456

Sample Expected Output Data: GH/DFC\3456

Sample Actual Output Data: GH/DFC3456

• azure
• azure-pipelines
• azure-data-factory

• Can you provide a sample input data and expected output data? – Rakesh Govindula Commented Mar 17 at 3:30
• Hi @RakeshGovindula , added the expected and actual Output at the end of my question for reference. – mohammed rifat khan Commented Mar 17 at 4:24
• I have tried the above scenario in my environment with below sample data i.imgur/wtodnCC.png and in source dataset I took the configurations like this i.imgur/wT5stYo.png. With these configurations, its working fine for me without any missing character. Can you re-check your dataset configurations as shown here. If still it results same, provide copy activity source and sink configurations along with dataset configurations? – Rakesh Govindula Commented Mar 17 at 6:33
• Thank you @RakeshGovindula after I saw your pictures, I went to my settings to check and set the 'Escape Character' settings on the dataset as "None" instead of BackSlash '\' and was able to fix the Issue, turns out ADF was treating \ as escape character and hence was skipping through it, kind of didn't really answer your question, however was able to solve it through the picture. – mohammed rifat khan Commented Mar 17 at 14:52

1 Answer 1

@mohammed rifat khan, posting our discussion as an answer for the community.

I took the below data as sample which is similar to your data.

Table2Id,Marks,name,age
1,94,"GH/DFC\3456",25
3,100,"Rakesh/DFC\3456",24
5,98,"GH/Laddu\3456",23

To not skip the \ character, you need to ensure that you have given the \ character as an escape character in the source dataset configurations.

Now, it will give the desired output which you can cross-check from the source dataset data preview.