最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

python - Writing Csv using Pandas causing memory issue - Stack Overflow

programmeradmin4浏览0评论

I have a large CSV file with 200,000 rows and need to create another CSV because Excel is consuming too much memory. However, since the code is deployed on a t3.small server, it's causing memory issues. Below is the code I'm using:

with open(file_path, 'w', newline='') as f:
    for i in range(0, len(dataframe), chunk_size):
        chunk = dataframe.iloc[i:i + chunk_size]
        chunk.to_csv(f, header=(i == 0), index=False, date_format='%Y-%m-%d %H:%M:%S')
        del chunk
        gc.collect()

How can I optimize this to reduce memory usage?

发布评论

评论列表(0)

  1. 暂无评论