python3 读取大文件分解成若干小文件

有个数据实在太大了,有1.7G,打开慢,改文件也慢,我们将其分解成若干个中等文件

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
 
f = open("123.sql",'r',encoding='utf-8')
readlist = []
i = 0
i=i+1
filename = "mytest_{0}.sql".format(i)
wf=open(filename, 'w', encoding='utf-8')
linecount=0
while 1:
    try:
        line = f.readline()
        wf.write(line)
        linecount=linecount+1       
        if(linecount >;= 200000):
            linecount = 0
            print(filename)
            i=i+1
            wf.close()
            filename = "mytest_{0}.sql".format(i)
            wf=open(filename, 'w', encoding='utf-8')                
        if not line:
            break        
    except Exception as e:
        print("read except:" + str(e))
        continue
        #print("read except:" + str(e))
 
f.close()
 
print(filename)
wf.close()        
 
 

有一个问题,之前的sql文件保存为utf-8格式,实际上在文件中夹杂中部分乱码,读取时会出错,所以需要try屏蔽

原文地址:https://www.cnblogs.com/luhouxiang/p/4940056.html