[Answered ]-Python – Parsing large text file and inserting data into database

by

1👍

✅

Welp, I’m dumb.

Bulk create is my new friend.

If anyone has the same problem, don’t insert rows one at a time. Each create() call is one insert statement. Instead, add the objects to a list, and then bulk_create(the_list).

👤Jesse McBride

1👍

Try this using itertools.islice

from itertools import islice
with open('hourlydump.txt', 'r') as f
    my_lines = islice(f, N) #number of lines taken in each iteration.
    #Do your operations here

my_lines is a generator object, that gives you each line of the file and can be used in a loop like this:

for line in mylines:
    print line
👤itzMEonTV

Leave a comment