Python file not closing

Question

I wrote a program to do some processing to a little bit big amount of data. Three steps are involved there:

1. Read data
2. Do Processing
3. Write Data to a file.

While running the code first two steps done successfully.(As data is huge for my laptop configuration, I am using swap space in linux to do the job).

Now the third step: Data is written to file successfully. But my code get stuck at the line feat.close() (feat is the file pointer). When I open the file while the process is running complete data is being written, but my file is not closing.

Code:

#!/usr/bin/env python

from __future__ import print_function
import pickle
import sys
import numpy as np
import posTagsToTriGramFrequency as pt
import itertools
import gc

if sys.argv[1] == '-h':
    print("Usage: ./featureSelection <n(authorLimit)> <k(SD weight)> <nGramSize> <folder>")
    sys.exit()

n = int(sys.argv[1])
k = int(sys.argv[2])
nGramSize = int(sys.argv[3])
folder = sys.argv[4]

print('reading features')
feat = open('../../../../dataDump/'+ folder +'/features.dump','r')
features = {}

tagsnGram = [tuple(x) for x in itertools.product(pt.getTags(), repeat=nGramSize)]

gramdict = {}
for gram in tagsnGram:
    gramdict[gram] = []

flag = 1
author = ''
for line in feat:
    if(line == '\n'):
        flag = 1
    elif flag == 1:
        author = line.split('/')[-1][:-1]
        print(line, end='')
        features[author] = gramdict.copy()
        flag = 0
    else:
        tagsFreq = iter(line.split())
        for tag in tagsnGram:
            features[author][tag].append(float(tagsFreq.next()))

feat.close()

print('Calculating waht to delete')

nflag = 0
kflag = 0
toDel = []
for tagGram in tagsnGram:
    nflag = 0
    for author in features:
        kflag = 0
        for doc in features[author][tagGram]:
            if doc == 0 : kflag += 1
            if kflag >= k:
                nflag += 1
                break
        if nflag >= n:
            toDel.append(tagGram)
            break

for item in toDel:
    for author in features:
        del features[author][item]

f = open('../../../../dataDump/'+ folder +'/tagsInfo.dump','w')
f.write('k:' + str(k) + ',\t n:' + str(n) + '\n')
f.write('Deleted tags:\n')
for item in toDel:
    f.write(str(item) + ' ')
f.write('\n\nSelected Tags:\n')
for tagGram in features.itervalues().next():
        f.write(str(tagGram) + ' ')
f.close()

print("Writing Back Features")
feat = open('../../../../dataDump/'+ folder +'/selectedFeatures.dump','w')
for author in features.keys():
    feat.write(author + '\n')
    print(author)
    for tag in features[author]:
        for doc in features[author][tag]:
            feat.write(str(doc) + ' ')
        feat.write('\n')
    feat.write('\n')
    del features[author]
    #gc.collect()
print('Closing File')
feat.close()

See the last line. At my console Closing file is being printed but after that my code is stuck.

My Console Output:

abhi@abhi-me~/Projects/workspace/irProject/completepythonbased/authAttrib (irProject)>>./featureSelection.py 35 125 3 3GramFreq
reading features
Ajit_Popat
Mukund_Mehta
Parajit_Patel
Priyadarshi
Kumarpad_Desai
Bhaven_Kacchi
Shantibhai_Agrawat
Pravin_Darji
Ankit_Trivedi
Sharad_Rawal
Tushar_Shukla
Chandrakant_Mehta
Jay_Vasavda
Dolat_Bhatt
Munindra
Mrugesh_Vaishnav
Kulinchandra_Yagnik
Zaverilal_Mehta
Priti_Shah
Vasant_Mistri
Vatsal_Vasani
Dinesh_Mistri
Devesh_Mehta
Dhaval_Mehta
Urvish_Kothari
Madhusudan_Parekh
Vihari_Chaya
Virendra_Kapoor
Mukul_Choksi
Joravarsinh_Jadav
Ashok_Dave
Nasir_Ismaeli
Joban_Pandit
Priyakant_Parikh
Sudarshan_Upadhyay
Gajendra_Shah
Altaf_Patel
Bhalchandra_Jani
Shashin
Hansal_Bhachech
Calculating waht to delete
Writing Back Features
Pravin_Darji
Ajit_Popat
Kulinchandra_Yagnik
Sharad_Rawal
Madhusudan_Parekh
Shantibhai_Agrawat
Gajendra_Shah
Hansal_Bhachech
Vihari_Chaya
Virendra_Kapoor
Sudarshan_Upadhyay
Priyadarshi
Tushar_Shukla
Dolat_Bhatt
Urvish_Kothari
Vasant_Mistri
Mukund_Mehta
Zaverilal_Mehta
Kumarpad_Desai
Vatsal_Vasani
Bhaven_Kacchi
Mrugesh_Vaishnav
Bhalchandra_Jani
Priyakant_Parikh
Chandrakant_Mehta
Mukul_Choksi
Joravarsinh_Jadav
Munindra
Joban_Pandit
Devesh_Mehta
Priti_Shah
Ankit_Trivedi
Dinesh_Mistri
Dhaval_Mehta
Ashok_Dave
Nasir_Ismaeli
Parajit_Patel
Jay_Vasavda
Altaf_Patel
Shashin
Closing File
^C
[1]+  Killed                  ./featureSelection.py 35 125 3 3GramFreq

Why this is happening?

You can find the strace here.

Edit : I tried to print something after close and it is printing, means problem is in exiting the program. It is using about 3 GBs of RAM and 3 GBs of swap space @Justing. To fill this memory it took 10-20 minutes and to clear this memory I waited for about 2 hrs, I think there is a problem. I uploaded the strace above, please see.

Solution

Try to work with with which will flush to disk when you are done writing:

with open('file.txt', 'r') as f:
     data = f.read()
f.closed

see the python documentation. It should solve most of the problems when writing and reading from files.

Comments

[Gareth Latty]

It's worth noting that when working with files in Python, the best practice is to use the with statement.

[Aya]

The code looks fine to me. If the file is huge, it's possibly just waiting for the OS to flush something to disk, or move pages in or out of swap, or somesuch. Have you tried just waiting for a while after the "Closing File" to see if it ever terminates by itself?

[glglgl]

Run the program with strace to see what happens.

[Keyur Padalia]

It seems very unlikely that closing a file takes a noticeable amount of time. Try printing something after the close() statement to check that it's really the close that hangs.