Permission denied when pandas dataframe to tempfile csv

Question

I'm trying to store a pandas dataframe to a tempfile in csv format (in windows), but am being hit by:

[Errno 13] Permission denied: 'C:\Users\Username\AppData\Local\Temp\tmpweymbkye'

import tempfile
import pandas

with tempfile.NamedTemporaryFile() as temp:
    df.to_csv(temp.name)

Where df is the dataframe. I've also tried changing the temp directory to one I am sure I have write permissions:

tempfile.tempdir='D:/Username/Temp/'

This gives me the same error message

Edit:

The tempfile appears to be locked for editing as when I change the loop to:

with tempfile.NamedTemporaryFile() as temp:
    df.to_csv(temp.name + '.csv')

I can write the file in the temp directory, but then it is not automatically deleted at the end of the loop, as it is no longer a temp file.

However, if I change the code to:

with tempfile.NamedTemporaryFile(suffix='.csv') as temp:
    training_data.to_csv(temp.name)

I get the same error message as before. The file is not open anywhere else.

Solution 1

Check your permissions and, according to this post, you can run your program as an administrator by right click and run as administrator.

We can use the to_csv command to do export a DataFrame in CSV format. Note that the code below will by default save the data into the current working directory. We can save it to a different folder by adding the foldername and a slash to the file

verticalStack.to_csv('foldername/out.csv').

Check out your working directory to make sure the CSV wrote out properly, and that you can open it! If you want, try to bring it back into python to make sure it imports properly.

newOutput = pd.read_csv('out.csv', keep_default_na=False, na_values=[""])

ref

Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.

With the use of this function may introduce a security hole in your program. By the time you get around to doing anything with the file name it returns, someone else may have beaten you to the punch. mktemp() usage can be replaced easily with NamedTemporaryFile(), passing it the delete=False paramete.

Read more.

After export to CSV you can close your file with temp.close().

with tempfile.NamedTemporaryFile(delete=False) as temp:
    df.to_csv(temp.name + '.csv')
    temp.close()

Solution 2

I encountered the same error message and the issue was resolved after adding "/df.csv" to file_path.

df.to_csv('C:/Users/../df.csv', index = False)

Solution 3

Sometimes，you need check the file path that if you have right permission to read and write file. Especially when you use relative path.

xxx.to_csv('%s/file.csv'%(file_path), index = False)

Solution 4

Sometimes, it gives that error simply because there is another file with the same name and it has no permission to delete the earlier file and replace it with the new file.

1. So either name the file differently while saving it, or
2. If you are working on Jupyter Notebook or a other similar environment, delete the file after executing the cell that reads it into memory. So that when you execute the cell which writes it to the machine, there is no other file that exists with that name.

Solution 5

I encountered the same error. I simply had not yet saved my entire python file. Once I saved my python file in VS code as "insertyourfilenamehere".py to documents(which is in my path), I ran my code again and I was able to save my data frame as a csv file.

View more solutions

Comments

[thebigspin]

Running as admin didn't change anything, and I tried changing the tempdir to one where I definitely have write permission

[RaminNietzsche]

@thebigspin I changed my answer

[RaminNietzsche]

@thebigspin try this: with tempfile.NamedTemporaryFile(delete=False) as temp:

[thebigspin]

This works (not sure why!?) but then the file isn't deleted at the end of the loop. Tried adding os.remove(temp.name) but then it says file is in use...

[RaminNietzsche]

@thebigspin I changed my answer. please close your file before delete.

[thebigspin]

Can you edit your answer to include only the relevant info (adding delete=false and then temp.close() after using the file) in case someone else hits the same problem?

[RaminNietzsche]

@thebigspin I add this part and make answer like as step-to-step guide for this error :)

[dataviews]

make sure to CHMOD the folder you are writing to (777)

[thebigspin]

This is not saving to a tempfile and so does not address the question

[thebigspin]

This would create a permanent csv file (df.csv) and not a tempfile

[scientific_explorer]

NamedTemporaryFile has a kwarg suffix which you can set to .csv so that you just have to use temp.name in your answer.