Writing a list to a file with Python

Question

Is this the cleanest way to write a list to a file, since writelines() doesn't insert newline characters?

file.writelines(["%s\n" % item  for item in list])

It seems like there would be a standard way...

Solution 1

You can use a loop:

with open('your_file.txt', 'w') as f:
    for item in my_list:
        f.write("%s\n" % item)

In Python 2, you can also use

with open('your_file.txt', 'w') as f:
    for item in my_list:
        print >> f, item

If you're keen on a single function call, at least remove the square brackets [], so that the strings to be printed get made one at a time (a genexp rather than a listcomp) -- no reason to take up all the memory required to materialize the whole list of strings.

Solution 2

What are you going to do with the file? Does this file exist for humans, or other programs with clear interoperability requirements?

If you are just trying to serialize a list to disk for later use by the same python app, you should be pickleing the list.

import pickle

with open('outfile', 'wb') as fp:
    pickle.dump(itemlist, fp)

To read it back:

with open ('outfile', 'rb') as fp:
    itemlist = pickle.load(fp)

Solution 3

The simpler way is:

with open("outfile", "w") as outfile:
    outfile.write("\n".join(itemlist))

You could ensure that all items in item list are strings using a generator expression:

with open("outfile", "w") as outfile:
    outfile.write("\n".join(str(item) for item in itemlist))

Remember that all itemlist list need to be in memory, so, take care about the memory consumption.

Solution 4

Using Python 3 and Python 2.6+ syntax:

with open(filepath, 'w') as file_handler:
    for item in the_list:
        file_handler.write("{}\n".format(item))

This is platform-independent. It also terminates the final line with a newline character, which is a UNIX best practice.

Starting with Python 3.6, "{}\n".format(item) can be replaced with an f-string: f"{item}\n".

Solution 5

Yet another way. Serialize to json using simplejson (included as json in python 2.6):

>>> import simplejson
>>> f = open('output.txt', 'w')
>>> simplejson.dump([1,2,3,4], f)
>>> f.close()

If you examine output.txt:

[1, 2, 3, 4]

This is useful because the syntax is pythonic, it's human readable, and it can be read by other programs in other languages.

View more solutions

Comments

[SingleNegationElimination]

do note that writelines doesn't add newlines because it mirrors readlines, which also doesn't remove them.

[Eldwin]

its between json vs pickle. read all about it - stackoverflow.com/questions/27745500/…

[Jason Baker]

This isn't terribly complex, but why not just use pickle or json so that you don't have to worry about the serialization and deserialization?

[Jason Baker]

Of course the first question that springs to mind is whether or not the OP needs it to end in a newline and whether or not the amount of space matters. You know what they say about premature optimizations.

[Jason Baker]

One should note that this will require that the file be opened as text to be truly platform-neutral.

[Jason Baker]

+1 - Why reinvent the wheel when Python has serialization built in?

[Alex Martelli]

For example because you want an output text file that can be easily read, edited, etc, with one item per line. Hardly a rare desire;-).

[RobM]

A downside: This constructs the entire contents for the file in memory before writing any of them out, so peak memory usage may be high.

[jilles de wit]

Note from the documentation of the os module: Do not use os.linesep as a line terminator when writing files opened in text mode (the default); use a single '\n' instead, on all platforms.

[Dan]

I can never get this to work. I get this error: "text = '\n'.join(namelist) + '\n' TypeError: sequence item 0: expected string, list found"

[wolf4]

You've to ensure that all elements in 'namelist' are strings.

[xxjjnn]

+1 - outfile is something like: open( "save.p", "wb" ) infile is something like: open( "save.p", "rb" )

[gwideman]

This writes a string to a file, not a list (of strings) as the OP asks

[Jorge Rodriguez]

I found that the \n in the first one was redundant on Python 2.7/Windows

[mozzbozz]

If you want to add a trailing newline here, simply add a + "\n" at the end. Don't see what problem @Dave has with this.

[TayTay]

This will write an extra newline character at the end... rather than loop, you could just write thefile.write('\n'.join(thelist))

[Harry Duong]

isn't it only put one line in myfile.txt, something like: ['a','b','c'] instead of writing each a,b,c on each line.

[iipr]

I would add: "Be careful with the list datatype". I was getting some weird results, maybe this can help someone: thefile.write(str(item) + "\n")

[kRazzy R]

are lists json serializable?

[Big Sam]

Yes indeed, they are!

[Big Sam]

import json ; test_list = [1,2,3]; list_as_a_string = json.dumps(test_list); #list_as_a_string is now the string '[1,2,3]'

[kRazzy R]

I'm doing this with open ('Sp1.txt', 'a') as outfile: json.dump (sp1_segments, outfile) logger.info ("Saved sp_1 segments") ; problem is my program runs thrice, and the results from three runs are getting mashed up. is there any way to add 1-2 empty lines so that the results from each run are discernible?

[Big Sam]

Absolutely! could you instead do json.dump(sp1_segments + "\n\n", outfile)?

[Filippo Mazza]

The problem is that the list must fit in memory. If this is not the case, line by line is indeed a possible strategy (or else going with some alternative as in stackoverflow.com/questions/7180212/…)

[Pyd]

i dont want to add "\n" for the last item, what to do? dont want if condition

[orluke]

@pyd Replace the for loop with file_handler.write("\n".join(str(item) for item in the_list))

[xlash]

For long list, this method is much much faster.

[queise]

In Python 2 use 'r' instead of 'rb' when reading the pickle if you get the "ValueError: insecure string pickle"

[Jonathan Morales Vélez]

How do you get the file variable?

[seralouk]

do I need to fp.close() after with open ('outfile', 'rb') as fp: ?

[SingleNegationElimination]

@serafeim: no; the with: block will close the file before continuing to the next statement outside of the with block.

[Thom Ives]

Just a little simpler for python 3.7'ish f.write(f'{item}\n')

[Blupon]

the right technique to save and load a list of lists without breaking a sweat

[Geoff Paul Bremner]

the clean solution for people who don't care about any memory limitations.

[An old man in the sea.]

are 'rb' and 'wb' options? What for?

[bballdave025]

Executable code in relation to the comments first from @Dave , and second (from @mozzbozz) >>> with open("outfile", "w") as outfile: \n ... outfile.write("\n".join(itemlist)) \n ... outfile.write("\n") \n ... \n >>> (Written as if from the interactive Python prompt to make indentation clear. Written for my case, where lists will (mathematically) never be longer than 50.

[luky]

this is much better for multiline strings

[Frederick Reynolds]

@Anoldmaninthesea. rb is to read as bytes, and wb is to write as bytes.

[Eldwin]

thank you for adding the output, immensely helpful

[starriet]

@Dave why 2x space compared to the loop?

[starriet]

@RobM Aren't other answers the same? Even if we use for loop, the entire list should be loaded to the memory, no?

[RobM]

@starriet: If you are writing a list to a file, the list will indeed be in memory but its string representation might not be. Consider the difference between looping through the list calling str() on every item (and storing a new list of strings) which you then iterate again to write out, versus doing it in one loop where you call str() just before writing that one item out, and then freeing the memory held by that one string.

[starriet]

@RobM Ah, so for example, the selected answer consumes less memory, right? Thanks! :)

[RobM]

@starriet: Yes, the main difference is going to be (peak) memory usage. See my own answer where I explore this: stackoverflow.com/a/5032851/83100

[John Glen]

It is not laziness, it is the pythonic way! Plus it works for non-strings.

[lsm_datsci]

THis leaves a dangling empty line

[titus]

what is this doing?