Writing integers in binary to file in python

Question

How can I write integers to a file in binary in Python 3?

For example, I want to write 6277101735386680763835789423176059013767194773182842284081 to a file in binary in exactly 24 bytes (unsigned, I will only be working with positive integers). How can I do this? I tried the following:

struct.pack("i", 6277101735386680763835789423176059013767194773182842284081)

This results in

ValueError: cannot fit 'int' into an index-sized integer

I have tried the same with some other formats ("l", "Q"), but those also result in errors:

struct.error: argument out of range

If I could convert the integer to exactly 24 bytes, I would be able to write to the file, since I know how to do that. However, I can't seem to convert the integers into bytes.

Also, how do I make sure that exactly 24 bytes are written per integer? I will also be writing smaller numbers (1000, 2000, 1598754, 12), however those should also take 24 bytes.

And how can I read the integers again from the file afterwards?

Solution 1

With Python 3 you can do the following:

i = 6277101735386680763835789423176059013767194773182842284081
with open('out.bin', 'wb') as file:
    file.write((i).to_bytes(24, byteorder='big', signed=False))

with open('out.bin', 'rb') as file:
    j = int.from_bytes(file.read(), byteorder='big')

print(j)

Output:

$ python3 tiny.py
6277101735386680763835789423176059013767194773182842284081

Solution 2

You can extract the least significant byte with

  x = value & 0xFF

and you can remove that byte from the number with

  value = value >> 8

Repeating this procedure 24 times will give you the bytes.

You can also speed up this process by taking out more bytes, for example with

 x = value & 0xFFFFFFFF
 value = value >> 32

you're processing 4 bytes at a time.

Comments

[Joran Beasley]

may want to make chr(value & 0xFF) but I removed my answer in favor of this answer(and +1'd you) (since mine was wrong ... I didnt really read the question right)

[Dasherman]

Why do you not need to indicate that 24 bytes should be read from the file? Does byteorder="big" already indicate this?

[Anton Zuenko]

No, there's no limit in read part. File contents would be converted into a number. If you want to read only first 24 bytes, replace file.read() with file.read()[:24]. byteorder='big' is a convention on how to store numbers en.wikipedia.org/wiki/Endianness

[Dasherman]

What does byteorder="big" indicate exactly?

[Anton Zuenko]

It means that if you open the file in hex editor and see 0xFF 0x00 it means that you had written 0xFF00 (65280)

[Dasherman]

Will removing the "byteorder="big"" argument make the script write to the file in 'normal' binary format? Since that is what I am looking for (00000000=0, 00000001=1, 00000010=2, 00000011=3, etc.), or does "byteorder="big"" already do that anyway?

[Fred S]

Endian is just the convention for byte order. Each 8-bit byte is stored as you show, but the order of the 24 bytes in your really long integers depends on your "endianness". As long as you read the same way as you write, doesn't matter if you pick big or little.

[clankill3r]

With python3 I get write() argument must be str, not bytes