Why is data written to a file opened with O_APPEND flag, always written at the end, even with `lseek`?

Question

I have been given a programming assignment:

Write a program that opens an existing file for writing with the O_APPEND flag, and then seeks to the beginning of the file before writing some data. Where does the data appear in the file? Why?

What I have come up with is:

main() {
    int fd = open("test.txt", O_WRONLY | O_APPEND);
    lseek(fd, 0, SEEK_SET);
    write(fd, "abc", 3);
    close(fd);
}

Upon trying the above, I have found that the data is always written at the end of the file. Why is that? Is it because I have indicated O_APPEND?

Solution 1

When you open a file with O_APPEND, all data gets written to the end, regardless of whatever the current file pointer is from the latest call to lseek(2) or the latest read/write operation. From the open(2) documentation:

O_APPEND
The file is opened in append mode. Before each write(2), the file offset is positioned at the end of the file, as if with lseek(2).

If you want to write data the end of the file and then the beginning of it later, open it up without O_APPEND, use fstat(2) to get the file size (st_size member within struct stat), and then seek to that offset to write the end.

Solution 2

In effect, O_APPEND only affects the behavior of write, but not that of read. Whatever how the current position of a file is changed by lseek, write will always append-only.

When you open a file with O_RDWR | O_APPEND, read will still start from the beginning of the file.

In the manual of open (man 2 open),

O_APPEND The file is opened in append mode. Before each write(2), the file offset is positioned at the end of the file.

In the manual of write (man 2 write),

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write.

In the Linux kernel fs/ext4 syscall write -> vfs_write -> ext4_file_write_iter, the ext4_file_write_iter will call ext4_write_checks

then call generic_write_checks

you will find the place where setting the pos = file.size

/* FIXME: this is for backwards compatibility with 2.4 */
if (iocb->ki_flags & IOCB_APPEND)
    iocb->ki_pos = i_size_read(inode);
pos = iocb->ki_pos;

The following demo can verify it.

cat open_append.cc
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

#include <string>
#include <iostream>

int main(int argc, char *argv[]) {
  std::string path = "./test.txt";
  std::string content = "hello_world";
  std::string read_buf(content.size(), 0x0);
  struct stat st_buf;
  ssize_t bytes_read = -1;
  ssize_t bytes_write = -1;
  int ret = -1;
  off_t cur_off = -1;
  int fd = ::open(path.c_str(), O_CREAT | O_RDWR | O_TRUNC, 0644);
  if (fd < 0) {
    std::cerr << "open err path " << path
              << " errno " << errno << std::endl;
    return -1;
  }
  std::cout << "open ok path " << path
            << " fd " << fd << std::endl;

  // Step 1 write some data into an empty file
  bytes_write = ::write(fd, content.data(), content.size());
  if (bytes_write < 0) {
    std::cerr << "write err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "write ok fd " << fd
            << " data " << content
            << " nbytes " << bytes_write << std::endl;
  ::close(fd);

  // Step 2 open the file again with O_APPEND
  fd = -1;
  fd = ::open(path.c_str(), O_CREAT | O_RDWR | O_APPEND, 0644);
  if (fd < 0) {
    std::cerr << "open again err path " << path
              << " errno " << errno << std::endl;
    return -1;
  }
  std::cout << "open again ok path " << path
            << " fd " << fd << std::endl;

  // Step 3 the current position of the file NOT affected by O_APPEND
  cur_off = ::lseek(fd, 0, SEEK_CUR);
  if (cur_off < 0) {
    std::cerr << "lseek err SEEK_CUR fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  // cur_off expected to be 0
  std::cout << "lseek ok SEEK_CUR fd " << fd
            << " cur_off " << cur_off << std::endl;

  // Step 4  the read will start from the beginning of the file
  bytes_read = read(fd, (char*)read_buf.data(), content.size());
  if (bytes_read < 0) {
    std::cerr << "read err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "read ok fd " << fd
            << " data " << read_buf
            << " nbytes " << bytes_read << std::endl;

  // Step 5 change the position to the half of the file size
  cur_off = ::lseek(fd, content.size() / 2, SEEK_SET);
  if (cur_off < 0) {
    std::cerr << "lseek err SEEK_SET fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  // cur_off expected to be content.size() / 2
  std::cout << "lseek ok SEEK_SET fd " << fd
            << " cur_off " << cur_off << std::endl;

  // Step 6 write will append data from the end of the file
  // the current position is ignored
  bytes_write = ::write(fd, content.data(), content.size());
  if (bytes_write < 0) {
    std::cerr << "append write err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "append write ok fd " << fd
            << " append data " << content
            << " append nbytes " << bytes_write << std::endl;

  // Step 7 the file size is double content.size()
  memset((void*)&st_buf, 0x0, sizeof(struct stat));
  ret = lstat(path.c_str(), &st_buf);
  if (ret < 0) {
    std::cerr << "lstat err path " << path
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "lstat ok path " << path
            << " st_size " << st_buf.st_size << std::endl;
  ret = 0;

out:
  if (fd >= 0) {
    close(fd);
  }
  return ret;
}

Output result

open ok path ./test.txt fd 3
write ok fd 3 data hello_world nbytes 11
open again ok path ./test.txt fd 3
lseek ok SEEK_CUR fd 3 cur_off 0
read ok fd 3 data hello_world nbytes 11
lseek ok SEEK_SET fd 3 cur_off 5
append write ok fd 3 append data hello_world append nbytes 11
lstat ok path ./test.txt st_size 22

Comments

[R.. GitHub STOP HELPING ICE]

Calls to lseek are not ignored. The position is honored for reading, and even if you don't read, it's maintained and can be read back. However, on each write the position moves back to the end of the file.

[NikitaBaksalyar]

You can also seek to the end of a file just by using lseek(fd, 0, SEEK_END) (i.e. instead of getting the file size).

[Andrew Henle]

When you open a file with O_APPEND, all data gets written to the end Not all data. Per POSIX, the pwrite function (ssize_t pwrite(int fildes, const void *buf, size_t nbyte, off_t offset);) can write to any offset: "The pwrite() function shall be equivalent to write(), except that it writes into a given position and does not change the file offset (regardless of whether O_APPEND is set)." But pwrite() on Linux is broken and appends data to the end of the file no matter what value is passed in offset.

[Dmitry M]

I have a question about atomicity with Adam's solution -- since the file was not opened with O_APPEND, isn't it possible for another program to write to the file between the first program's seek and write? If so, what's another solution if I need to be able to both append and write to a file regularly?

[lfmunoz]

@DmitryM Exactly why O_APPEND was created, because without it you cannot guarantee writing to the end of the file (because you would have to seek and write)