How to delete logs automatically after a certain time and restart the process that fills up the log file?

Question

Server is Ubuntu 16.04. I have a process running with nohup that logs into a local file. This file gets big enough to consume 100% disk space. To resolve this I have to kill the process first and then remove the log file. Then I restart the process. How can I resolve this with a script or some other tool?

Solution 1

I guess that you start the script/program with nohup like

nohup scriptname 1>logfile.log 2>& &

I would recommend instead of deleting the log file just to clear it with

echo -n >logfile.log

If you delete/move an open file it will be written until the process will close the file or the process will end.

Solution 2

With logrotate you can configure how big a log file may get or after how much time:

• the log files are rotated (log.n becoming log.n+1, and the last log file being deleted)

• the current log file is truncated without disturbing the writing process.

Take a look at man 8 logrotate.

Comments

[DEVCNN]

Solves my problem. Thanks.

[DEVCNN]

Maybe this would have worked if the file wasn't open at the time i run logrotate. It tries renaming the file and fails. nohup keeps streaming data into the log file. So i guess it must be open all the time.

[rexkogitans]

Useless use of echo. Just >logfile.log

[0x0C4]

using "echo" doesn't hurt. it's build-in

[Andrew Henle]

This is an extremely unreliable way to free disk space from a full log file. The process writing to the log file has a current offset, and truncating the file with > doesn't change that offset. The next time the file gets written to, you'll likely get either a sparse file if the underlying filesystem supports sparse file, or you'll wind up with a file the same size, almost entirely full of \0/NUL characters and taking up the same amount of space.

[Andrew Henle]

@DEVCNN You are learning why using standard output and redirecting it to a log file is just about the worst possible approach to logging. Your log file is tied to the logging process, and you can't do anything about it. The actual solution is to use a proper logging system that would allow you to separate the actual log file from the process and allow you to use tools like logrotate. There are a lot of really good reasons things like syslog and logrotate exist - and you're learning a lot of them.

[DEVCNN]

Sure. I'll look into that.

[Mark Wagner]

@AndrewHenle this answer is simply wrong since the shell is being used for redirection. However, if the redirection is changed to >> then it works because the log file is opened with O_APPEND. See unix.stackexchange.com/questions/202797/…. In general though, your warning is correct. All log files should be opened with O_APPEND to make rotation easy but alas some programs don't do that.

[DEVCNN]

@MarkWagner So if i use >> instead of >, can i use logrotate then? Does >> open the log file only when writing?

[0x0C4]

">>" append data to an already existing file. so if you re-direct output during calling the process again ("nohup script 1>>logfile.log 2>&1") it would append output of stdout and stderr to "logfile.log" and this has nothing to do with the work of logrotate. This is only about how it works with already existing data in a already existing log-file.

[DEVCNN]

Ok. I get it. Seems like I have a lot of reading to do.

[Mark Wagner]

@DEVCNN If you use >> you can use the simple >logfile.log method of log rotation without having to restart the process which was the main goal. You can implement rotation with a cron job, logrotate, or probably systemd (what can't it do?).