how to transfer log file to another linux server for processing

Question

Is there an reliable way to transfer logs from one server to another?

Currently i'm using cron script to transfer file with scp to another server every 5 minutes.

*/5 * * * * root

scp ~/my_log_file.txt user@remote_host.com:/remote/log/directory

Solution

1. My preferred method as suggested by ryekayo is to just send the syslog messages to local file AND to remote host:

/etc/rsyslog.conf:

authpriv.* /var/log/secure
authpriv.* @remote_host.com

(This part of the facility I think works with pretty much all syslog daemons. The part about sorting things out at the other end is specific to rsyslog.)

then at the remote end you need to setup the receiving server to filter the remote messages into the appropriate place (by sending host with %HOSTNAME% as part of file name, by facility %syslogfacility-text%, program name %programname%, etc.)

2. As you say, you can use scp with cron
3. Alternatively, I think if you use rsync

   rsync ~/my_log_file.txt user@remote_host.com:/remote/log/directory

   with cron you can save recopying the first part of the log file (I express uncertainty since you definitely save COPYING the already transferred portion, but you still do a comparison on each side to locate the already transferred portion to skip. This should be gentler on SSDs and avoids dirtying kernel IO buffers, but it is still incurring IO ops to scan the beginning at both ends.).
4. NFS (or Ceph/GlusterFS/SMBFS) mount one machine's directory onto the other, and do a continuous tail onto the other machine.

Those are the options I can think of.

Depending on your goal for transferring the logs, you can be more efficient by including the transfer in the post-rotate of logrotate if the goal is to store them long term on a bigger disk. If the goal is to have access to the logs from the last 5 minutes (eg for OSSEC), then the above transfer methods are better.

The other thing to note is that if you feel the machine generating logs is unreliable, or insecure, it might be better to have the log-storing host copy from the log generating machine than give it permission to write to the remote machine.

Comments

[ryekayo]

There is a way of doing this w/o using scp. You can do this using rsyslog. Take a look at the following link: tecadmin.net/setup-centralized-logging-server-using-rsyslogd