Checking for read only file systems remotely

Question

I need to remotely periodically check if a Linux file system is mounted read only on lots of servers. The only tools at my disposal in this instance are the usual CLI scripting languages (BASH/Perl/Python), SSH and SNMP.

My thoughts so far are:

• Share SSH key and execute the mount command remotely, grep/awk-ing for the RO flag on the given mountpoint.
• Check via SNMP, although I'm not certain if this information is available or how easy that is.
• Attempt to SCP a file across, or remotely touch a file.

The SNMP method is the nicest/cleanest I can think of, however I'm at a loss on how to do this as CLI SNMP seems like a nightmare, especially if I need to run through a decision process to find an index based on a given mountpoint.

Ideally I'd like to pull this off using only BASH (inc. sed/awk/grep etc) and SNMP but need some guidance on where to begin! snmpd is already instaled and configured for v2 on all the servers in question.

The final use of this script will be as a Nagios command.

Many thanks in advance for any help!

Note We already have cacti/nagios set up monitoring 100s of servers, I forget to mention this when originally posting!

Solution 1

In your snmpd.conf check out the exec argument. I use this to remotely poll the usercount of some of my IRC servers, like this:

exec usercount-irc /home/irc/usercount-irc.sh

To poll this you snmpwalk .1.3.6.1.4.1.2021.8, and would get something like this:

# snmpwalk -v 1 -c public localhost .1.3.6.1.4.1.2021.8

UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extNames.1 = STRING: usercount-irc
UCD-SNMP-MIB::extCommand.1 = STRING: /home/irc/usercount-irc.sh
UCD-SNMP-MIB::extResult.1 = INTEGER: 0
UCD-SNMP-MIB::extOutput.1 = STRING: 558
UCD-SNMP-MIB::extErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::extErrFixCmd.1 = STRING:

Check the Extending Agent Functionality part of http://linux.die.net/man/5/snmpd.conf for further details!

Edit 1:
UCD-SNMP-MIB:dskTable contains information about mounted volumes and usage. It is located under OID .1.3.6.1.4.1.2021.9 and documented at:
http://www.oidview.com/mibs/2021/UCD-SNMP-MIB.html
http://www.net-snmp.org/docs/mibs/ucdavis.html#dskTable

But I see now it doesn't contain mountflags, so you can't see read only there. Gah. Back to the script above and how to call it! :P

Solution 2

For most "how do I do X in nagios?", you'll find that someone has already written a plugin that will do all of, or at least some of, what you need.

In this case, you could probably use check_ro_mounts in combination with the 'negate' plugin (part of standard nagios plugins), or just modify it to suit your needs:

http://exchange.nagios.org/directory/Plugins/Uncategorized/Operating-Systems/Linux/check_ro_mounts/details

Solution 3

Another way is to use Perl's, Ruby's Net::SSH or Python's paraminko libraries to remotely login and (no need to setup keys beforehand) then parse the contents of the 'mount' command.

Comments

[S19N]

Use the following via ssh: serverfault.com/questions/193971/…

[Zoredache]

Having you considered using something like monit instead? It has tests specifically for checking the state of mounted filesystems.

[SimonJGreen]

@Zoredache Monit is to distributed for this use case unfortunately. We already have cacti/nagios set up monitoring 100s of servers, this is something we'd like to be able to just drop in on top. I should probably have mentioned that as it's quite an important detail!

[Mattias Ahnberg]

1.3.6.1.4.1.2021 is the MIB to UCD-SNMP MIB tree, .8 is where external scripts is placed; see here for a view of the MIB tree: oidview.com/mibs/2021/UCD-SNMP-MIB.html

[SimonJGreen]

Thanks @Mattias that's not a bad idea. I've used the exec part of snmpd in the past to monitor specifics from other processes so it's a possibility. Unfortunately it would mean going on to 100s of Linux servers to set this up, but it's one to bare in mind!

[Mattias Ahnberg]

D'uh. And AGAIN senile, that MIB tree doesn't contain mount point flags, so I believe you're still forced to use the exec of snmpd or check with, for example, SSH. :/

[SimonJGreen]

I came to the same conclusion sadly :( Hey ho, thanks for the answer!

[SimonJGreen]

That's true in most instances and would normally be my first stop. In this case however we'd like to avoid deploying a script out to 100s of servers if possible. It does look like though the RW state is not represented through the standard snmp mounts MIB so something like this is going to be the way.

[SimonJGreen]

Unfortunately checking the state of mounts can't justify a major change like rolling out puppet. Also in our scenario we need the management of these servers decentralised which kinda goes against the puppet way! Thanks anyway though.

[SimonJGreen]

I'm leaning more towards this answer although sadly would mean storing a password in plain text. It's a strong contender!

[loislo]

@SimonJGreen Puppet with variables and various configuration "blocks" that can be contorted to amazing things. Just feeling the need to evangelize.