Best way to go about changing a root password on 3000+ Solaris, AIX, and Linux servers?

Question

Long story short: Big old corporation, lots of UNIX/Linux servers.

I inherited responsibility for a bunch of scripts that that left a few years ago. One of them was a script that would be run every $X amount of months to globally update the root password across all of our servers.

The script is a mess of Shell Script and Expect, and it works on the SSH trust that is set up between all of our servers and a central command-and-control server.

The problem is, the script is a giant mess. The Expect commands are trying to account for every possible version of "passwd" that exists on any UNIX/Linux box out there - and they vary quite a bit.

As we're expanding and upgrading a lot of our infrastructure, the script is getting really unmanageable.

My question is: Is there a better way to do this? Assuming there's an already established SSH trust, what's the best way to change the root password on 3000+ servers at the same time?

Solution 1

Use Puppet.

Puppet is very flexible, easy to maintain and uses SSL. Might sound a bit overkill and you'll have to put some extra effort to build Puppet system up.

But. Most probably this is not the last mass-update you'll be doing to these machines. Puppet will and does save you a lot of time when actual whatever mass-update procedure begins and scripts are very readable/reusable.

At least this worked for me few years back and still I'm able to re-use some of those Puppet recipes (aka scripts). I have also used it in a bit smaller environments just make sure that every machine is actually having a known state.

I have proven it many times (at many companies) that all customised deployment scripts become pain in the butt after a while or when that next guy steps in. And as long as you carry a cellphone, old scripts will and do haunt you.

If you think that this actually sounds good, here's a great Puppet tutorial with virtual environment included to get you started.

Solution 2

I've used the Perl module Authen::PAM on Solaris with great success. Here's a sample script:

#!/usr/bin/perl

use Authen::PAM;

my $username = 'root';
my $password = '1234567';

die qq{Error: Unknown user\n} unless getpwnam($username);

die qq{Error: You must run this as root.\n} unless ($> == 0);

my $pamh;

sub my_conv_func
{
    my @res;
    while ( @_ )
    {
        my $code = shift;
        my $msg = shift;
        my $ans = "";

        if ($code == PAM_PROMPT_ECHO_OFF() )
        {
            if (($msg =~ /^New Password:/i) or ($msg =~ /^Re-enter new Password:/i))
            {
                $ans = $password;
            }
            else
            {
                die qq{Unknown message: $msg\n};
            }
        }
        else
        {
            print qq{$msg\n};
        }

        push @res, (PAM_SUCCESS(), $ans);
    }
    push @res, PAM_SUCCESS();

    return @res;
}

ref($pamh = new Authen::PAM("passwd", $username, \&my_conv_func)) || die "Error code $pamh during PAM init!";

my $res = $pamh->pam_chauthtok;

print $pamh->pam_strerror($res),"\n" unless $res == PAM_SUCCESS();

exit 0;

Solution 3

If you can write Perl, the module Net::OpenSSH::Parallel allows to write scripts that perform actions in parallel on remote hosts via SSH quite easyly.

It contains a sample script for changing passwords that you can use as a base. As it seems that you have an heterogeneous environment you would like to group the hosts by type and use a different dialogue handling sub for every one.

Solution 4

This is my solution so far. still need to see if it works on multiple systems

#!/usr/bin/env bash

ChangePassword()
{
    echo "changing password for server $ServerIp"
    ssh root@$ServerIp "echo root:${NewPassword} | chpasswd" < /dev/null
}

CreatePassword()
{
    while true;
    do
        echo "Please enter the new password :"
        read -s NewPassword <&0
        echo "Confirm the password :"
        read -s ConfirmPassword <&0 
        # /proc/${PPID}/fd/0

        if [ "$NewPassword" == "$ConfirmPassword" ]
        then
            break
        else
            echo "Passwords do not match"
            echo "Try again..."
        fi
    done
    ChangePassword
    echo "end of createpassword"
}

SetUpPasswordlessSSH()
{   
    echo "enter the old password from server $ServerIp when asked"
    ssh root@$ServerIp mkdir -p .ssh
    cat .ssh/id_rsa.pub | ssh root@$ServerIp 'cat >> .ssh/authorized_keys'

    echo "Passwordless SSH is now available"
    echo "Now you can change the password"
    CreatePassword
}

NoSSH()
{
    echo "Passwordless SSH for this server with ip $ServerIp is not yet set up."
    read -p "Do you want to set it up now? " -n 1 -r <&0
    echo "" 
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

AcceptHostKey()
{
    read -p "Do you trust the server? " -n 1 -r <&1 
    echo ""
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

Main()
{
    while read -r ServerIp <&9
    do
        echo  "Server $ServerIp ..."
        status=$(ssh -o BatchMode=yes -o ConnectTimeout=5 $ServerIp echo ok 2>&1)
        if [[ $status == ok ]]
        then
            echo "creating password"
            CreatePassword
            echo "password changed"
        elif [[ $status == "Permission denied"* ]]
        then
            NoSSH
        elif [[ $status == "Host key verification failed"* ]]
        then
            echo "Error: $status"
            AcceptHostKey
        else
            echo "ERROR OCCURED FOR SERVER WITH IP: $ServerIp"
            echo "Error: $status"
        fi
    done 9< servers.txt
    history -cw
    clear
}

Main $*

Solution 5

Recently I have done this Using Bash Script..

#!/usr/bin/env bash

# Change Password of Remote Server Using SSH

#--------------------------------------------
# Define User which you want to
# Change Password in remote server
#--------------------------------------------
Uname="root"
# Create Password in Encrpyted Form Using below command,
# and store in this script
# perl -e'print crypt("YourPassword", "salt")' ; echo -e
# then copy and past in following varible,
# password should be single qouted*

Password='safv8d8ESMmWk'

Update_Pass() {
  ssh $ruser@$Server_ip  -p $port "echo ${Uname}:${Password} | chpasswd -e"
}

Show_Help(){
cat <<_EOF
Usage $0        [OPTION]..
Mandatory arguments to long options are mandatory for short options too.
  -i, --ip     <IP_ADDR_OF_SREVER> IP Address Of Remote Server
  -u, --user   <Username>          Username Of Remote Server    <Default User is root>
  -p, --port   <port>              Port Of Remote Server        <Default is 22>

Note:- For Security Reason Do Not Provide Password to the script, because
       it will get save in history, so do not provide it,
       script will prompt for password

Report $0 bugs to [email protected]
_EOF
}



Main() {

        case $1 in
           -i|--ip) Server_ip=$2;
                    ruser="$4"; [[ -z $ruser ]] && ruser=root
                    port="$6";  [[ -z $port  ]]  && port=22
                    Update_Pass ;;
                *)  Show_Help ;;
        esac
}

Main $*

View more solutions

Comments

[egorgry]

I don't know what to say but this sounds like a security nightmare. can you ldap? You really need to rethink the entire current approach. How much time and resources do you have and can you get a project approved to re-do the whole thing?

[EEAA]

Hire an intern? :)

[the-wabbit]

@DmitriChubarov using "sudo" would imply having a different account to maintain (and change) a password on - so where is the point?

[Zachw6]

@syneticon-dj the other accounts may be created afresh and stored on some centrally maintained database e.g. LDAP or NIS whatever is better supported. It is not the same thing as ripping away local root and replacing it with the one coming from ldap.

[Philip]

Possible duplicate of: serverfault.com/questions/493090

[fizzlepapers]

We already have several other security measures in place: Sysadmins log into boxes via LDAP-managed accounts. We do use sudo to get elevated privileges. However, the root account still exists and still has a set password. In the event of total failure of other authentication methods, we still need to be able to log in directly as root via a the console port or other management connection.

[warren]

If you've got >3k machines, you need management tools ... yesterday!

[kasperd]

I think the script shouldn't even see the password in the first place. Find a decent hashing function supported across most of the platforms and choose a password strong enough for that hash, then put that hash in the script. There may be a minority of systems that don't support the hash function, use a different password for those and consider phasing them out.

[Philip]

And what command are you going to run? passwd is different on the different versions mentioned. pw not always available....

[AngerClown]

passwd on RHEL Linux at least does have a --stdin parameter

[mpez0]

I believe editing /etc/shadow is the way to go. However, won't the redirect wipe out /etc/shadow before the edit? This looks like a case for ed or ex.

[tink]

The format of shadow may be, but the hashes aren't guaranteed to be interpreted by the same algorithm everywhere. Plus your little script just removed all entries from shadow altogether. :}

[Dennis Kaarsemaker]

Agred, but do not use a user{"root":} to set the password. Instead, use an exec{"chpasswd -e ..."}, which is much safer.

[dawud]

Please correct your answer, puppet uses SSL, not SSH. Agree with the rest, though ansible seems like a lightweight alternative much better suited for this use case. Just pip install ansible in a GNU/Linux box, build a list of hosts and ansible -m user -a "password=$crpyt". No agents needed. Manages GNU/Linux, AIX and Solaris out of the box.

[HaxElit]

lol oops, should be fixed. I hope you didn't sudo run that tink ;)

[tomi]

True it uses SSL, not SSH. I use to be maintainer in a system where Puppet was used via SSH by executing a client - long story short. Answer corrected. Thanks dawud pointing that out!

[fizzlepapers]

I provided my own answer below, but I'm accepting yours, as it's probably the most "correct" and manageable long-term solution to the problem. However, it does require a significant investment in terms of infrastructure, planning, licensing, and time in order to roll out Puppet in an environment our size. Some day we'll get there...

[Marcin]

Please do not put cleartext passwords on commandline, as it gets logged in shell history. Or set the shell history to /dev/null before you do it.

[Tom Marthenal]

@DennisKaarsemaker why is that safer?

[Dennis Kaarsemaker]

user{"root":} uses the libshadow library, which broke more than one system for me. I have far more faith in good old chpasswd.

[Rahul Patil]

yes , also do not put password in script, check my script to add encrypted passwd

[Bill Weiss]

@DennisKaarsemaker: you should report that as a bug to Puppet Labs. They would be super interested in hearing it.