backup and restoration of a freeipa infrastructure

Question

I'm finding the documentation on ipa server backup and restoration sadly lacking, and being so centrally critical it's not something i'm really happy about shooting in the dark with - could some kind soul more knowledable in the matter please attempt to provide an idiot-proof guide to backing up and restoring of IPA server(s) ? Particularly the main server (the cert signing one).

...We're looking towards rolling out ipa in a two server setup (1 master, 1 replica). I'm using dns srv records to handle failover, hence a loss of the replica isn't a big deal as i could make a new one and force a resync to happen - it's losing the master that bothered me.

The thing that i'm really struggling with is locating a step-by-step procedure for backing up and restoring the master server. I'm aware that whole-VM snapshot is the recommended way of doing IPA server backup, but that isn't an option at this time for us.

I'm also aware that freeipa 3.2.0 includes some sort of backup command build in, but that isn't in the ipa version of centos, and i don't expect it will be for some time yet.

I've been trying many different methods, but none of them seem to restore cleanly, amongst others, i've tried;

• a command similar to db2ldif.pl -D "cn=directory manager" -w - -n userroot -a /root/userroot.ldif

• the script from here to produce three ldif files -- one for the domain ({domain}-userroot), and two for the ipa server (ipa-ipaca and ipa-userroot):

Most of the restores i've tried have been similar to the form of: ldif2db.pl -D "cn=directory manager" -w - -n userroot -i userroot.ldif

which seems to work and reports no errors, but totally borks the ipa install on the machine and i can no longer login with either the admin password on the backed up server, or the one i set it to on installation before attempting the ldif2db command (i'm installing ipa-server and running ipa-server-install, then attempting the restore).

I'm not overly bothered about losing the CA, having to rejoin the domain, losing replication etc etc (although it'd be awesome if that could be avoided) but in the instance of the main server dropping i'd really like to avoid having to re-enter all the user/group information.

I guess in the instance of losing the main server i could promote the other one and replicate in the other direction, but i've not tried that, either. Has anyone done that ?

tl;dr: Can someone provide an idiots guide to backing up and restoring an IPA server (preferably on CentOS 6) in a clear enough way that'd make me feel confident it'll actually work when the dreaded time comes ? Crayons are optional, but appreciated ;-) I can't be the only person struggling with this, seeing how widely used IPA is, surely ?

Solution 1

I don't have a proper solution to backup and restore a FreeIPA server on CentOS, only a workaround to have a server operative with the same configuration in the shortest time possible. You do lose the CA and you need to rejoin the hosts to the server.

This is the way I dealt with "disaster recovery" while using the 2.x series. I did many trial and error experiments and got tired of restoring my settings from scratch:

1. provision a new host using DHCP+PXE+TFTP+Kickstart.
2. ensure the kickstart script installs the puppetlabs repo and register itself with the puppetmaster, there is (was) an entry in autosign.conf for this purpose. (The puppetlabs repo is not mandatory, but I was using syntax not present in the stock version of puppet).
3. write a module containing a package resource to have the server and its dependencies installed, and an exec resource to run a shell script (all kept under version control) defining all the infrastructure needed in the domain.

I'll give you a snippet of the script here, you get the general idea:

#!/usr/bin/env bash
# vim:syn=sh:ts=2:fdm=marker
# IPASERVER BOOTSTRAP {{{
# HOSTGROUPS {{{
# foo {{{
ipa hostgroup-add foo --desc='Foo Bar Baz'
ipa hostgroup-add-member sanfernando --hosts={foo,bar,baz}.domain.com
ipa netgroup-add net_foo --nisdomain=domain.com --desc='Foo Bar Baz'
ipa netgroup-add-member net_foo --hostgroups=sanfernando
# }}}
# }}}
# PWPOLICY {{{
ipa pwpolicy-mod global_policy --history=24
ipa pwpolicy-mod global_policy --lockouttime=1200
ipa pwpolicy-mod --setattr=krbpwdmindiffchars=4
ipa pwpolicy-mod --setattr=krbpwdminlenght=14
ipa pwpolicy-mod --setattr=krbpwdmaxfailure=5
ipa pwpolicy-mod --setattr=krbminpwdlife=168
ipa pwpolicy-mod --setattr=krbpwdfailurecountinterval=1200
# }}}
# USERS/GROUPS/HBAC {{{
# developers {{{
ipa user-add jdoe --first='Jane' --last='Doe' --email='[email protected]' --gecos='Jane Doe' --shell='/bin/rbash' --sshpubkey='AAA......XXGDGHU='
ipa group-add foo --desc='Foo Staff'
ipa group-add-member foo --users=jdoe
ipa hbacrule-add developers_access --desc='Developers access'
ipa hbacrule-add-host developers_access --hostgroups=development
ipa hbacrule-add-user developers_access --groups=developers
ipa hbacrule-add-service developers_access --hbacsvcs=sshd
ipa hbacrule-add-service developers_access --hbacsvcgroups=Sudo
# }}}
# }}}
# SUDO CMD/RULE/GROUP {{{
# networking {{{
ipa sudocmd-add --desc='administration tool for IPv4 packet filtering and NAT' '/sbin/iptables'
ipa sudocmd-add --desc='view and manipulate media-independent interface status' '/sbin/mii-tool'
ipa sudocmd-add --desc='display or change ethernet card settings' '/sbin/ethtool'
ipa sudocmd-add --desc='show and manipulate routing, devices, policy routing and tunnels' '/sbin/ip'
ipa sudocmd-add --desc='sudoedit configuration file of IPv4 packet filtering and NAT' 'sudoedit /etc/sysconfig/iptables'
ipa sudocmdgroup-add networking --desc='commands for network configuration and troubleshooting'
ipa sudocmdgroup-add-member networking --sudocmds=/sbin/{iptables,mii-tool,ethtool,ip}
ipa sudocmdgroup-add-member networking --sudocmds='sudoedit /etc/sysconfig/iptables'
ipa sudorule-add networking_4_operators_2 --desc='Operator Level 2 access to networking management commands'
ipa sudorule-add-allow-command networking_4_operators_2 --sudocmdgroups='networking'
ipa sudorule-add-user networking_4_operators_2 --groups='operators_2'
ipa sudorule-add-host networking_4_operators_2 --hostgroups=foo-hosts
# }}}
# }}}
# }}}

Solution 2

FreeIPA (now branded as Red Hat Identity Manager on RHEL) as of 4.x includes CLI tools: ipa-backup (producing a GPG-encrypted backup dump of either all server information or LDAP data only) and ipa-restore.

One caveat: after restoring a data-only backup on a freshly installed server I found that it clobbered the admin credentials and I couldn't log in as admin or any other use no matter what I did ("incorrect password" error message). So it was pretty useless. This could have been a quirk of my setup, however. Be sure to test restore before relying on it!

Solution 3

There is no simple answer to backup and restore in FreeIPA. We are talking about a multi-master replication environment, where every server may have different configuration, different service (CA, DNS) which we want to preserve.

Since the beginning of the project, a common answer to backup and restore in FreeIPA was - have replicas. With multiple FreeIPA replicated servers in you keep the redundancy and availability even when some of the server crashes. If you have other services like CA or DNS, simple have more of those on FreeIPA replicas to avoid being caught off guard. If you use DNS SRV records to discover FreeIPA services by your clients, that they should not even notice if you loose some, they should simply switch to other available FreeIPA.

It may be also wise to do regular VM snapshots of one FreeIPA backup server to make sure that you have something to start with in case your whole replicated environment breaks. If you are interested not in structural backup, but also data backup, I would advise db2ldif as the best way of backing up the data.

I hope this brief advice helps, you can find more resources at FreeIPA.org site:

• Backup and Restore strategies
• Experimental Backup and Restore scripts introduced in FreeIPA 3.2.0

Comments

[Sirex]

yeah so this is basically recreating the directory structure as a batch file. I don't think that'll really work for us as there's several hundred users and groups, and it'd be unmaintainable to make the bootstrap script stay in sync.

[dawud]

Indeed. As I said, it's just a workaround setup the basics. You might be able to import a complete ldif onto such a server, though.

[Sirex]

ill try. so far i've had no sucess importing the ldif, but i have heard from people that it can be done... but they didn't say how ;p (i used ldif2db.pl)

[Sirex]

yeah, we take a snapshot of one node for just this reason, because replicas are not backups, the same way RAID is not a backup.

[Martin Kosek]

It is not a full backup solution per se, but it should cover redundancy and availability of IdM for most scenarios. The snapshots you referred to should be only needed in special or catastrophic scenarios. If I am mistaken based on your practical experience, please correct me.