Use custom SSH authentication mechanism (OpenSSH)

Question

I have a collection of servers (Debian/Ubuntu) for which I need to create a custom ssh authentication mechanism.

I found a couple of 'same-ish' examples for reference with Duo and Authy. They are using their own API to do the authentication but the idea of delegating the ssh authentication process to a separate application is what I'm trying to achieve.

Ideally I would like to write the authentication program in C++ but I'm happy to script it if thats what is needed.

Solution

If you just want to implement your own way of checking the entered password, then write a PAM module. It is a simple .so library that contains a pam_sm_authenticate() function. (It can also provide functions for authorization, password-change, etc.)

Start with something like this (in C), compile it as a .so library using -shared, copy to /lib/security/pam_foobar.so, and finally add it to the server's PAM config (/etc/pam.d/sshd) – possibly replacing the standard pam_unix module.

#define PAM_SM_AUTH

#include <security/pam_modules.h>
#include <security/pam_appl.h>

int pam_sm_authenticate(pam_handle_t *pamh, int flags, int argc, const char *argv[]) {
    return PAM_SUCCESS;
}

struct pam_module pam_foobar_modstruct = {
    .name                 = "pam_foobar",
    .pam_sm_authenticate  = pam_sm_authenticate,
    .pam_sm_setcred       = NULL,
    .pam_sm_acct_mgmt     = NULL,
    .pam_sm_open_session  = NULL,
    .pam_sm_close_session = NULL,
    .pam_sm_chauthtok     = NULL,
};

For more details see the Linux-PAM Module Writer's Guide as well as various other sources.

[You can use pamtester to verify the module locally, before messing with sshd's configuration.]

---

Note, however, that pam_sm_authenticate() is only called for password-like logins – in SSH that means either the password or kbd-interactive auth methods. It is not called for checking SSH keypairs or Kerberos tickets – the former is internal to sshd, the latter is done by libkrb5.

(Also note that PAM has a separate authorization phase for checking whether a user is allowed to use the system. The SSH server always consults PAM for authorization, regardless of what authentication method was used. For doing this, implement pam_sm_acct_mgmt() in your module.)

Comments

[tvdo]

There's also a pam_exec that can execute arbitrary external commands without having to create a custom PAM module.