Remote Access error 789

Question

I need some help with our VPN solution, if you would be so kind :)

The full error message is

Remote Access error 789 - The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer

This only happens on one single users PC, and only when he is on his home network. No other clients is having issues.

I've tried to change the network profile from Public to Private. Did not help. I've also tried to change it to Domain network, and that didn't help either.

We've got Symantec Antivirus on all machines, but again, there hasn't been any troubles like this on ANY other client.

The firewall SEEMS to be turned off when on Private and Domain network, but not on Public. But this is controlled by the antivirus.

I haven't got immediate access to his router, and would prefer if this could be leaved "untouched"

The connection is a L2TP/IPsec, requires encryption, with a Pre-shared key. The clients use a batch, which is using rasdial to connect.

The server itself, is a Meraki firewall.

Please, ask any questions if you have any.

Solution 1

From: Fix Windows 10 VPN error 789 connection failed due to security issues:

Remote Access error 789 pops up when your system is not properly set up to connect to an L2TP server, thus the connection attempt fails even before you establish a connection with the server.

It is also linked to incorrect configuration of your operating system like Windows 10 in this case. This generic error is thrown when the IPSec negotiation fails for the L2TP/IPSec connections.

Other possible causes include:

• L2TP based VPN client (or VPN server) is behind NAT.
• Wrong certificate or pre-shared key is set on the VPN server or client.
• Machine certificate or trusted root machine certificate is not present on the VPN server.
• Machine Certificate on VPN Server does not have ‘Server Authentication’ as the EKU.

Here are solutions you can use to fix Windows 10 VPN error 789 on your computer.

1. Reset network adapter
2. Check the certificate
3. Re-enable IPSec on your computer

Before trying any of these solutions, ensure that L2TP and IPSec pass-through options are enabled from your router. If you configured your VPN service manually, then make sure you use the preshared key.

1. Reset network adapter

   • Right-click Start and select Device Manager.

     

   • Find Network adapters and click to expand the list

     

   • Identify your network adapter and right-click on it, then select Uninstall.

   • Click OK.
   • Restart your computer. The device will reinstall and should reset it to default settings.

   
   If this doesn’t fix error 789, try the next solution.

2. Check the certificate

   Ensure the correct certificate is used both on the client and the server side. In case Pre Shared Key (PSK) is used, ensure that the same PSK is configured on the client side, and the VPN server machine.

3. Re-enable IPSec on your computer

   • Right-click Start and select Run.

     

   • Type services.msc

   • Find 'IKE and AuthIP IPSec Keying Modules'.

     

   • Find 'IPSec Policy Agent'.

     

   • Check the status. If it says 'started' click to restart. If the 'started' option is disabled, enable it.

   • Double-click on each of the two.

   • Select Startup type.

     

   • Change it to Automatic.

     

   • Save the changes.

   • Restart your VPN service.

Once you have done all the steps above carefully, the VPN should work smoothly as the protocol settings have been reset to default. If, however, it doesn’t work, you have to manually set the encryption method both for the server and the client side, in order for them to be compatible.

In case you have a user-specific issue on your computer yet you still get error 789 after trying any of the above solutions, you can also contact the customer care or tech support team for your specific VPN provider and share the details for further assistance.

Solution 2

It can be a protocol incompatibility (AES/3DES, etc).

• Check the firewall settings
• UDP port 500 and 4500 should be NATed and 1701 forwarded
• Run services.msc as administrator.
• Find “IKE and AuthIP IPsec Keying Modules” and “IPsec Policy Agent” Check the status, right click to “restart” if it states “started” If the “started” option is disabled, enable it.
• Right click and scroll down and click on Properties Select the “Startup type” and change it to “Automatic” and save Restart your VPN and it should work smoothly now as the protocol settings should reset to default
• If that does not work, you will have to manually set the encryption method for both server and clients so they are compatible.

Solution 3

Though this solution is implied by a number of other answers, the only thing I had to do is described by the third option on this blog. I copy it here in case the link goes bad:

• IKE and AuthIP IPsec Keying Modules disabled: Solution: This occurs most often when 3rd party VPN software has been installed and disables the IKEEXT service. This can be re-enabled by navigating in Windows to Control Panel > Administrative Tools > Services. Find the service named “IKE and AuthIP IPsec Keying Modules” and open it. Change the Startup type to “Automatic”. it may be necessary to remove the 3rd party VPN software.

In my case, I didn't have to uninstall any 3rd party VPN software. I happened to be running Windows 10 (1803) at the time.

Solution 4

The January 2022 Cumulative Update for Windows 10 can interfere with IKEv2 VPN connections on some version of Windows 10 [KB5009543 (OS Builds 19042.1466, 19043.1466, and 19044.1466)]. The update has not been fixed as of January 20, 2022 so it seems the only remedy is to uninstall and block it for now.

This blog post suggests downloading the Windows Update Troubleshooter if you don't already have it.

Uninstall KB5009543 by going to:

1. Control Panel -> All Control Panel Items -> Programs and Features, then,
2. click 'view installed updates'
3. select the update for KB5009543,
4. click 'Uninstall' (at top of installed updates list).
5. click 'restart now' to reboot your computer.

Run the above mentioned troubleshooter:

1. click next,

2. click Hide updates to †see a list of available updates and select KB5009543.

   †note: you must have automatic updates turned on in settings for this to be work.

Hopefully one of the next cumulative updates will repair this issue.

Solution 5

In my case, the network administrator changed the vpn server configuration, so I got error 789, after changing my vpn config from VPN Type L2TP to the correct PPTP or Automatic, it worked.

View more solutions

Comments

[harrymc]

I assume the client is on Windows - which version? Error 789 is a generic error thrown when IPSec negotiation fails. The reasons can be a bad key, firewall blocking ports 500 or 4500, or in Control Panel > Administrative Tools > Services the service named "IKE and AuthIP IPsec Keying Modules" is not started - in that case start it and if that's the problem change its Startup type to "Automatic". Same for the "IPsec Policy Agent" service.

[Thomsen1707]

It is Windows 7 (pro i believe). The key is good, i've double-triple checked that. I will check the firewall settings, but that is controlled by the Antivirus. Which every other client has the same version of. I will check the services also. Thank you!

[Thomsen1707]

I will try that. Thank you! But doesn't it seem odd, when he can connect via VPN, if he is anywhere but on his home network?

[Overmind]

Default type of protocol encryption can be changed by various apps so I never exclude this issue when debugging VPN connections.

[Thomsen1707]

Didnt seem to help. The client still can't connect.. Have you got any more up your sleve?

[Overmind]

On the problem-client, go to the IPsec Settings tab. IPsec defaults, enter Customize. Under Data protection, select Advanced, and then Customize. Under Data integrity and encryption, select an algorithm that is compatible with the destination. If that is good, use an external port checker to see if the ports are open.

[Thomsen1707]

I will check that, thank you :) Also i told the guy to send me his home network public ip. The i can check the ports. Port 500, 4500 and 1701 should be open? Or am i getting that wrong?

[Overmind]

Only 1701 should be open/forwarded, the others can be closed, but they must not be intentionally blocked. Open means OK from out-to-in and reverse, closed means OK from in to out, but not out to in and blocked means denied in any direction.

[Frank Lesniak]

"LT2P based VPN client is behind NAT" - is this requirement for real? Practically, wouldn't almost all VPN clients be behind NAT?

[SeriousM]

It's also possible to just pause the windows10 update for the next 7 days, maybe this is enough to survive this problem

[SeriousM]

a hotfix kb5010793 was released to patch the issue qtithow.com/2022/01/… but its distribution is very slow...