Is it possible to restart a computer in response to an email notification?

Question

I'm looking for a way to reboot machines after getting an email notification.

The setup I want to use this on is a render farm. We have 5 boxes with 8 Nvidia GTX 980Ti in them. They're running Windows 10 Professional 64-bit.

The issue we are having is that they sometimes freeze or "stall" while rendering. I have it set up so once this happens I receive an email saying "x machine has stalled."

I have automated everything on the machines; they log in automatically, open up the necessary programs, initiate a connection with our license server, and map all the required network drives.

Basically all you have to do is turn it off and back on, and it works again.

So what I'm wondering is, can I set up a program, event, or a script that will tell the machines to reboot after I receive the email saying a machine has stalled?

The machine that will be running this is running Windows 7 Pro 64-bit.

Solution 1

You want a device called a "Remote Power Switch", one that we have used at my workplace to remotely restart cable modems is the UIS-322b below:

You can set it up with an IP address, and then log in to it to power on, power off, or power cycle either device connected, and set it on a schedule. You cannot control it via email, but through the web interface. You could still rig something up via wget or curl to automatically send a web request to it. It may have other features and control possibilities I'm unaware of.

If you are running rendering systems that consume a lot of power I am not sure if the specific device above can handle them, but there may be others out there that do. Another thing to look for is "Controllable Power Strip."

Solution 2

Can I setup a program, event, or a script that will tell the machines to reboot?

It depends on what you mean by:

they freeze or "stall"

If the remote machine has hung in a way that requires a "hard reset" then a remote restart will not work.

Having said that, you can use shutdown to remotely restart a device.

Example:

shutdown /r /f /m \\computer

• /r - Shutdown and restart the computer.

• /f - Force running applications to close without forewarning users.

• /m \\computer - Specify the target computer.

You can run the above command from Outlook when you receive the appropriate email.

The following link How To: Trigger an application via email in Outlook should get you started.

---

shutdown usage

F:\test>shutdown /?
Usage: shutdown [/i | /l | /s | /r | /g | /a | /p | /h | /e] [/f]
    [/m \\computer][/t xxx][/d [p|u:]xx:yy [/c "comment"]]

    No args    Display help. This is the same as typing /?.
    /?         Display help. This is the same as not typing any options.
    /i         Display the graphical user interface (GUI).
               This must be the first option.
    /l         Log off. This cannot be used with /m or /d options.
    /s         Shutdown the computer.
    /r         Shutdown and restart the computer.
    /g         Shutdown and restart the computer. After the system is
               rebooted, restart any registered applications.
    /a         Abort a system shutdown.
               This can only be used during the time-out period.
    /p         Turn off the local computer with no time-out or warning.
               Can be used with /d and /f options.
    /h         Hibernate the local computer.
               Can be used with the /f option.
    /e         Document the reason for an unexpected shutdown of a computer.
    /m \\computer Specify the target computer.
    /t xxx     Set the time-out period before shutdown to xxx seconds.
               The valid range is 0-315360000 (10 years), with a default of 30.
               If the timeout period is greater than 0, the /f parameter is
               implied.
    /c "comment" Comment on the reason for the restart or shutdown.
               Maximum of 512 characters allowed.
    /f         Force running applications to close without forewarning users.
               The /f parameter is implied when a value greater than 0 is
               specified for the /t parameter.
    /d [p|u:]xx:yy  Provide the reason for the restart or shutdown.
               p indicates that the restart or shutdown is planned.
               u indicates that the reason is user defined.
               If neither p nor u is specified the restart or shutdown is
               unplanned.
               xx is the major reason number (positive integer less than 256).
               yy is the minor reason number (positive integer less than 65536).

---

Further Reading

• An A-Z Index of the Windows CMD command line - An excellent reference for all things Windows cmd line related.
• shutdown - Shutdown the computer.

Solution 3

If you have actual servers, they likely have a remote management feature that can be connected to even if the machine's main OS is completely frozen. Generally, you use SSH to connect to the controller. The specific command issued depends on the brand of your server.

For a Dell servers, which use DRAC (Dell Remote Access Controller), the command to execute after SSH'ing into the DRAC is:

racadm serveraction hardreset

That performs a hard reset, equivalent to yanking out the power cord and then restarting the machine. If the machine isn't completely hung, powercycle instead of hardreset might do the job. (Source.)

HP servers use iLO, which is also accessible over SSH. The hard reset command is simply:

reset hard

(Source, the official PDF reference to iLO scripting.)

You can wire up the e-mail receiving to the execution of these commands with DavidPostill's answer. There are SSH clients for Windows; I believe PuTTY has a command-line interface.

Solution 4

Use Outlook.

Link at the bottom of the answer has information that helped with this answer. Kudos to GuidingTech.com

Create a .bat file first.

shutdown.exe /r /t 00

Save as Restart.bat

Make sure the computer is logged in with the email you receive the email Notification in.

You can setup different accounts for each computer if you wish and have your email setup to forward emails your receive as notifications.

Example.

[email protected] is the Host PC with a Graphic card that is not Rendering.

Testemail.com is your email you receive notifications on.

When the Email is received have your outlook auto forward the email back to the corresponding email. which in this case is [email protected]

[email protected] received at [email protected] then have that email forward to the all Computers.

The Trick here is to setup a different Notification email to each computer so it knows which to shutdown.

For Setting up a Rule to Forward follow these instructions and enter in a word in the subject based on the pc you want to restart.

Note: This is for setting up your Computer which you receive the emails next we are going to setup the individual pcs.

Step 1. Now open MS Outlook. Under Home tab, go to Rules and expand the >menu. Select Manage Rules and Alerts.

Step 2. On the Rules and Alerts window chick on New Rule while under Email >Rules tab.

Step 3. Start off by selecting your template as Apply rule on messages I >receive. Click on Next.

Step 4. Once again we need to specify which rule you want to use on the Receiving end which will be the Computer the forwarded email shows up at. This will be the rule for restarting the Computer you Specified.

A. With specific words in the subject– Another constraint so that the >receiving email is identified better.

B. With specific words in the body– Sometimes, only the subject may be >ambiguous. So a body makes it more refined.

Select them and hit next.

Now Specify what you want the email to look for to trigger the Restart.

Example if you choose subject.

Say the word you choose is Computer1-RenderIssue

You will type that in the "Specify for words or Phrases to search for in the subhect"

Now that you have the that setup its time to do the final step.

Step 5. Next comes the action to be applied when an email is received and >all the conditions match. Select start application and delete it.

**Note:**If you do not delete it. If the computer is restarted it if outlook opens it will go in a loop.

Step 5a For starting application action, click on the link and select the file you created Restart.bat Click on Next.

Step 5b Click next on *"Are there any Exceptions?"

Step 6 Give the rule a name for future references, turn it on and click on >Finish.

Now whenever One of you computers have an issue when Rendering the Email will be sent to you with the Specified Computer. This will forward the email to that computer and restart the machine.

Hope this helps. Tried to format it the best I could while doing my Job.

Reference for some of the work here

Solution 5

To be honest, if all you want to do is to (hard) bounce a box, the easiest way is to "press the reset switch".

Since you need to do a HARD reset, you could either mess about on the High Voltage end (rather risky), or do it on the motherboard.

Some high-end motherboards have a built-in network card/controller for bouncing the PC. However you can also wire the reset-switch to a relay board and control that using a separate machine.

I would advise connecting a Raspberry Pi via the GPIO to a relay-board. Then attach each relay to the reset switch on the motherboards.

View more solutions

Comments

[Shaun07776]

Do you know what is stalling and how often ? A better plan maybe to make the computer restart daily avoiding the stall all together

[RyanIG]

@Shaun07776 Unfortunately that is the issue, it is random, i have sent log file after log file to each software vendor we are using trying to resolve this issue, as of right now we don't know what is causing them, or when they happen, which is why i am trying to automate everything i can in the meantime until we figure it out.

[exussum]

Why cant the job which detects the stall and sends the email just restart it ? Why does the email need to be the trigger ?

[RyanIG]

@exussum The program we are using is Called Deadline 7.2 it is meant to do this, i have configured it so the software does that, but it doesn't work for some reason, the program allows for python scripting through a web service but i can't get that to do much either.

[stefgosselin]

@RyanIG - Just out of curiosity, does this need some special sort of shutdown procedure, the machine not responsive enough?

[RyanIG]

Thanks for the fast response!When they "Freeze or Stall" sometimes the system is completely hung and it requires a hard reboot, other times you can get away with rebooting it via command line or through team-viewer. this trouble i'm having with figuring out how to link scripts to the email, would you have any threads or resources on how to do this? We are using the gmail but i could run it through outlook if required. cheers!

[RyanIG]

No worries, thanks for you help anyway! ill keep digging.

[DavidPostill]

@RyanIG How To: Trigger an application via email in Outlook should give you a starting point. Answer updated.

[DavidPostill]

Nice. Will this work if the attached device requires a "hard reset"?

[LawrenceC]

It basically remotelu unplugs and plugs in the device for you.

[RyanIG]

I was actually looking into things like this, dailymail.co.uk/sciencetech/article-3305524/… that's a programmable button that would work i reckon, the only difficulty i'm having is linking it to our mail server, these crashes usually happen at night time when I'm asleep, once I'm in the office its not a problem i can usually spot the stall before it even happens, need to automate the whole process in order for it to be efficient

[kasperd]

There also exist similar devices designed for data center usage. Those are designed to be mounted in a rack and have a lot more output ports. Those I have seen were controlled over RS-232. That way you could attach them to a unit with lots of RS-232 ports such that this one unit would give you remove console and remote power cycling of all the individual servers in an entire rack.

[RyanIG]

Ill be taking a look into this as we use 3 dell powerblade servers, quick question though, does it have to be on the Domain controller or if we had a license server could it be setup on that? the machines are constantly connected to both. Thanks for your answer!

[TOOGAM]

Some UPSs can provide this functionality too. You can connect to the UPS (via HTTP, or Telnet, or software released by the manufacturer) to connect via a cable (Ethernet, or a "serial cable"). Specific details vary based on individual solutions. Some UPSs only support this when using an add-on card (inserted in the UPS), purchased separately. I imagine that they are more expensive than the device LawrenceC is showing, because the UPSs also provide the feature of being battery backed up. However, the higher cost may be acceptable due to the inclusion of the extra useful feature.

[Ben N]

The DRAC should be accessible from anywhere on the same network. Therefore, any computer that can receive the e-mail and reach the DRAC should be able to handle the whole system.

[RyanIG]

Awesome, ill look into this and let you know how i get on, cheers!

[No Username]

There is software that is part of the Red Hat Cluster Suite called fencing agents, which provides an abstract CLI interface to restarting these devices, including KVM as a "power fencing" device. It has been useful for me.

[NetworkKingPin]

Alright David I got what you meant its taken care of now.

[DavidPostill]

That's much better ;)

[clhy]

I know where you are going with Virtual Machine concept but your statement about comparable performance doesn't make much sense.

[null]

@RyanIG Maybe you can set one of these devices up so that they perform the hard reset automatically every 10 minutes or so unless they get a command via ethernet. Then you can hook the computers up to their own power supply and let them send the don't reboot me command every 3 minutes or so. If the computer hangs, it will also stop sending the command and get a hard reset. It's basically what's called a "watch dog (timer)" in embedded systems and it would bypass the whole email communication.

[TMN]

How so? You're basically just wrapping a physical server with a VM and running it on the same hardware. It's not like you're trying to run multiple VMs. According to this paper from 2009 "if you're using this year's VMware or Xen technologies, Intel Nehalems and Shanghai Opterons and the new i/o adapters -- virtualization has most performance issues pretty much beat." I don't imagine virtualization has backslid much since then.

[Ben N]

Fun reading, but please include the important parts in your post - answers should be as self-contained as possible.

[mckenzm]

Indeed, it is common for "watchdog ping" to be a feature of these devices, with the caveat that consecutive pings must fail over minutes or loss of the network results in mass reboots.

[TMN]

Forgot to mention: this is a rendering farm, so most of the work is being done by the graphics cards, and their performance won't be degraded by running the host in a VM.

[WeSee]

VM access to GPU's can be tricky (very VM specific, very card specific, if supported at all), so verify that it works for you.