xenserver - xe-toolstack restart - i lost connection from xencenter

Question

After Xencenter-toolstack restart i lost connection from xencenter

I try to restart using ssh - putty following message is showing and i lost the browser access also ............

All xe command is not working and postponed on ssh ....

@xs01p04 log]# xe-toolstack-restart Stopping xapi: [ OK ]
Stopping the v6 licensing daemon: [ OK ]
Stopping the memory ballooning daemon: [ OK ]
Stopping perfmon: [FAILED]
Stopping the xenopsd daemon: [ OK ]
Stopping the XCP RRDD daemon: [ OK ]
Stopping the XCP networking daemon: [ OK ]
Stopping the fork/exec daemon: [ OK ]
Starting the fork/exec daemon: [ OK ]
Starting the XCP networking daemon: . [ OK ]
Starting the XCP RRDD daemon: [ OK ]
Starting the xenopsd daemon: [ OK ]
Starting perfmon: [ OK ]
Starting the memory ballooning daemon: [ OK ]
Starting the v6 licensing daemon: [ OK ]
Starting xapi: OK [ OK ]

Thank Advance ...

Update #1

@yagmoth555

1) i try to migrate one vm to another storage location that time some unexpected error occurred and yello state is appear Then i create new vm from snapshot

Dead vm still there

After one week 2) yesterday i try to reboot a vm and its again yellow state ( its not previous one)

Then i try to restart xentoolstack from xencenter after few minute i lost xencenter connection I cannot connect through xencenter

But https:ip is appearing

Then i login through ssh and restart toollstack through command line After that i lost http acess also

Nom XE command is hanged up but Vm are still working

Update #2 @yagmoth555 Thanks advance

tail -n 15 /var/log/messages

Oct 16 20:08:30 xs01p04 ovsdb-server: 01993|reconnect|INFO|ssl:10.101.2.52:6632: waiting 8 seconds before reconnect

Oct 16 20:08:33 xs01p04 fe: 26045 (/usr/bin/ovs-appctl bond/show bond2) exitted with code 2

Oct 16 20:08:33 xs01p04 xcp-networkd: [error|xs01p04|1|monitor_thread|network_utils] Call '/usr/bin/ovs-appctl bond/show bond2' exited with code 2 [stdout = ''; stderr = 'no such bond ovs-appctl: ovs-vswitchd: server returned reply code 501 ']

Oct 16 20:08:38 xs01p04 ovsdb-server: 01994|reconnect|INFO|ssl:10.101.2.52:6632: connecting...

Oct 16 20:08:38 xs01p04 ovsdb-server: 01995|reconnect|WARN|ssl:10.101.2.52:6632: connection attempt failed (No route to host)

Oct 16 20:08:38 xs01p04 ovsdb-server: 01996|reconnect|INFO|ssl:10.101.2.52:6632: waiting 8 seconds before reconnect

Oct 16 20:08:38 xs01p04 fe: 26054 (/usr/bin/ovs-appctl bond/show bond2) exitted with code 2 Oct 16 20:08:38 xs01p04 xcp-networkd: [error|xs01p04|1|monitor_thread|network_utils] Call '/usr/bin/ovs-appctl bond/show bond2' exited with code 2 [stdout = ''; stderr = 'no such bond ovs-appctl: ovs-vswitchd: server returned reply code 501 ']

Oct 16 20:08:43 xs01p04 fe: 26063 (/usr/bin/ovs-appctl bond/show bond2) exitted with code 2

Oct 16 20:08:43 xs01p04 xcp-networkd: [error|xs01p04|1|monitor_thread|network_utils] Call '/usr/bin/ovs-appctl bond/show bond2' exited with code 2 [stdout = ''; stderr = 'no such bond ovs-appctl: ovs-vswitchd: server returned reply code 501 ']

Oct 16 20:08:46 xs01p04 ovsdb-server: 01997|reconnect|INFO|ssl:10.101.2.52:6632: connecting...

Oct 16 20:08:46 xs01p04 ovsdb-server: 01998|reconnect|WARN|ssl:10.101.2.52:6632: connection attempt failed (No route to host)

Oct 16 20:08:46 xs01p04 ovsdb-server: 01999|reconnect|INFO|ssl:10.101.2.52:6632: waiting 8 seconds before reconnect

Oct 16 20:08:48 xs01p04 fe: 26081 (/usr/bin/ovs-appctl bond/show bond2) exitted with code 2

Oct 16 20:08:48 xs01p04 xcp-networkd: [error|xs01p04|1|monitor_thread|network_utils] Call '/usr/bin/ovs-appctl bond/show bond2' exited with code 2 [stdout = ''; stderr = 'no such bond ovs-appctl: ovs-vswitchd: server returned reply code 501 ']

Update #3 @yagmoth555 Thanks advance
xe-toolstack-restart Stopping xapi: [ OK ]

Stopping the v6 licensing daemon: [ OK ]

Stopping the memory ballooning daemon: [ OK ]

Stopping perfmon: [FAILED]

Stopping the xenopsd daemon: [ OK ]

Stopping the XCP RRDD daemon: [ OK ]

Stopping the XCP networking daemon: [ OK ]

Stopping the fork/exec daemon: [ OK ]

Starting the fork/exec daemon: [ OK ]

Starting the XCP networking daemon: . [ OK ]

Starting the XCP RRDD daemon: [ OK ]

Starting the xenopsd daemon: [ OK ]

Starting perfmon: [ OK ]

Starting the memory ballooning daemon: [ OK ]

Starting the v6 licensing daemon: [ OK ]

Starting xapi: OK [ OK ]

service xapissl restart

which: no stunnel4 in (/sbin:/usr/sbin:/bin:/usr/bin) Stopping xapi SSL: Starting xapi SSL:

Solution

xe toolstack-restart will always cut your xencenter connection. (but the vm continue to run correctly on the host)

Please log in normal console (iDRAC, iLO, etc..), as that mean your XAPI can't restart and your server state is unknown.

I would check with dmesg or in /var/log for xensource.log for any errors.

Comments

[yagmoth555]

Did you closed both VM, and simply rebooted the server first ?

[Jobish Jose]

@ yagmoth555 , I rebooted sever still XE-command sill frorzen and only one vm is automatically up and ............................. I can see only one vm by domain_lis command ....................... xencenter is not connecting

[yagmoth555]

start by doing a df -m via the console/ssh on the host, is the disk is full ?

[Jobish Jose]

disk is not full only 79% ...................

[yagmoth555]

Your state.db is probably corrupt then. cd /var/xapi, check the file timestamp, if old, it's corrupt. if so, mv state.db state.db_bak, after service xapi restart

[Jobish Jose]

I did but its same ... after xapapi restart, state.db not automatically generating .... ...

[yagmoth555]

just seen your new log, never seen that error. I would do xe-switch-network-backend bridge to disable openvswitch, but it's a guess at this point.

[yagmoth555]

after all that, if your data is on a different partition I would re-install xen over your install without touching the vm storage location, and re-join the member to the pool. be sure to have backup of the vm

[Jobish Jose]

Brother , Thanks for your replay .. ,,,,, I can acess server through ssh and i can access all vm ........ can you explain little bit more i am not familiar in xenserver ....

[yagmoth555]

@JobishJose The toolstack-restart cut the comminucation with the console (but the vm continue to work), but usually you can reconnect back some minute later with xencenter when the service restart. I would check dmesg to check if a error occurred somewhere. (or xensource.log in /var/log)