EC2 instance initializing with only 1/2 check thus not able to login into my Linux server

Question

Hello friends,

Our EC2 instance is initializing with only " 1/2 checks passed " after I detached and reattached my volumes (I did it for some specific debugging purpose). Thus i am not able to login into my Linux server now (using putty .. it shows network error) where I have some complex scripts which i need to retrieve in case we decide to terminate the instance.

I don't know much abt EC2 programming as we only use it for SAP HANA cloud access.I have generated a SYSTEM LOG of the instance which shown below:

souvenir Xen Minimal OS!
  start_info: 0x9248000(VA)
    nr_pages: 0x1117000
  shared_inf: 0x7e68b000(MA)
     pt_base: 0x924b000(VA)
nr_pt_frames: 0x4d
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x6000d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 929b
    max_pfn: 1117000
Mapping memory range 0x9400000 - 0x1117000000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 11b4d000(11b4d000)-1117000000(1117000000)
MM: done
Demand map pfns at 1117001000-3117001000.
Heap resides at 3117002000-5117002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x1117001000.
Initialising scheduler
Thread "Idle": pointer: 0x3117002010, stack: 0x1113680000
Initialising xenbus
Thread "xenstore": pointer: 0x31170027c0, stack: 0x1113690000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x3117002f70, stack: 0x11136a0000
"main" "root=/dev/sda1" "ro" "4" 
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/390/2049
Failed to read /local/domain/0/backend/vbd/390/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2049/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2144 is hd1
******************* BLKFRONT for device/vbd/2144 **********


backend at /local/domain/0/backend/vbd/390/2144
Failed to read /local/domain/0/backend/vbd/390/2144/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2144/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2160 is hd2
******************* BLKFRONT for device/vbd/2160 **********


backend at /local/domain/0/backend/vbd/390/2160
Failed to read /local/domain/0/backend/vbd/390/2160/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2160/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2176 is hd3
******************* BLKFRONT for device/vbd/2176 **********


backend at /local/domain/0/backend/vbd/390/2176
Failed to read /local/domain/0/backend/vbd/390/2176/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2176/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2192 is hd4
******************* BLKFRONT for device/vbd/2192 **********


backend at /local/domain/0/backend/vbd/390/2192
Failed to read /local/domain/0/backend/vbd/390/2192/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2192/feature-flush-cache.
20971520 sectors of 512 bytes
**************************
vbd 2208 is hd5
******************* BLKFRONT for device/vbd/2208 **********


backend at /local/domain/0/backend/vbd/390/2208
Failed to read /local/domain/0/backend/vbd/390/2208/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2208/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2224 is hd6
******************* BLKFRONT for device/vbd/2224 **********


backend at /local/domain/0/backend/vbd/390/2224
Failed to read /local/domain/0/backend/vbd/390/2224/feature-barrier.
Failed to read /local/domain/0/backend/vbd/390/2224/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
[H[J

    GNU GRUB  version 0.97  (71680000K lower / 0K upper memory)



       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]




grubdom>                                                                       [9;10HXen Minimal OS!
  start_info: 0x9248000(VA)
    nr_pages: 0x1117000
  shared_inf: 0x7e692000(MA)
     pt_base: 0x924b000(VA)
nr_pt_frames: 0x4d
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x6000d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 929b
    max_pfn: 1117000
Mapping memory range 0x9400000 - 0x1117000000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 11b4d000(11b4d000)-1117000000(1117000000)
MM: done
Demand map pfns at 1117001000-3117001000.
Heap resides at 3117002000-5117002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x1117001000.
Initialising scheduler
Thread "Idle": pointer: 0x3117002010, stack: 0x1113680000
Initialising xenbus
Thread "xenstore": pointer: 0x31170027c0, stack: 0x1113690000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x3117002f70, stack: 0x11136a0000
"main" "root=/dev/sda1" "ro" "4" 
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/391/2049
Failed to read /local/domain/0/backend/vbd/391/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2049/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2144 is hd1
******************* BLKFRONT for device/vbd/2144 **********


backend at /local/domain/0/backend/vbd/391/2144
Failed to read /local/domain/0/backend/vbd/391/2144/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2144/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2160 is hd2
******************* BLKFRONT for device/vbd/2160 **********


backend at /local/domain/0/backend/vbd/391/2160
Failed to read /local/domain/0/backend/vbd/391/2160/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2160/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2176 is hd3
******************* BLKFRONT for device/vbd/2176 **********


backend at /local/domain/0/backend/vbd/391/2176
Failed to read /local/domain/0/backend/vbd/391/2176/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2176/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2192 is hd4
******************* BLKFRONT for device/vbd/2192 **********


backend at /local/domain/0/backend/vbd/391/2192
Failed to read /local/domain/0/backend/vbd/391/2192/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2192/feature-flush-cache.
20971520 sectors of 512 bytes
**************************
vbd 2208 is hd5
******************* BLKFRONT for device/vbd/2208 **********


backend at /local/domain/0/backend/vbd/391/2208
Failed to read /local/domain/0/backend/vbd/391/2208/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2208/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
vbd 2224 is hd6
******************* BLKFRONT for device/vbd/2224 **********


backend at /local/domain/0/backend/vbd/391/2224
Failed to read /local/domain/0/backend/vbd/391/2224/feature-barrier.
Failed to read /local/domain/0/backend/vbd/391/2224/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
[H[J

    GNU GRUB  version 0.97  (71680000K lower / 0K upper memory)



       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]




grubdom>                                                                       [9;10H

Hope u experts can help me with this issue.

thanx

Solution

I've had this happen as well -- the second health check is network connectivity, and until that gets sorted out SSH is not going to work (obviously). I would suggest

1. A full reboot of the instance from within the AWS console or AWS CLI.
2. You may also want to detach the elastic IP of the instance, wait a few seconds and then re-attach it.

You could also spin up a new EC2 instance in the same AZ, detach the volume from your unhealthy instance, and mount it as a secondary/tertiary volume in the new instance -- either to examine it, or to get out the scripts/files you need.

Sorry I can't give more concrete advice beyond that, but I am wondering if your /local/ partition has become unreadable?

Comments

[David Levesque]

When you look at the "Volumes" section of the EC2 Console, do you see the Volume Status as Okay for all volumes? Also check the IO Status under the Status Checks tab. Sometimes EC2 will disable IO following a crash or some other event.

[Renato Gama]

I had the same problem, in my case it was related to the Kernel version. I had to change for it to work.

[user3762812]

Thanx for d advice Neal but I was able to resolve this issue by attaching the correct volume as Root (which was d mistake i made earlier as i am ignorant with AWS stuff). but now even though I am able to Login into my SUSE linux server but I cant see any data/directory from other volumes other than the root.Some icons are showing up as inactive-Red shortcuts to folders on other volumes where my HDB and HDB00 exit. hope u can help me with this issue