VMware VM with more than 2TB of disk space

Question

I'm a VCP 4 so I'm aware of the configuration maximums for LUNs/RDMs/VMDKs in vSphere 4, but say I want to create a VM that has a logical drive bigger than 2TBs. The only ways I can think of doing this are:

1. Creating the VM with multiple VMDKs each less than 2TB on different LUNs and spanning or software RAIDing them in the guest operating system.

2. Creating a VMFS extent across multiple LUNs and creating multiple VMDKs and spanning or software RAIDing them in the guest operating system (just a slight variation of #1).

3. Use multiple 2TB or smaller RDMs and spanning or software RAIDing them in the guest operating system.

4. Mount an iSCSI LUN larger than 2TB from the guest operating system.

All of these methods rely on spanning/software RAIDing from the guest operating system. Is this pretty much it or are there other possibilities?

Update TomTom pointed out a 4th option.

Update 2 Tom's hardware has an interesting article about a petabyte sized storage device. They have a screenshot of it mounted via NFS in vSphere 4.1. As to JakeRobinson's suggestion with NFS, the sky is the limit with these types of datastores!

Solution 1

That's pretty much it in 4.x.

vSphere 5.0 allows for pass-thru RDMs to be larger (about 60 TB), but still has the 2 TB limit in place for VMDKs and non-pass-thru RDMs. (reference). Pass-thru (physical) RDMs and non-pass-thru (virtual) RDMs are distinguished by what degree the SCSI commands are virtualized by vSphere. Pass-thru RDMs are sent all SCSI commands with the exception of the REPORT_LUN directly whereas the the non-pass-thru RDMs have all their hardware characteristics hidden and VMKernel only passes the READ and WRITE commands to the device as it is presented to the guest operating system as any other VMDK on a VMFS volume (see KB2009226).

If you can upgrade to 5.0 and use RDMs, then great; otherwise, you're stuck gluing VMDKs or <2TB RDMs together in the guest OS with LVM or a software RAID.

Solution 2

Don't forget network attached storage type protocols (eg. NFS, CIFS). That is, if your SAN/NAS supports serves them up. Obviously a VM providing either of those would kinda defeat the purpose.

Linux guests can use multiple VMDKs for an LVM, which is not exactly spanning and not exactly RAID.

Comments

[ewwhite]

What's your guest operating system? What is your underlying storage solution?

[Andy Arismendi]

@ewwhite The question is more hypothetical asking to identify all possible options to do this using any storage type and guest OS.

[TomTom]

You forgot 4: Use ISCSI and dont have VmWare even know there is a larger LUN attached to a VM.

[Andy Arismendi]

@TomTom That's an interesting idea I hadn't thought of. Thanks!

[Andy Arismendi]

Thanks Jake. NFS is a another option. This would allow a datastore to be bigger than 2TB like VMFS extents. Correct me if I'm wrong but LVMs are similar in the respect that multiple "physical" discs can be presented as a single contiguous logical volume. Right? BTW nice avatar :-)

[JakeRobinson]

Correct. A single logical volume can span multiple physical volumes, allowing you to defeat the 2TB-512byte limit. You could also do the same with a Windows guest using the OS's software RAID or JBOD. I can't speak to the performance on Windows though. Thanks for the avatar compliment. ;)

[Andy Arismendi]

This is already listed as an option in the question. The VM is still limited to 2TB VMDKs though even though the datastore is larger than 2TB using extents.

[sjw]

your correction is wrong, due to incorrect wording. you said "guest operating system". since you say you're a VCP, you should be well aware that extents are not at the "guest" level. you do not create "an extent" across multiple LUNs if you want large vmdks. you create "a datastore", and that spans multiple extents, each on multiple LUNs. you can then create large vmdk files on this large datastore.

[Andy Arismendi]

#2 says that. The extent would make the datastore larger than 2TB. However multiple 2TB VMDKs would still have to be created on that datastore and software RAID or LVMs would have to be implemented on the guest OS for it to have a logical volume greater than 2TB.