Can and should UDF be used as a hard drive format?

Question

Several time recently I've seen UDF suggested as the solution to a cross platform format for a drive used on Linux, Mac OS X and Windows XP and above.

I've searched here and not found the same suggestion (most are suggesting ntfs-3g which seems to cost money and isn't preinstalled on a Mac).

So my question is: how is this done right, and has anyone done this? Have you then filled up the drive and deleted some files to make space finding that everything works like a real r/w format even though it seems to have been primarily a write once format?

Call me crazy but I'd really like it if the UDF system would also automount and be writable by the logged in user. What I've tried so far (udftools formatting as mentioned by kicsyromy) doesn't address this wish.

Solution 1

No.

We're in 2015 at the time of this reply. I am using OSX Yosemite, Ubuntu 14.10, and the Windows 10 technical preview for enterprises on a Mactel machine (Macmini 7,1).

I tried both UDF and exFat. I use Ubuntu for development and do need Unix-style permissions.

All former guides do not apply anymore: UDF drivers have evolved and all operating systems will accept a UDF partition, with more problems and instabilities than I can name.

• UDF drive formatted on Mac OS: can't be mounted on Windows 10.
• UDF drive formatted on Linux: can't be mounted on Windows 10.
• UDF drive formatted on Windows 10: mounts read/write on Linux, read-only on OSX.

However, Windows doesn't allow you to specify a block size when formatting a UDF volume, and as a result, your logical block size might differ from the physical block size for the partition.

I am unclear whether this has to do with the difficulties I had mounting it read/write on OSX, but after deleting a certain number of files using Linux, I was never able to mount the drive again on OSX.

The system goes into kernel panic and crashes disgracefully.

This, and a variety of answers on the subject, indicate inconsistent support for this format at this point.

It would seem there are ways I can use a NTFS volume to achieve a balance between the features of a modern file system, Unix-style permissions - I might be able to set them - and read/write mount on all operating systems.

Solution 2

Someone did some research into how to format a flash drive with udf so it can be used on as many operating systems as possible. Here are his findings (used to be there, now offline):

• Windows 7 have full support up to UDF v2.6, but the UDF block size must match the block size of the underlying device (which for USB-sticks and most disks is 512 bytes; "advanced format" disks are 4096 bytes). Apparently the disk must be partitioned.

• Linux 2.6.30 and up supports UDF fully at least up to version 2.5.

• Mac OS X 10.5 supports UDF fully up to UDF 2.01, but only when used on a full disk, so not partitioned.

As explained above, for USB harddisks, Windows requires the disk to be partitioned. On the other side, UDF only works in OS X when it is used on a full disk (unpartitioned). Rather surprisingly, there is a solution which works for both: having the disk partitioned and unpartitioned at the same time.

DOS partition tables are stored in bytes 446-510 of the master boot record. This master boot record is stored in the first sector on disk, sector 0. Typically, the first partition specified will start some kilobytes further. However, it seems possible to construct a partition table whose first partition starts at sector 0, so the result is a partition which contains the partition table itself. Partition editor programs seem to refuse to create such a table, but at least recent Linux and Windows kernels don’t seem to bother.

The nice thing is that UDF does not (I suppose deliberately) use the first few kilobytes of the partition or disk it is placed on, so this place can really be used to store a legacy partition table, referring to a partition that spans the whole disk. Some testing shows that this really works on Linux, Windows and Mac OS X:

• Mounts automatically read-write in Linux 2.6.30+, Mac OS X 10.5+, Windows Vista+
• Can be used read-only in Windows XP, and be used after a command line mount in Linux 2.6.0+
• Supports large files, UNIX permissions, Unicode filenames, symlinks, hardlinks, etc.

Script to format the disk properly: Perl script or Bash script

Solution 3

I just tested this out in a VM. It seems that you need to (re)create your partition in Windows assign it a drive letter but don't format it to any filesystem. After that boot into Ubuntu and just follow the directions and it should work for read/write.

Remember to backup all your data!

First off install UDF tools:

sudo apt-get install udftools

Replace the first block with nothing on the partition you wish to format to UDF^:

sudo dd if=/dev/zero of=/dev/sdxN bs=512 count=1

And finally format to UDF^:

sudo mkudffs --media-type=hd --blocksize=512 /dev/sdxN

^where by:

• x is a placeholder for the letter curently assigned to your hardisk

• N is a placeholder for the partition number

Best of luck and let me know if it worked out for you.

Comments

[Eliah Kagan]

One clarification: ntfs-3g is free. Its source code is gratis (i.e., available at no cost). It is also free as in freedom. ntfs-3g is the NTFS driver in Ubuntu! It's slightly technical to manually build/install it on OS X, and Tuxera (its developer) offers a proprietary payware version that is essentially the free ntfs-3g driver built and packaged for easy installation and use on OS X. Without an add-on driver, OS X will only read (not write) NTFS volumes, so you're right to consider another filesystem.

[user29020]

@EliahKagan So if ntfs-3g is "free", then why hasn't Apple included it to allow r/w support for NTFS?

[dlamblin]

@user29020 Support costs (what if it's a bit buggy)? Legal restrictions? Free under GNU GPL License means that it can only be added to existing source / binaries which are in turn also available under the GPL. (LGPL would allow it to be used as a library by other non-GPL code). Given that apple's code is largely proprietary and otherwise under the APSL which is not a GPL compatible license, that is a restriction they must abide by. gnu.org/philosophy/apsl.html

[dlamblin]

Thanks this answers the how; I would still be interested in finding out if it works well enough with the above scenario.

[Nemo]

This worked perfectly for me. The only issue is that I have to manually mount the partition in Windows using Disk Manager. Other than that, I can confirm that it works with multiple partitions on the hard-disk as well (I'm using 1udf+2ext4)

[DanMan]

The key here seems to be to actually create a partition and then create a UDF file system in there. If you don't create any partition, it'll work in Linux, but it won't in Windows (7). That's my experience.

[dlamblin]

@DanMan Dan seems to have found someone who solved that issue and provides a script for formatting a whole disk with UDF for Linux, Mac and Windows 7, as well as read-only in Win XP.

[Tim Banchi]

@kicsyromy: formatting a partition (sdxN) to UDF did not work, executing sudo mkudffs --media-type=hd --blocksize=512 /dev/sdd1 gives the following error message: trying to change type of multiple extents I did however succeed in formatting UDF using the whole disk (sdd) as described here: superuser.com/questions/39942/using-udf-on-a-usb-flash-drive

[Janus Troelsen]

Which system is crashing? You say kernel panic, but in the previous sentence you make it sound like it is OSX that is having problems.

[Mauro Colella]

OSX is based on BSD. So yes, it is OSX Yosemite (10.10.2) which experiences a kernel panic. I ended up using NTFS, and NTFS 3d for OSX.

[JDługosz]

re Windows doesn't allow you to specify a block size when formatting a UDF volume: UDF doesn't have an adjustable block size. It uses the device sector size, period. It doesn't use any kind of clustering either. What setting are you referring to?

[Mauro Colella]

Apparently not. See: manpages.ubuntu.com/manpages/lucid/man1/mkudffs.1.html and tanguy.ortolo.eu/blog/article93/usb-udf

[JDługosz]

@Argo the second reference states, "The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size." So it does let you specify the block size but it must be specified as being the required value.

[Mauro Colella]

@JDługosz yes, with mkudffs on Linux or BSD. When formatting using Windows, you cannot specify a block size.

[bmaupin]

You mention exFAT but don't provide any details. I know exFAT doesn't support POSIX file permissions; is that why you didn't use it? I've been trying exFAT on a flash drive with no issues so far, but I don't need POSIX permissions.

[DrYak]

The Windows 10 not mounted by OS X / OS X not mounted is due to difference in how each OS handle partition tables. Windows 10 : Needs one. Mac OS X: Needs UDF to be on the bare drive, as a whole drive partition.

[velop]

There is a formating tool which takes care of the incosistencies: github.com/JElchison/format-udf

[Mauro Colella]

Seems great. But in an ideal world it's the standard and implementers that should take care of inconsistencies.

[Hi-Angel]

It's a bit confusing, perhaps you meant UDF drivers have degraded? Also, did you report bugs to Mac and Microsoft?

[Arc676]

@DrYak Does this mean that if I add a UDF partition on my internal HDD, macOS won't be able to mount it? I'm researching filesystems and, if possible, planning on putting all my data on a UDF partition so I can access it from different operating systems.

[DrYak]

@Arc676 : I haven't experimented the detailed behaviour (I don't use mac OS X that often). But basically : - if you want multiple different partitions on this drive, one of which is UDF : that seems to be difficult to pull of with mac OS X - if you want just the UDF partition to be visible to both mac OS X and anything else you dual boot : it's possible, by using the trick mentionned in the other answer

[DrYak]

So basically : align the UDF partition at sector 0 (so it begins at the drive start, and mac OS X sees it as a full-drive raw patition) and hide a partition table in the boot-zone at the beginning of the partition (so Windows sees one and is happy too). But there's a high chance (untested) that Mac OS X will only be able to see and mount the first UDF partition and nothing else on this drive. (Thus use the other partitions for Linux or Windows -specific stuff)

[phuclv]

it's possible to store POSIX permissions on NTFS and FAT

[Lavakesh]

+1 The prelimary text is important: Prepare new volume in Windows, then only format in Linux. So that UDF partition can be recognized by both Windows and Linux. Tested with primary partition on local hard drive.

[phuclv]

for UDF you need to specify the correct options so that the partition can be mounted in all OSes: en.wikipedia.org/wiki/Universal_Disk_Format#Compatibility