How to build an sd card image, without an sd card?

Question

We have a scripted process that builds up an embedded install of Debian Jesse on an sd card. The relevant parts of the script look like this:

export DISK=/dev/sdb
umount ${DISK}1   # make sure no partitions mounted at the moment
umount ${DISK}2   # ditto
dd if=/dev/zero of=${DISK} bs=1M count=16   # zero partition table zone, overkill
sfdisk --in-order --Linux --unit M ${DISK} <<-__EOF__    # make partitions
    1,48,0xE,*
    ,,,-
__EOF__

mkfs.vfat -F 16 ${DISK}1 -n boot    # install file systems
mkfs.ext4 ${DISK}2 -L rootfs

After that, the auto mounter seems to kick in and get the sd card remounted, so we can do things like:

cp -v ${DIR}/u-boot/spl/u-boot-spl.bin /media/$username/boot/BOOT.BIN
cp -v ${DIR}/u-boot/u-boot.img /media/$username/boot/
cp -v ${DIR}/u-boot/uEnv.txt /media/$username/boot/
rsync -axHAX --progress ${DIR}/jessieRoot/ /media/$username/rootfs/

After one of us does that, then we can use dd to copy the contents of the card and share it with each other, making more sd cards using dd.

THE PROBLEM with this is twofold: 1) It's very Ubuntu/machine specific right now (assumes card is at sdb, etc 2) It needs an actual card, so doesn't lend itself to a build machine.

Is there a way to do the above without a card?

I tried using dd to just make an 8G file and then ran sfdisk on that (everything's a file, right?) and that part worked. But it's not clear how I'd run the mkfs parts to work, they seem to want to work on block device files, not sub regions of a single file that has a partition table embedded in it. And then I have the problem of mounting it. I assume I use some incantation of mount -o loop, but again, not sure how to do that on the sub region of the virtual image file, I've always just down that with .iso files.

(Feel free to be pedantic, I am not an expert (obviously) with this kind of stuff. I get some of it, and other parts seem a bit magic...)

Solution 1

I think this page has everything you need.

• instead of sdb make use of loopback device

• instead of actual card make use of virtual filesystem

• you are in the right track making use of dd to create a file for the virtual filesystem.

• you are in the right track using loopback device. The trick is mounting the loopback device in the offsets where the partitions are.

This is the article.

A virtual filesystem is filesystem that exists in a file, which in turn exists on a physical disk. There's a lot of great things you can do with virtual file systems; the reason I was messing with them was to configure a virtual machine on a linux host. Other uses include encrypting filesystems without encrypting entire disks; Mac OS X's File Vault encrypts users home directories this way. Maybe you went ahead and made yourself one giant partition and then realized for one reason or another that you want multiple partitions! Virtual filesystems can help (to some extent) with that as well.

So how do you make a virtual file system? Easy. The first thing you need to do is make a file for the filesystem to live in. This is where 'dd' starts to come in. Consider, for example, the following command:

dd if=/dev/zero of=~/myFileSystem.img bs=1024 count=1048576

This command will read 1,048,576 blocks from /dev/zero and write them to ~/myFileSystem.img; each block is 1024 bytes, resulting in a 1 gigabyte file containing all zeroes. The actual values that you use for the blocksize (bs) and count aren't really important, the key is to get your math right: bs * count = imageSize.

So now you have your file; great. It's time to make a file system! this part is even easier... we'll create an EXT2 filesystem, using the following command:

mke2fs myFileSystem.img

You may notice a warning prompt, saying that myFileSystem.img is not a block device, would you like to proceed? We'll get to that in just a second, for now, go ahead and say yes. Everything should go smooth, it'll look just as if you'd created a filesystem on an actual disk drive. You now have a virtual file system! The only thing left to do is mount your filesystem so you can access it...

mkdir /mnt/virtual

mount -o loop ~/myFileSystem.img /mnt/virtual

Now any file you put into /mnt/virtual is actually being put directly into myFileSystem.img! Wasn't that easy?

Fantastic. Now that you know how to make a virtual filesytsem, why not make a while virtual disk image? What's the difference you ask? A disk image is going to have a partition table that defines some number of partitions, and each partition contains its own filesystem; so a virtual filesystem is essentially a "virtual partition" of a virtual disk image; the virtual disk image contains multiple virtual filesystems, and a virtual partition table that describes where the bounds of each partition are.

Creating a virtual disk image starts out the same; the first thing you need is a big empty file, just you created above. This time, though, instead of making a file system, we'll want to partition the file using fdisk. To make things a little nicer though, we're going to throw loopback devices into the mix. You should make sure you have loopback device support enabled in your kernel (most distributions do by default; but if you're a kernel compiling linux junky, you might wanna check ). So we'll create a big file, and attach it to a loopback device, as follows:

dd if=/dev/zero of=~/myDisk.img bs=1024 count=1048576

losetup /dev/loop0 ~/myDisk.img

By attaching the disk image to the loopback device, we can use /dev/loop0 the same way we would use ~/myDisk.img; the main difference is that /dev/loop0 is a what's known as a "block device". You'd have to ask someone with more experience than I've got what precisely this gets you, but what I do know is that the filesystem utilities work better with block devices than they do with the flat files. Plus, it's fun.

So good, we've got a big empty file attached to a loopback device (/dev/loop0)... now it's time to create partitions in the disk image. To do this, we'll run fdisk on our loopback device:

fdisk /dev/loop0

Lets create three partitions... if you're following this, you should already be familiar with fdisk, so go ahead and create the following: Device Boot Start End Blocks Id System

/dev/loop0p1 1 17 136521 83 Linux

/dev/loop0p2 18 80 506047+ 82 Linux swap

/dev/loop0p3 81 130 401625 83 Linux

Once you've made your partitions, write your changes and quit fdisk. What we'll need to do next is create filesystems on each partition. Remember how easy that was back with Virtual Filesystems? Not quite so much anymore...

Not to panic though... the trouble is that "mkfs" can't "reach into" our virtual disk image and make a filesystem just on individual partition. Infact, if you try, you'll probably wind up wiping our your virtual disk image and having to rerun fdisk . So what to do... what to do?? Loopback devices to the rescue. What we'll do is attach a loopback device to the myDisk.img file at the specific offsets where each partition begins.

It's helpful then to look at the partitions in terms of blocks. Execute the following command:

fdisk -ul /dev/loop0

should look (hopefully exactly) like this:

Disk /dev/loop0: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors

Units = sectors of 1 * 512 = 512 bytes

  Device Boot      Start         End      Blocks   Id  System

/dev/loop0p1 63 273104 136521 83 Linux

/dev/loop0p2 273105 1285199 506047+ 82 Linux swap

/dev/loop0p3 1285200 2088449 401625 83 Linux

These numbers are important for the math... we'll use the losetup command like we did before, only this time we'll reach in specifically to the start of each of the three partitions. losetup takes offsets as the number of bytes to skip at the beginning of the file. The output from fdisk -ul /dev/loop0 shows us that the first partition starts at block 63, and that each block is 512 bytes. So partition 1 starts at byte 32,256

losetup -o 32256 /dev/loop1 /dev/loop0

That command reaches 32,256 bytes into /dev/loop0 and mounts it at /dev/loop1. Remember that since /dev/loop0 is attached the myDisk.img file, this is the same as reaching 32,256 bytes into that file... follow? Ok, good. Same logic for partitions 2 and 3:

losetup -o 139829760 /dev/loop2 /dev/loop0

losetup -o 658022400 /dev/loop3 /dev/loop0

So now we have four loopback devices set up; /dev/loop0 is attached to the myDisk.img file. /dev/loop1 is the first partition of the virtual disk represented by /dev/loop0; /dev/loop2 is the 2_nd, and /dev/loop3 is the 3_rd.

Now it's finally time to make those file systems! This is now just as easy as making a regular filesystem, since that's all we're doing. Remember, mkfs doesn't know the device isn't a physical device! We'll make three kinds of file systems, an ext2 file system for partition 1, a swap filesystem for partition 2, and an XFS fileystem for partition 3:

mkfs /dev/loop1

mkswap /dev/loop2

mkfs.xfs /dev/loop3

Since loop1, loop2, and loop3 are tied directly to loop0, and loop0 is ~/myDisk.img, everything that we just did to loop1, loop2, and loop3 affected myDisk.img directly! We can now mount /dev/loop3, for instance, on /mnt/virtual as an XFS file system, and use it as a regular file system!

So I hope you found that helpful... you can do some pretty neat things with virtual file systems and virtual disk images; and loopback devices make a world of difference for making things go smooth.

Solution 2

The accepted answer to this problem is factually correct, but—as I write this five years later—there's a simpler approach that ought to work for most people. Manually juggling offsets and creating multiple loopback devices is, in general, no longer necessary. The 'secret weapon' for creating virtual filesystem images in most Linux distributions nowadays is the -P option to the losetup command:

-P, --partscan
              Force the kernel to scan the partition table on a newly created loop device.

You can basically think of -P as the "pretend this file is a disk" option. An example should make this clear. This command creates a 4GB image file filled with zeroes:

$ dd if=/dev/zero of=./sdcard.img bs=1024 count=4194304
4194304+0 records in
4194304+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 39.7914 s, 108 MB/s

The following command maps the first available loopback device to this file, and does so in a way that the kernel interprets the bytes at the beginning as a partition table:

$ sudo losetup -fP ./sdcard.img

$ losetup --list
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /home/evadeflow/Desktop/sdcard.img

It may seem like a small thing, but passing -P means that you can now use tools like fdisk to partition the 'disk' mapped via /dev/loop0. Here, I create a 128 MB partition at the beginning, and a second partition to house all the remaining space:

$ sudo fdisk /dev/loop0
GNU Fdisk 1.3.0a
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
...
Using /dev/loop0
Command (m for help): p

Disk /dev/loop0: 4 GB, 4293596160 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

     Device Boot      Start         End      Blocks   Id  System
Command (m for help): n
Partition type
   e   extended
   p   primary partition (1-4)
p
First cylinder  (default 0cyl):
Last cylinder or +size or +sizeMB or +sizeKB  (default 521cyl): +128M
Command (m for help): n
Partition type
   e   extended
   p   primary partition (1-4)
p
First cylinder  (default 15cyl):
Last cylinder or +size or +sizeMB or +sizeKB  (default 521cyl):
Command (m for help): p

Disk /dev/loop0: 4 GB, 4293596160 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

     Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1               1          16      128488   83  Linux
Warning: Partition 1 does not end on cylinder boundary.
/dev/loop0p2              16         522     4064445   83  Linux
Command (m for help): w
Information: Don't forget to update /etc/fstab, if necessary.


Writing all changes to /dev/loop0.

Note that Linux created new loopback devices on your behalf for each of the two partitions: /dev/loop0p1 and /dev/loop0p2. You can see their relationship with /dev/loop by running lsblk:

$ lsblk /dev/loop0
NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0       7:0    0     4G  0 loop
├─loop0p1 259:2    0 122.1M  0 loop
└─loop0p2 259:3    0   3.9G  0 loop

To populate these partitions, you can create filesystems using the loopback devices /dev/loop0p1 and /dev/loop0p2, then simply mount them, and copy files into the mount points. The process looks sorta like this:

$ sudo mkfs.ext4 /dev/loop0p1
$ sudo mkfs.ext4 /dev/loop0p2
$ mkdir part1-stuff part2-stuff
$ sudo mount /dev/loop0p1 ./part1-stuff
$ sudo mount /dev/loop0p2 ./part2-stuff
$ cp /stuff-source/part1/* ./part1-stuff
$ cp /stuff-source/part2/* ./part2-stuff
$ sudo umount ./part1-stuff
$ sudo umount ./part2-stuff

Once the partitions look the way you want them to, simply 'detach' /dev/loop0 from the file using:

$ sudo losetup -d /dev/loop0

Note that this also detaches the /dev/loop0p1 and /dev/loop0p2 devices that Linux created for you.

So, all in all the accepted answer remains 100% correct, and provides valuable insight into what's going on under the covers. But if your distribution's losetup supports the -P option, you can just use standard partitioning tools and let Linux handle creation (and deletion) of the loop 'sub-device' for each partition.

Comments

[Saad Malik]

The accepted answer should be this one!