Backup/clone live to a new partition which can be booted

grub2 bash upgrade backup clone

16,240

Solution 1

Bash script to clone active Ubuntu Partition to clone partition

The clone-ubuntu.sh bash script will seamlessly and safely replicate 16.04 LTS into a partition for upgrading to 18.04 LTS:

Important points to consider:

You must create an empty ext4 partition large enough to hold a Ubuntu 16.04 clone
When calling the script clone-ubuntu.sh the partition cannot be mounted. The script automatically mounts and unmounts the partition.
The command rsync is used to copy files from / to the clone partition. The first time you run clone-ubuntu.sh it will take a few minutes. The second time you run the script only file changes are updated and it should take less than a minute.
You might run this script and reboot multiple times. Any new data on the clone target will deleted to mirror the current / files & directories.
All Cron /etc/cron.d (reboot) jobs are moved to a new sub-directory called /etc/cron.d/hold. After booting clone remember to run sudo crontab -e to prevent selected cron jobs from running.
The cloned file /etc/fstab is modified with the appropriate UUID for the partition it is on.
The cloned file /boot/grub/grub.cfg is modified with the appropriate UUID for successful clone booting. The cloned file's quiet splash is changed to nosplash so you get scrolling terminal messaging. This gives visual awareness booting a clone rather than "Real" version.
update-grub is run to update Grub with new menu options pointing to the cloned partition.
The cloned file /etc/update-manager/release-upgrades is modified to change Prompt=never to Prompt=lts. When you boot the clone and perform do-release-upgrade -d this allows Ubuntu 16.04 to be upgraded to 18.04.
Before cloning a confirmation screen is displayed (shown in the next section) and you must type y/Y to proceed.

Confirmation Screen

After selecting a target clone partition it is first verified to be ext4 partition type and is not already mounted. If this test is passed, a confirmation message then appears:

=====================================================================
Mounting clone partition /dev/nvme0n1p8 as /mnt/clone16.04

=====================================================================
PLEASE: Carefully confirm Source (Live) and Target (Clone) partitions

SOURCE (BOOT /):  /dev/nvme0n1p5      TARGET (CLONE):  /dev/nvme0n1p8
ID:               Ubuntu              ID:              Ubuntu
RELEASE:          16.04               RELEASE:         16.04
CODENAME:         xenial              CODENAME:        xenial
DESCRIPTION:      Ubuntu 16.04.3 LTS  DESCRIPTION:     Ubuntu 16.04.3 LTS
 Size  Used Avail Use%                 Size  Used Avail Use%
  44G   17G   26G  40%                  24G   17G  5.8G  74%

NOTE: If you are recloning, new files in clone will be deleted,
      modified files are reset to current source content and,
      files deleted from clone are added back from source.

Type Y (or y) to proceed. Any other key to exit:

In this example a previous clone has been selected for recloning. The available space on the clone is a mute point because we already know there is enough space available.

If you have multiple Ubuntu installations, please verify you have selected the right partition to clone the currently booted Ubuntu, mounted as / (root), to.

This is your last chance to abort by pressing any key except y or Y.

Output listing

When you run the script you will get this output (excluding the output already listed above):

=====================================================================
Using rsync to clone / to /dev/nvme0n1p8 mounted as /mnt/clone16.04
          6.11G  38%   86.46MB/s    0:01:07 (xfr#139123, to-chk=0/647700)   

Number of files: 647,700 (reg: 470,100, dir: 104,694, link: 72,903, special: 3)
Number of created files: 127,824 (reg: 72,472, dir: 15,825, link: 39,526, special: 1)
Number of deleted files: 73,318 (reg: 59,766, dir: 9,701, link: 3,847, special: 4)
Number of regular files transferred: 139,123
Total file size: 15.92G bytes
Total transferred file size: 6.11G bytes
Literal data: 6.11G bytes
Matched data: 0 bytes
File list size: 8.50M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 6.14G
Total bytes received: 7.82M

sent 6.14G bytes  received 7.82M bytes  89.74M bytes/sec
total size is 15.92G  speedup is 2.59

Time to clone files: 68 Seconds

=====================================================================
Making changes in: /mnt/clone16.04/etc/update-manager/release-upgrades
     from Prompt=: never
       to Prompt=: lts
Allows running 'do-release-upgrade -d' when rebooting clone target
Consider 'do-release-upgrade -d -f DistUpgradeViewNonInteractive' This
allows you to go to bed or go to lunch whilst upgrade runs.

* * *  When you Upgrade, TURN OFF screen locking for inactivity. * * *

=====================================================================
Making changes in: /mnt/clone16.04/etc/fstab
        from UUID: f3f8e7bc-b337-4194-88b8-3a513f6be55b
          to UUID: 113f9955-a064-4ce2-9cae-74f2a9518550

=====================================================================
Making changes in: /mnt/clone16.04/boot/grub/grub.cfg
        from UUID: f3f8e7bc-b337-4194-88b8-3a513f6be55b
          to UUID: 113f9955-a064-4ce2-9cae-74f2a9518550
Also change 'quiet splash' to 'nosplash' for environmental awareness
Suggest first time booting clone you make wallpaper unique

=====================================================================
Calling 'update-grub' to create new boot menu
Generating grub configuration file ...
Found background: /home/rick/Pictures/1600x900/21.jpg
Found background image: /home/rick/Pictures/1600x900/21.jpg
Found linux image: /boot/vmlinuz-4.14.34-041434-generic
Found initrd image: /boot/initrd.img-4.14.34-041434-generic
Found linux image: /boot/vmlinuz-4.14.31-041431-generic
Found initrd image: /boot/initrd.img-4.14.31-041431-generic
Found linux image: /boot/vmlinuz-4.14.30-041430-generic
Found initrd image: /boot/initrd.img-4.14.30-041430-generic
Found linux image: /boot/vmlinuz-4.14.27-041427-generic
Found initrd image: /boot/initrd.img-4.14.27-041427-generic
Found linux image: /boot/vmlinuz-4.14.15-041415-generic
Found initrd image: /boot/initrd.img-4.14.15-041415-generic
Found linux image: /boot/vmlinuz-4.14.10-041410-generic
Found initrd image: /boot/initrd.img-4.14.10-041410-generic
Found linux image: /boot/vmlinuz-4.14.4-041404-generic
Found initrd image: /boot/initrd.img-4.14.4-041404-generic
Found linux image: /boot/vmlinuz-4.14.2-041402-generic
Found initrd image: /boot/initrd.img-4.14.2-041402-generic
Found linux image: /boot/vmlinuz-4.13.9-041309-generic
Found initrd image: /boot/initrd.img-4.13.9-041309-generic
Found linux image: /boot/vmlinuz-4.10.0-42-generic
Found initrd image: /boot/initrd.img-4.10.0-42-generic
Found linux image: /boot/vmlinuz-4.9.77-040977-generic
Found initrd image: /boot/initrd.img-4.9.77-040977-generic
Found linux image: /boot/vmlinuz-4.4.0-104-generic
Found initrd image: /boot/initrd.img-4.4.0-104-generic
Found linux image: /boot/vmlinuz-3.16.53-031653-generic
Found initrd image: /boot/initrd.img-3.16.53-031653-generic
Found Windows Boot Manager on /dev/nvme0n1p2@/EFI/Microsoft/Boot/bootmgfw.efi
Found Ubuntu 16.04.3 LTS (16.04) on /dev/nvme0n1p8
Found Windows Boot Manager on /dev/sda1@/efi/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for EFI firmware configuration
done

=====================================================================
Unmounting /dev/nvme0n1p8 as /mnt/clone16.04

`rsync` status display for new clone

When cloning for the first time, rsync will give an update from 0 to 100% of all files created. No files will be deleted or changed as the clone is empty:

`rsync` status display when recloning

When rsync reclones it never hits 100% because files that never changed are not copied. There will be delays in the update progress as rsync scans for the next file to be copied and when it deletes new files created in the clone that never existed in the original:

Bash script - `clone-ubuntu.sh`

#!/bin/bash

# NAME: clone-ubuntu.sh
# PATH: /usr/local/bin
# DESC: Written for AU Q&A: https://askubuntu.com/questions/1028604/bash-seemless-safe-script-to-upgrade-16-04-to-18-04/1028605#1028605
# DATE: Apr 27, 2018. Modified May 6, 2018.

# UPDT: May 02 2018 - Display selected parition and get confirmation.
#       May 06 2018 - Revise `do-release-upgrade -d` instructions.
#                     Correct listing of files in empty target partition.
#       Aug 09 2018 - Add --inplace parameter to `rsync`
#                     Comment out disabling `/etc/cron.d` on clone target.
#                     Users may uncomment and/or revise to their needs.

# $TERM variable may be missing when called via desktop shortcut
CurrentTERM=$(env | grep TERM)
if [[ $CurrentTERM == "" ]] ; then
    notify-send --urgency=critical \ 
                "$0 cannot be run from GUI without TERM environment variable."
    exit 1
fi

# Must run as root
if [[ $(id -u) -ne 0 ]] ; then echo "Usage: sudo $0" ; exit 1 ; fi

#
# Create unqique temporary file names
#

tmpPart=$(mktemp /tmp/clone-ubuntu.XXXXX)   # Partitions list
tmpMenu=$(mktemp /tmp/clone-ubuntu.XXXXX)   # Menu list
tmpInf1=$(mktemp /tmp/clone-ubuntu.XXXXX)   # Soucre (Booted) Ubuntu Info
tmpInf2=$(mktemp /tmp/clone-ubuntu.XXXXX)   # Target (Cloned) Ubuntu Info
tmpInf3=$(mktemp /tmp/clone-ubuntu.XXXXX)   # Work file used by DistInfo ()

#
# Function Cleanup () Removes temporary files
#

CleanUp () {
    [[ -f "$tmpPart" ]] && rm -f "$tmpPart" # If we created temp files
    [[ -f "$tmpMenu" ]] && rm -f "$tmpMenu" #  at various program stages
    [[ -f "$tmpInf1" ]] && rm -f "$tmpInf1" #  then remove them before
    [[ -f "$tmpInf2" ]] && rm -f "$tmpInf2" #  exiting.
    [[ -f "$tmpInf3" ]] && rm -f "$tmpInf3"

    if  [[  -d "$TargetMnt" ]]; then        # Did we create a clone mount?
        umount "$TargetMnt" -l              # Unmount the clone
        rm  -d "$TargetMnt"                 # Remove clone directory
    fi
}

#
# Function GetUUID () gets UUIDs of source and clone target partitions in menu.
#

GetUUID () {

    SrchLine="$1"                           # menu line passed to function
    UUID_col=0                              # start column of UUID in line
    lsblk -o NAME,UUID > "$tmpPart"         # Get list of UUID's

    while read -r UUID_Line; do             # Read through UUID list

        # Establish UUID position on line
        if [[ $UUID_col == 0 ]] ; then      # First time will be heading
            UUID_col="${UUID_Line%%UUID*}"  # Establish column number
            UUID_col="${#UUID_col}"         #  where UUID appears on line
            NameLen=$(( UUID_col - 1 ))     # Max length of partition name
            continue                        # Skip to read next line
        fi

        # Check if Passed line name (/dev/sda1, /nvme01np8, etc.) matches.
        if [[ "${UUID_Line:0:$NameLen}" == "${SrchLine:0:$NameLen}" ]] ; then
            FoundUUID="${UUID_Line:UUID_col:999}"
            break                           # exit function
        fi

    done < "$tmpPart"                       # Read next line & loop back
}


#
# Function DistInfo () builds information about source & target partitions
#

DistInfo () {

    Mount="$1"                              # Mount name is '/' or $TargetMnt
    FileName="$2"                           # "$tmpInf1" or "$tmpInf2" work file
    cat "$Mount"/etc/lsb-release >> "$FileName"
    sed -i 's/DISTRIB_//g' "$FileName"      # Remove DISTRIB_ prefix.
    sed -i 's/=/:=/g' "$FileName"           # Change "=" to ":="
    sed -i 's/"//g' "$FileName"             # Remove " around "Ubuntu 16.04...".

    # Align columns from "Xxxx:=Yyyy" to "Xxxx:      Yyyy"
    cat "$FileName" | column -t -s '=' > "$tmpInf3"
    cat "$tmpInf3" > "$FileName"
}


#
# Mainline
#

lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT > "$tmpMenu"

i=0
SPACES='                                                                     '
DoHeading=true
AllPartsArr=()      # All partitions.

# Build whiptail menu tags ($i) and text ($Line) into array

while read -r Line; do
    if [[ $DoHeading == true ]] ; then
        DoHeading=false                     # First line is the heading.
        MenuText="$Line"                    # Heading for whiptail.
        FSTYPE_col="${Line%%FSTYPE*}"           
        FSTYPE_col="${#FSTYPE_col}"         # Required to ensure `ext4`.
        MOUNTPOINT_col="${Line%%MOUNTPOINT*}"
        MOUNTPOINT_col="${#MOUNTPOINT_col}" # Required to ensure not mounted.
        continue
    fi

    Line="$Line$SPACES"                     # Pad extra white space.
    Line=${Line:0:74}                       # Truncate to 74 chars for menu.

    if [[ "${Line:MOUNTPOINT_col:4}" == "/   " ]] ; then
        GetUUID "$Line"
        SourceUUID=$FoundUUID
        # Build "/dev/Xxxxx" FS name from "├─Xxxxx" lsblk line
        SourceDev="${Line%% *}"
        SourceDev=/dev/"${SourceDev:2:999}"
    fi

    AllPartsArr+=($i "$Line")               # Menu array entry = Tag# + Text.
    (( i++ ))

done < "$tmpMenu"                           # Read next "lsblk" line.

#
# Display whiptail menu in while loop until no errors, or escape,
# or valid partion selection .
#

DefaultItem=0

while true ; do

    # Call whiptail in loop to paint menu and get user selection
    Choice=$(whiptail \
        --title "Use arrow, page, home & end keys. Tab toggle option" \
        --backtitle "Clone 16.04 for upgrade.  ONLY CLONES / PARTITION" \
        --ok-button "Select unmounted partition" \
        --cancel-button "Exit" \
        --notags \
        --default-item "$DefaultItem" \
        --menu "$MenuText" 24 80 16 \
        "${AllPartsArr[@]}" \
        2>&1 >/dev/tty)

    clear                                   # Clear screen.
    if [[ $Choice == "" ]]; then            # Escape or dialog "Exit".
        CleanUp
        exit 0;
     fi

    DefaultItem=$Choice                     # whiptail start option.
    ArrNdx=$(( $Choice * 2 + 1))            # Calculate array offset.
    Line="${AllPartsArr[$ArrNdx]}"          # Array entry into $Line.

    # Validation - Don't wipe out Windows or Ubuntu 16.04:
    # - Partition must be ext4 and cannot be mounted.

    if [[ "${Line:FSTYPE_col:4}" != "ext4" ]] ; then
        echo "Only 'ext4' partitions can be clone targets."
        read -p "Press <Enter> to continue"
        continue
    fi

    if [[ "${Line:MOUNTPOINT_col:4}" != "    " ]] ; then
        echo "A Mounted partition cannot be a clone target."
        read -p "Press <Enter> to continue"
        continue
    fi

    GetUUID "$Line"                         # Get UUID of target partition.
    TargetUUID=$FoundUUID

    # Build "/dev/Xxxxx" FS name from "├─Xxxxx" menu line
    TargetDev="${Line%% *}"
    TargetDev=/dev/"${TargetDev:2:999}"

    break                                   # Validated: Break menu loop.

done                                        # Loop while errors.

#
# Mount Clone Target partition
#

Release=$(lsb_release -rs)                  # Source version ie: '16.04'
TargetMnt="/mnt/clone$Release"

echo ""
echo "====================================================================="
echo "Mounting clone partition $TargetDev as $TargetMnt"
mkdir -p "$TargetMnt"                       # '-p' directory may already exist
mount -t auto -v $TargetDev "$TargetMnt" > /dev/null

# Confirm partition is empty. If not empty confirm it's Ubuntu. If not exit.
# If Ubuntu display prompt with the version it contains and get confirmation.

echo ""
echo "====================================================================="
echo "PLEASE: Carefully confirm Source (Live) and Target (Clone) partitions"

# Build source information (our current boot partition)
echo "SOURCE (BOOT /)=$SourceDev"  > "$tmpInf1"
DistInfo "/" "$tmpInf1"                     # /etc/lsb_release information
df -h --output=size,used,avail,pcent "$SourceDev" >> "$tmpInf1"

# Build target information (the partition selected for cloning to)
LineCnt=$(ls "$TargetMnt" | wc -l)
if (( LineCnt > 1 )) ; then 
    # More than /Lost+Found exist so it's not an empty partition.
    if [[ -f "$TargetMnt"/etc/lsb-release ]] ; then
        echo "TARGET (CLONE)=$TargetDev" > "$tmpInf2"
        DistInfo "$TargetMnt" "$tmpInf2"    # /etc/lsb_release information
    else
        # TO-DO: might be cloning /boot or /home on separate partitions.
        #        the source partition is still `/` so can display message.
        echo "Selected partition has data which is not Ubuntu OS. Aborting."
        CleanUp                             # Remove temporary files
        exit 1
    fi
else
    echo "Target (Clone) partition appears empty" > "$tmpInf2"
    echo "/Lost+Found normal in empty partition" >> "$tmpInf2"
    echo "Head of '/Clone/' files & directories:" >> "$tmpInf2"
    ls "$TargetMnt" | head -n2 >> "$tmpInf2"
fi

# Target device free bytes
df -h --output=size,used,avail,pcent "$TargetDev" >> "$tmpInf2"

# Display source and target partitions side-by-side using bold text.
echo $(tput bold)       # Set to bold text
paste -d '|' "$tmpInf1" "$tmpInf2" | column -t -s '|'
echo $(tput sgr0)       # Reset to normal text

echo "NOTE: If you are recloning, new files in clone will be deleted,"
echo "      modified files are reset to current source content and,"
echo "      files deleted from clone are added back from source."
echo ""

read -p "Type Y (or y) to proceed. Any other key to exit: " -n 1 -r
echo    # (optional) move to a new line
if [[ ! $REPLY =~ ^[Yy]$ ]] ; then
    CleanUp             # Remove temporary files
    exit 0
fi

# Copy non-virtual directories to clone. Credit to TikTak's Ask Ubuntu answer:
# https://askubuntu.com/questions/319805/is-it-safe-to-clone-the-current-used-disk?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

SECONDS=0
echo ""
echo "====================================================================="
echo "Using rsync to clone / to $TargetDev mounted as $TargetMnt"
rsync -haxAX --stats --delete --info=progress2 --info=name0 --inplace  \
      /* "$TargetMnt"                                                   \
      --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found}
# For 16GB on Samsung Pro 960: First time 98 seconds, second time 27 seconds.
rsyncTime=$SECONDS  
echo ""
echo "Time to clone files: $rsyncTime Seconds"

# Change /etc/update-manager/release-upgrades prompt from never to LTS
echo ""
echo "====================================================================="
echo "Making changes in: $TargetMnt/etc/update-manager/release-upgrades"
echo "     from Prompt=: never"
echo "       to Prompt=: lts"
echo "Allows running 'do-release-upgrade -d' when rebooting clone target"
echo "Consider 'do-release-upgrade -d -f DistUpgradeViewNonInteractive' This"
echo "allows you to go to bed or go to lunch whilst upgrade runs."
echo ""
echo "* * *  When you Upgrade, TURN OFF screen locking for inactivity. * * *"
echo ""
sed -i 's/Prompt=never/Prompt=lts/' "$TargetMnt"/etc/update-manager/release-upgrades

## This section commented out to prevent surprises. You may uncomment.
## You may want to revise to include `cron.daily`, `cron.hourly`, etc.
# Move `/etc/cron.d` reboot jobs to `/etc/cron.d/hold` to prevent running
# scripts such as daily backup or Ubuntu 16.04 specific problem fixes.
#echo ""
#echo "====================================================================="
#echo "Moving '$TargetMnt/etc/cron.d' to '.../hold' to prevent running."
#echo "After booting clone, move back individual files you want to run"
#if [[ ! -d "$TargetMnt"/etc/cron.d/hold ]]; then
#    mkdir "$TargetMnt"/etc/cron.d/hold
#fi
#cp -p  "$TargetMnt"/etc/cron.d/* "$TargetMnt"/etc/cron.d/hold/
#rm -fv "$TargetMnt"/etc/cron.d/*

# Update /etc/fstab on clone partition with clone's UUID
echo ""
echo "====================================================================="
echo "Making changes in: $TargetMnt/etc/fstab"
echo "        from UUID: $SourceUUID"
echo "          to UUID: $TargetUUID"
sed -i "s/$SourceUUID/$TargetUUID/g" "$TargetMnt"/etc/fstab

# Update /boot/grub/grub.cfg on clone partition with clone's UUID
echo ""
echo "====================================================================="
echo "Making changes in: $TargetMnt/boot/grub/grub.cfg"
echo "        from UUID: $SourceUUID"
echo "          to UUID: $TargetUUID"
echo "Also change 'quiet splash' to 'nosplash' for environmental awareness"
echo "Suggest first time booting clone you make wallpaper unique"
sed -i "s/$SourceUUID/$TargetUUID/g" "$TargetMnt"/boot/grub/grub.cfg
sed -i "s/quiet splash/nosplash/g" "$TargetMnt"/boot/grub/grub.cfg

# Update grub boot menu
echo ""
echo "====================================================================="
echo "Calling 'update-grub' to create new boot menu"
update-grub

# Unmount and exit

echo ""
echo "====================================================================="
echo "Unmounting $TargetDev as $TargetMnt"

CleanUp             # Remove temporary files

exit 0

Copy and paste the bash code above to a new file called /usr/local/bin/clone-ubuntu.sh. Then make the new file executable using:

sudo chmod a+x /usr/local/bin/clone-ubuntu.sh

To call the script use:

sudo clone-ubuntu.sh

How to upgrade 16.04 LTS clone to Ubuntu 18.04 LTS

This is a "bonus" section that may interest many people.

Reboot your machine. The grub menu will contain a new menu option pointing to the cloned partition. You can also select a specific kernel version from the clone's Advanced Options menu.

One way to convert the cloned 16.04 LTS to 18.04 LTS is to run:

sudo do-release-upgrade

Note the -d flag was required prior to July 26, 2018 but is no longer necessary.

Read the 18.04 upgrade confirmation carefully.

Before proceeding ensure inactive screen locking is turned off. The upgrade process could crash if your computer goes to the lock screen due to keyboard inactivity.

Summary of 18.04 upgrade process

This section will be machine specific because different apps are installed by different users. Here is a brief summary I made based on notes and memory:

confirmation to proceed: ENTER
packages will be removed: Y
replace longind.conf: Y
Configuration file '/etc/sane.d/dll.conf', default N, take Y
'/etc/NetworkManager/conf.d/default-wifi-powersave-on.conf' Take default N
Configuration file '/etc/pulse/default.pa' default N, take Y
Configuration file '/etc/grub.d/30_os-prober'' default N, take N
Full screen grub menu config appears. Take option: keep the local version currently installed
Configuration file '/etc/cron.d/anacron', default N, take Y to see what
An error message multiple times: /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf: No such file or directory
Non-standard: Configuration file '/etc/vnstat.conf' (display differences 1.13 vs 1.18) take Y
220 packages are going to be removed. (can take hours) enter Y
To finish the upgrade, a restart is required. Take Y

Note step 10 most people will never see. I have an old nVidia driver installation on my Ubuntu 16.04 I never got around to fully removing. It's included because you might have similar old packages never fully removed.

Grub changes boot menu to clone under UEFI

Unlike 18.04 upgrades I found the 18.04.1 upgrade changed the Grub UEFI configuration to use the clone's grub menu instead of the original grub menu as per this answer: Dual boot and the files /boot/grub/grub.cfg -- which one is used?

$ sudo cat /boot/efi/EFI/ubuntu/grub.cfg
search.fs_uuid b40b3925-70ef-447f-923e-1b05467c00e7 root 
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg

$ sudo grub-install
Installing for x86_64-efi platform.
Installation finished. No error reported.

Running sudo update-grub after booting the original partition isn't enough to change which grub.cfg is loaded by grub. You need to use sudo grub-install to force grub to use the original partition's configuration.

Both the original partition and clone partition can use update-grub to maintain their own /boot/grub/grub.cfg file but only one can be used during boot.

Of course if you want grub to use the Clone's grub menu do not do the above steps on the original's partition.

Edit history

Edit May 6, 2018 - Information display for an empty clone (target) partition corrected.

Edit August 26, 2018 - Use --inplace option with rsync so large files such as a 2 GB trash file are not duplicated on clone during copy process. This can lead to "out of disk space" error. Comment out /etc/cron.d overrides because users may want to keep or want a different directory altogether. Update instructions to use do-release-upgrade without -d flag because Ubuntu 18.04.1 LTS is now released.

Solution 2

Problems with upgrading from previous releases to 18.04 LTS

It is not at all straightforward to upgrade from previous releases to 18.04 LTS. I don't know if there are more problems than such upgrading in the past, but people who do release upgrading now take a risk for themselves. On the other hand they find bugs and if the bugs are reported, it will help develop the upgrading tool as well as debug the Ubuntu 18.04 LTS system itself.

This means that people who are patient enough to wait until the upgrading is officially released with the first point release (18.04.1 LTS) will get a smoother ride.

Testing before doing the full upgrade ...

The method that @WinEunuuchs2Unix describes here makes it possible to test with a copy of the real system, if upgrading to 18.04 LTS will work with your current [more or less modified] operating system with your computer hardware.

This can prevent several disasters with corrupted operating system.

I have not used @WinEunuuchs2Unix's script yet, but I understand, that it is very useful, and I intend to use it. I was able to upgrade from 8.04 to 10.04 to 12.04, which I used for a long time. But when I set out to upgrade via 14.04 to 16.04 I failed and could not find the errors.

... and if problems, make a fresh installation

I had good backups, so I made a fresh install of 16.04 and later on copied, what I wanted to keep and checked for tweaks, scripts, aliases and installed programs. If I had used @WinEunuuchs2Unix's method I had seen the problem early, I would have made a fresh install directly.

Upgrading a persistent live system

Persistent live systems are used to get very portable Ubuntu systems, that are sometimes used for testing but sometimes used for a long time.

It is a known problem, that you should not update & upgrade such a system like you do with an installed system because it will get corrupted sooner or later. Furthermore, you are stuck with the kernel and kernel drivers, that come with the iso file, because they are started before the overlay system is started.

But it is usually possible to keep the /home directory, like you can, when you make a fresh installation with a separate 'home' partition. If you create a home-rw partition, a persistent live system will find and use it automatically during boot.

I am developing and testing a shellscript, mk-persistent-live_with_home-rw, that can create a persistent live system with a home-rw partition, and that can later upgrade it,

replace the iso file with a newer one. At least you should upgrade when a new iso file is released, but you can also upgrade an LTS system once a month and use the current daily iso files from the testing tracker,
modify grub for the new iso file.
wipe (reformat) the casper-rw file which stores the modifications of the operating system (so you must re-install the program packages that you added to the system),
preserve the home-rw partition with your personal files, settings and tweaks.
See this link for more details.

Extra link

General tips for people who want to test the latest and greatest version of Ubuntu can be found in the following link,

How to participate in testing and development of Ubuntu

16,240

WinEunuuchs2Unix

Software development is my main hobby. Check out the new websites created in October 2021: www.pippim.com and pippim.github.io

Updated on September 18, 2022

Comments

WinEunuuchs2Unix over 1 year

Ubuntu 18.04 came out April 26, 2018 and I want to try it to upgrade my programs and convert my data but don't want to commit if there are bugs.

I've shrunk Windows from 410 GB to 385 GB, rebooted Ubuntu 16.04 and ran gparted to create a new 25 GB partition labeled "Ubuntu18.04". I ran rm-kernels and removed about 20 kernels to eliminate about 10 GB on Ubuntu 16.04.

Now I want a script that will populate the new partition with 16.04 LTS and create a new Grub menu option to it. Only relevant directories should be copied. For example /sys, /run, /proc and /dev are virtual directories created during boot and shouldn't be copied.

I also want /etc/fstab patched with the correct UUID and cron reboot jobs disabled such that daily backups on cloned data isn't run after booting the clone.

I anticipate running the script many times over the next few weeks/months. As such the cloning process should effortlessly repeatable.

The same script could be used for testing security updates and new Ubuntu Kernel Team updates without effecting production systems.
- Lie Ryan over 5 years
  
  20 kernels takes 10GB!!!? Did you accidentally install kernel bloated edition?
- WinEunuuchs2Unix over 5 years
  
  @LieRyan It's actually 400MB per kernel and not 500 MB as I rounded the numbers (about 20 kernels). To see how much each kernel takes up (fluctuates from 394MB to 419MB on my current install, see:.askubuntu.com/questions/949411/…
sudodus about 6 years

+1. Thanks for a useful tool :-)
WinEunuuchs2Unix about 6 years

@sudodus I hope people use this tool before going full in with 18.04 upgrade. At the vary least they can use this as a tutorial and repeat the steps manually using a Live USB boot. It's tragic to read so many errors people are having on the brand new 18.04 because they've upgraded without testing first and committed by wiping out their previous 16.04, 17.04 or 17.10 installation. Sadly I've seen similar things many times over the decades. Thank you for your up-vote! It helps reinforce the effort put into such endeavours as clone-ubuntu.sh But I've used this 4 times myself anyway!
sudodus about 6 years

I have not used your script yet, but I understand, that it is very useful, and I intend to use it. -- I was able to upgrade from 8.04 to 10.04 to 12.04, which I used for a long time. But when I set out to upgrade via 14.04 to 16.04 I failed. I had good backups, so I made a fresh install of 16.04 and later on copied over, what I wanted to keep and checked for tweaks, scripts, aliases and installed programs. -- By the way, I work on a script to create and upgrade persistent live drives 'home-rw'. Welcome to try it :-)
WinEunuuchs2Unix about 6 years

Minor typo in your link: I tested all of them in 8.04 LTS should read 18.04 LTS at the end. Your tool is the number 1 go to for many people that don't have enough HDD space to clone their Ubuntu 16.04. It's also easier (though slower) than creating a new partition on their HDD.
sudodus about 6 years

I fixed the typo, thanks :-) You are right about cloning (or copying the content) to a HDD. But for people who want to carry their persistent live system in their pocket, it is convenient to do it with my tool.
WinEunuuchs2Unix about 6 years

@sudodus I can't vouch enough for the ability you give for a live persistent boot USB. It's absolutely fantastic! I also encourage you to write an answer on this thread. You get my automatic up-vote! Another advantage of cloning 16.04 I just realized is to install nVidia drivers which sometimes muck up a system. Do it on a clone first before a real system.
WinEunuuchs2Unix about 6 years

The extra link is a nice touch. The more people that report problems the more problems that will be solved for others.
WinEunuuchs2Unix about 6 years

Very good point for LTS users (cautious by nature) to wait for the first point release 18.04.1. For larger organizations they might still want to clone 16.04 to a test partition and upgrade it to 18.04 for staff training purposes. Users would have access to data they are accustomed to. Of course IT staff would benefit from extensive testing on cloned data months before July 26th first point release too.
dentex over 5 years

Works perfectly on my Linux Mint 19.1 system... I'll try it again to test the boot (and eventually switch) to a NVMe SSD. It should be enough easy to move/reinstall/whatever grub to the new drive.
Naveed Abbas over 4 years

I had ubuntu installed on a USB and I wanted to clone it to another USB. I used this script but now I'm stuck how to make that usb bootable (copy grub commands) not working.
WinEunuuchs2Unix over 4 years

@NaveedAbbas This script is not for cloning MBR, GPT, Grub or Windows. It only clones Ubuntu.To clone the whole drive (USB) see: ubuntupit.com/top-15-best-disk-cloning-software-for-linux
AmanicA over 4 years

I think you should stick this on github were it can be maintained more easily..
WinEunuuchs2Unix over 4 years

@AmanicA Someone else has already done that and made improvements: github.com/thiggy01/clone-ubuntu
Jurriaan almost 4 years

I can confirm this working for Ubuntu 20.04 also. However, the script should probaby error out when the target partition cannot be mounted correctly, since I didn't mkfs.ext4 the target partition (my bad) and the script started happily copying / to the /mnt/.... subdirectory, even if nothing was mounted there.
Yogi Katba over 3 years

I used this script to transfer my HDD data to SSD. After booting up when I selected SSD ubuntu from grub menu, it still boots into old HDD's ubuntu. Help me
Emanuel Moecklin about 3 years

not sure what I'm doing wrong but I get "umount: /mnt/clone20.04: not mounted." at the very end
WinEunuuchs2Unix about 3 years

@EmanuelMoecklin Before running the script make sure the partition you created can be manually mounted and unmounted. Maybe copy a file or two to it and ensure they are correct. See: askubuntu.com/questions/1029040/…
kurja over 2 years

I'm getting an error message as well, new partition can be mounted manually and files can be written there. Script exits and returns: 'Unmounting /dev/me0n1 as /mnt/clone20.04 umount: /mnt/clone20.04: not mounted. rm: cannot remove '/mnt/clone20.04': Directory not empty' and no files are copied to target device
WinEunuuchs2Unix over 2 years

@kurja What kind of drive is your /dev/me0n1?
kurja over 2 years

it's an m.2 nvme in a pcie expansion slot/adapter. It does work just like any other drive as far as I can tell though. Are there known issues with these? I'll have to take another look at exactly what goes on when the script runs when I get home in the evening.
kurja over 2 years

df /mnt/clone20.04 tells me that it's stored on sdc5 (which is my / ). /dev/me0n1 is more interesting - it does not exist, the drive is actually nvme0n1! It is displayed correctly when choosing the target partition in the whiptail menu, and things go wrong after that. Here's the complete output from running the script: pastebin.com/QdNtEqu1
WinEunuuchs2Unix over 2 years

@kurja You can't pick nvme0n1 as a clone target, you must pick one of the partitions on it. For example nvme0n1p3 for partition #3. Before cloning to a partition you must create the partition first. Did you create a clone partition?
kurja over 2 years

@WinEunuuchs2Unix so an rtfm error, you say? :) I seem to have confused myself by creating a file system on the device, instead of making a partition first. Script runs without errors now, thank you.