Unable to zfs send | zfs receive datasets in same zpool

Question

I have two environments Staging (staging) and Development (choang). I am unable to replicate (including snapshots) from dataset zfs/staging.assets to another dataset zfs/choang.assets within the same zpool zfs.

NOTE: I am assuming I need to unmount both origin and destination datasets.

zfs unmount zfs/staging.assets
zfs unmount zfs/choang.assets

zfs send -R zfs/staging.assets | zfs receive -F zfs/choang.assets

When executed, the above commands generate the following error:

Error: Unsupported flag with filesystem or bookmark.
cannot receive: failed to read from stream

When I remove the -R option and execute the commands, it succeeds:

zfs send zfs/staging.assets | zfs receive -F  zfs/choang.assets

However, no snapshots are received and a single snapshot zfs/choang.assets@--head-- is created.

Finally, I attempted to send a snapshot-- thinking maybe I might send a snapshot at a time:

zfs send zfs/staging.assets@sha512_hash | zfs receive -Fduv zfs/choang.assets

This didn't work either and generated the following error:

internal error: Invalid argument
cannot receive: failed to read from stream

How can I replicate all snapshots?

Solution 1

A kernel update recently occurred on the server (via CPanel) and the server hasn't been restarted. After reviewing log files I noticed that ZFS has been updated as well. I believe the commands in this question are malfunctioning due to version conflict between kernel module and CLI tools.

I arrived at this conclusion after successfully executing the following on a different server.

# zfs send -R tank1@--refresh-- | zfs recv -Fu tank2

# zfs list -t snapshot
NAME                USED  AVAIL  REFER  MOUNTPOINT
tank1@turtle          9K      -    34K  -
tank1@tiger           9K      -  47.5K  -
tank1@squirrel       10K      -  58.5K  -
tank1@rabbit         10K      -    70K  -
tank1@owl            11K      -  80.5K  -
tank1@deer           11K      -  90.5K  -
tank1@bear             0      -   106K  -
tank1@--refresh--      0      -   106K  -
tank2@turtle          9K      -    34K  -
tank2@tiger           9K      -  47.5K  -
tank2@squirrel       10K      -  58.5K  -
tank2@rabbit         10K      -    70K  -
tank2@owl            11K      -  80.5K  -
tank2@deer           11K      -  90.5K  -
tank2@bear            1K      -   106K  -
tank2@--refresh--      0      -   106K  -

I then returned to the original server and switch ZFS module from DKMS to kABI module system for the kernel and restarted server. The commands executed correctly. See https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS

Solution 2

Several things are important here. Your errors stem from combinations of them:

• Normally you send a specific snapshot or several snapshots instead of a whole file system. This means you do not need to unmount the datasets and disrupt your users, and you can incrementally send/recv later on.
• If you do not specify a snapshot on the source, you will get the automatically generated snapshot @--head--, which is the state of your source at the moment of sending (if you would have sent an existing snapshot, that snapshot would take the place of @--head-- on the destination side).

• The send -R | recv -F combination means full replication (recursive and including properties on source, destroy old stuff on destination), so you need to decide how to expand the file system hierarchy: you can use either -e, -d, or no flag on receiving (no flag means merging the contents under the new dataset without preserving the name of the parent dataset on the source side):

  The -d and -e options cause the file system name of the target snapshot
  to be determined by appending a portion of the sent snapshot's name to
  the specified target filesystem.  If the -d option is specified, all
  but the first element of the sent snapshot's file system path (usually
  the pool name) is used and any required intermediate file systems
  within the specified one are created. If the -e option is specified,
  then only the last element of the sent snapshot's file system name
  (i.e. the name of the source file system itself) is used as the target
  file system name.
  

• Your last idea (single send, full receive) should work (I tested it in a simple environment and it did work), but it would not be what you wanted anyway.

---

So, to sum it up and apply to your specific situation:

1. First recursively create a current snapshot or choose an older one that contains all the older stuff you want to replicate):

   zfs snapshot -r zfs/staging.assets@now
   

2. Destroy any old snapshots on the destination side which are on the source side (show all snapshots with zfs list -Hr -o name -t snap zfs/choang.assets or take the hints from the error message). Alternatively destroy the destination dataset and recreate it if it does not contain anything of importance.

3. Send recursively and fully receive, destroying all old datasets on the second dataset, merging the sub-datasets into the destination so that they mirror the source:

   zfs send -R zfs/staging.assets@now | zfs recv -Fu zfs/choang.assets
   

Solution 3

Using Send / Receive on ZFS you are able to use it only with snapshots and it is not necessary to unmount the datasets. You will be not able to send / receive a volume.

The zfs send command creates a stream representation of a snapshot that is written to standard output. By default, a full stream is generated. You can redirect the output to a file or to a different system. The zfs receive command creates a snapshot whose contents are specified in the stream that is provided on standard input. If a full stream is received, a new file system is created as well. You can send ZFS snapshot data and receive ZFS snapshot data and file systems with these commands.

The proper commands are:

zfs send tank/data@snap1 | zfs recv spool/ds01

The better way to send / receive a zfs snapshot is using mbuffer to minimize the risk of I/O latency and to fill up the network buffer.

On the sending machine:

zfs send pool/image@test | mbuffer -s 128k -m 1G -O 127.0.0.1:9090

On the receiving machine:

mbuffer -s 128k -m 1G -I 9090 | zfs receive -F pool/image1

As you can use 127.0.0.1 for the same machine or xx.xx.xx.xx for remote machine.

Source: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html

Comments

[Michael Niño]

Thank you for the example using mbuffer. That was on my list of tools to implement

[Michael Niño]

I attempted the step-by-step however I am still receiving the following error message: zfs send -R zfs/staging.assets@refresh | zfs recv -Fu zfs/choang.assets internal error: Invalid argument cannot receive: failed to read from stream

[Michael Niño]

In addition, please note that each of the several snapshots for zfs/staging.assets as well as the zfs/staging.assets@refresh (read: @now in your example) were taken with -r. Could it be because I am sending replicating to and from datasets within the same zpool? Or do you think it is something else?