scp remote file and append to local file

Question

I have multiple files to be pulled from remote server. For further processing of the files in the local server, I need to merge (concatenate) them into single file, which can't be done in the remote file though.

I am not sure how scp work internally, but for the best performance I believe instead of writing those files into local directory and then merge, I feel, I should merge them on the fly and then write into single file. Can you please let me know if merging (appending) the files on fly during scp from remote to local files possible?

If not any better idea?

Solution 1

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:{file1,file2...} local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

Solution 2

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$ mkfifo p
$ while :; do cat p >> output ; done  &
$ scp somehost:test/\* p
bar       100%    4    10.9KB/s   00:00    
doo       100%    4     8.6KB/s   00:00    
foo       100%    4    13.6KB/s   00:00  
$ kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

Comments

[Betta]

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?

[muru]

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.

[Betta]

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?

[muru]

@Betta why do you imagine ssh and cat is not as secure as scp?

[Betta]

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!

[Betta]

I said, I can't merge on the remote server! :)

[muru]

@Betta ssh cp is not sufficient, because cp has no idea about the local system. There's no equivalent of scp foo remote:bar -- ssh cp foo ???