Copy data through SSH tunnel over multiple hops
Solution 1
By far, the easiest way is to just copy it via scp. Plus, this syntax actually works unlike some of the other suggestions.
You can't beat this syntax for ease. It allows you to recursively copy, rsync or what ever you'd like without the hassle of considering potentially complex pipes. This syntax is intuitively clear, will be more readily supportable by Sys Admins that follow you and does not make useless use of cat.
scp -3 devappserver:/path/to/copy/from qaappserver:/path/to/copy/to
From the scp man page: -3
Copies between two remote hosts are transferred through the local host. Without this option the data is copied directly between the two remote hosts. Note that this option disables the progress meter.
In the example below
- Your workstation is named MacBook-Pro.
- Dev Jump Box is named devjumpserver
- Dev Application Server is named devapplicationserver
- Is on LAN DNS zone named .local
- QA Jump Box is named qajumpserver
- QA Application Server is named qaapplicationserver
- Is on LAN DNZ zone named .local
- We'll perform a test copy of a 670GB /etc/hosts file ;-)
- Assumption is made that you have SSH public key authentication configured.
Here is an ~/.ssh/config file that sets up the direct access from your workstation to the application servers via the appropriate jump (aka bastion server).
MacBook-Pro:~ barrychapman$ cat ~/.ssh/config Host * ServerAliveInterval 60 Host devapplicationsever HostName devapplicationserver.local ProxyCommand ssh -i ~/.ssh/id_rsa [email protected] -W %h:%p User barrychapman Host qaapplicationserver HostName qaapplicationserver.local ProxyCommand ssh -i ~/.ssh/id_rsa [email protected] -W %h:%p User barrychapman MacBook-Pro:~ barrychapman$
Testing for presence of file on target server, it won't be there.
MacBook-Pro:~ barrychapman$ ssh qaapplicationserver ls /tmp/hosts ls: cannot access /tmp/hosts: No such file or directory Killed by signal 1. MacBook-Pro:~ barrychapman$
Now let's copy a file from Dev Application server to QA Application via your workstation.
MacBook-Pro:~ barrychapman$ scp -3 devapplicationserver:/etc/hosts qaapplicationserver:/tmp/ Killed by signal 1. Killed by signal 1. MacBook-Pro:~ barrychapman$
Now let's check for the presence of the copied file on the QA Application Server. It will be there this time.
MacBook-Pro:~ barrychapman$ ssh qaapplicationserver ls /tmp/hosts /tmp/hosts Killed by signal 1. MacBook-Pro:~ barrychapman$
Note
When closing a ProxyCommand connection, you will see the warning message "Killed by signal 1". This is SSH tearing down the ProxyCommand connection and is nothing to be alarmed about. You can get rid of it by adding LogLevel Quiet
to your bastion host config stanza.
Solution 2
PIPES!
If the internet is a series of tubes, Unix is a series of pipes -- something like:
cat ginormous-file | ssh user@host1 "cat | ssh user@host2 \"cat >out\" "
should work.
If you need to traverse more hosts, add more pipes (and more nested layers of \
-escaped quotation) as needed. (Note however that if the pipeline/escaping gets so complex that you have to draw a diagram or resort to counting on your fingers to determine how many times you have to double up on escapes it's probably time to admit defeat and set up a proper VPN!)
Solution 3
OpenSSH v7.3 onward supports -J
.
-J [user@]host[:port]
Connect to the target host by first making a ssh connection to the jump host and then establishing a TCP forwarding to the ultimate destination from there. Multiple jump hops may be specified separated by comma characters. This is a shortcut to specify a ProxyJump configuration directive.
For A → B → C, on A, just:
tar -cf - file1 file_n | pv | ssh -C -J userB@B:portB userB@C -p portC 'tar -C destDir -xvf -'
- Use as many hops as you want with ssh's
-J
option. - Omit the tar's remote
-C
to leave the files on home folder. - Send any files at once (text or binary).
- Increase speed by compressing the stream with or ssh's
-C
(or tar's-z
). Particularly useful if the data is plain text (uncompressed). -
pv
monitor the progress of data through a pipe. An alternative could beprogress
.
Inspired on Florian Fida and Dan Garthwaite's answers.
Solution 4
If I understand correctly, you have two jump servers (jump-qa and jump-dev) protecting two app servers (app-qa and app-dev); the jump servers can ssh to each other; no box other than the relevant jump server can ssh to the corresponding app server. The app servers can ssh to noone. A file is to be transferred from app-dev to app-qa. Both jump servers lack the space for an interim copy of the data.
You can solve this with ssh tunneling. We set up a connection to one remote app server, carrying a remote tunnel that connects back to an unused port on its jump server. We set up a second connection from one jump server to the other jump server, carrying a tunnel that picks up the dangling end of the remotely-forwarded port from tunnel one and sends it on to the other app server's ssh port.
Set up the tunnels (each of these commmands will need to be run in a separate window on jump-qa
):
jump-qa% ssh app-qa -R 2345:localhost:2346
jump-qa% ssh jump-dev -L 2346:app-dev:22
You should now find that on app-qa, you can do telnet localhost 2345
and get app-dev's ssh banner. You may then copy the datafile:
app-qa% scp -P 2345 localhost:/path/on/app-dev/data.dat data.dat
Related videos on Youtube
Mohit Gupta
Web developer extraordinaire, programming pirate, all around nice guy! I am a software engineer and architect at Volkswagen Group of America. I write cool software and deal with special under-the-radar projects. I have been programming for something like 20+ years. I started in your typical languages, and moving into the web-space during the height of the browser wars. I have written code in these languages and many more: Basic (Yay basic!) Turbo Pascal Fortran COBOL ColdFusion HTML (I have run into a few people that argue it is a language. Although my argument is always that it's a markup language, not logic - but it is argued, so it makes the list) JavaScript (because.) PHP (Keep your jokes to yourself, everyone makes the leap somewhere) Java Python Perl C++ (It was college) ASM (never again) That my non-exhaustive list of languages that I remember working with. There are probably more. So aside from programming languages, I have also worked with the following dialects, scripting, markups, whatever: XML/XSLT HTML4,5/XHTML TypeScript SQL [MySQL, Oracle, MSSQL] CSS (Yay!) JavaScript Frameworks: React, Angular JavaScript Frontend Libraries: jQuery, Bootstrap, and a bunch of ones that showed up in the early to mid 2000's.
Updated on September 18, 2022Comments
-
Mohit Gupta over 1 year
We have two main environments in question:
Development and QA
Each environment has two servers:
- Jump Box
- Application server
In order to connect to the application server, you must connect to the jump box first, and then SSH to the Application server.
There are a few rules in place courtesy of the firewall:
- You MUST connect to the application server via the jump box
- The application server cannot connect to either jump boxes
- The jump boxes are on the same subnet, and CAN talk to each other.
Our Problem
We have a lot of content (670 GB) on the
DEVELOPMENT APPLICATION SERVER
, and we need to get this to theQA APPLICATION SERVER
.Copying this data to the jump boxes is not an option because they lack the required amount of space.
I did some research, and learned that we could potentially do a series of tunnels through these servers so that we can stream the data straight from one app server to the other via the tunnels. However, the problem that we cannot connect to the jump box from the application server.
Do we have any options? This is getting to be a desperate situation, and time is of the essence. We do not have time to download the data and re-upload it. Copying across the network on the servers will go quickly, as it is a gigabit connection.
-
Alex_www over 10 yearsOnce connection is established, you may copy either way: $devel_host $ tar -cf - | ssh -t jumbox 'ssh app_serv "tar -xf -" ' or other way around tar.
-
Mohit Gupta over 10 yearsso we can connect from dev jump box to dev app server, but not the other way. If we have that connection established, we can copy either way? what is the best way to move this content to the other app server without first saving it on the jump boxes?
-
Alex_www over 10 yearsExactly, "piping" "tar" is a simplest solution.
-
MadHatter over 10 yearsDoes client have space for an interim copy?
-
Mohit Gupta over 10 yearsNo, there is not enough space
-
Mohit Gupta over 10 yearswhen you say client, what server are you referring to?
-
MadHatter over 10 yearsI had misunderstood. My current understanding is that there is no client; there are two jump servers, and two app servers.
-
Mohit Gupta over 10 yearsThe first step works, however, when we try to ssh to the jump-dev via -L 2346:app-dev:22, it times out.
-
MadHatter over 10 yearsYou said you can ssh from jump-qa to jump-dev. Is this not so? I also don't get what you mean by "via -L 2346:app-dev:22". Please make sure you have read and understood what I said about doing these two commands from two separate windows on
jump-qa
. They are not a pair of commands to be typed into the same window in sequence. -
Mohit Gupta over 10 years
jump-qa% ssh jump-dev -L 2346:app-dev:22
- when we execute that command (with our parameters) it times out. However, we are getting: Warning: remote port forwarding failed for listen port xxxx -
MadHatter over 10 yearsWhat happens if you just do
jump-qa% ssh jump-dev
? -
Mohit Gupta over 10 yearsYou are right, it is failing from QA -> DEV but not from DEV -> QA. Is there a way we can operate within that?
-
MadHatter over 10 yearsYes, the situation is symmetric. Change all the
dev
toqa
, and vice-versa. The final copy will also become local to remote, instead of remote to local. -
Mohit Gupta over 10 yearsDoes it matter that ONLY port 22 is open between the jump servers? We are getting: Destination network unreachable
-
MadHatter over 10 yearsFrom where, doing what?
-
MadHatter over 10 yearsYou have assured me that dev jump box can ssh to qa jump box. Is this not so?
-
MadHatter over 10 yearsI have five more minutes, then I have to go. Let us continue this discussion in chat
-
voretaq7 over 10 yearsAs Alex_www pointed out in his comment if you don't need the intermediate file for anything you can also just pipe the output of
tar
around. (Also in you don't need thecat
in the intermediate stages of the pipeline -ssh
is happy to eat stdin and relay it. Thecat
just makes me feel better and is a placeholder for other useful commands you might want to use, liketee
.) -
MadHatter over 10 yearsI think the OP's problem is there is no machine that both has the data and can make the connections, nor is there any one machine that can see everything and everyone (to act as connector for all the pipes).
-
voretaq7 over 10 years@MadHatter If that's the case a combination of our two answers would work (connect to the dev server, forwarding a port back to the jump server's SSH port, then run the SSH pipeline over that port). This is of course making the solution increasingly disgusting to the point where people will start protesting outside your office with big signs that say
VPN! NOW!
on them... -
MadHatter over 10 yearsWhy, yes. Yes, they would. Hopefully!
-
Mohit Gupta over 10 yearsCheers, this solved our problem. Merry Christmas!
-
Andrew Wolfe over 8 yearsWhat do you mean by "bastion host config stanza" ?? @BraveNewCurrency ?
-
BraveNewCurrency over 8 yearsI consider each "Host" line and the configs under it to be a stanza (like in poetry.) If you add "LogLevel Quiet" under your bastion "Host" line, it will only apply to that host.
-
hello_there_andy over 7 yearsWill doing this mean the intermediate server (in this case
user@host1
) will at some point have the fullcat ginormous-file
in storage at any point? Or is the data just sent directly touser@host2
? Or is it somehow streamed? How istar
relevant to this? I guess it's relevant to the second to last question I asked. None of these questions are rhetorical btw... -
hello_there_andy over 7 yearsIn my case, the intermediate
user@host1
server ran out of storage whenscp
'ing from from theuser@host2
. And I can'tscp
directly fromuser@host2
touser@local
(I need to do twoscp
s:user@host2 --> user@host1 --> user@local
, each one manually). But I don't have a clue what myuser@local
's IP address is because it's a guest ubuntu VM in a Win10 host... I canssh
fromuser@local
-->user@host1
-->user@host2
, just not the other way (i.e.user@host1
-->user@local
I can't do, since I do not know user@local's IP address, working on it...) -
Mohit Gupta over 3 yearsThis project is long gone, but this solution will undoubtedly serve me (and anyone else) very well in the future. Thank you for the addition! This is great to know.
-
Federal Reserve about 2 yearsI found this answer in 2022, It's god damn good, how can I didn't know this before.