Mount HTTP Server As File System

Question

I have a machine on which I wish to mount multiple remote servers to access them all centrally. For remote Linux based systems I am using SSHFS which works fine. But for Windows systems, or systems without SSH, they all have some form of HTTP server installed sharing the files (so they all have directory browsing enabled).

Can I mount a HTTP server as a local file system like SSHFS, so I can have all these remote servers mounted locally and presented in a uniform manner?

Solution 1

You can do this using WebDAV. This is an HTTP extension that is supported by most web servers, including IIS and Apache. WebDAV can be mounted in linux via the davfs2 FUSE module.

Solution 2

The problem with plain HTTP is that its just for retrieving content. There's no concept of folder or file list.

Some web servers let you browse directory structures by generating nice HTML pages with links to files for directories. And at least things like lftp are able to parse some of the common formats those indexes are generated as and give you the impression that it is a file transfer protocol, but it's not really, and I don't know of any fuse file systems that can parse indexes the way lftp does.

DAV is an extension that just does that and is already covered by jordanm's answer.

But note that if you don't need to list directories, avfs at least can let you access web pages over the filesystem.

$ mkdir AVFS
$ avfsd AVFS
$ grep -w Reputation 'AVFS/#http:unix.stackexchange.com|users|12583|javano'
 <span class="count">731</span> Reputation

Now also note that you can install sshd (via cygwin) or a FTP or Apache with DAV on a Windows machine as well, so you should be able to do the same things as you do on Unix machines.

Solution 3

the main issue already covered above is: directory/tree listing.

Although your question is quite old and you might have done otherwise, I have recently written a fuse filesystem that does exactly what you need, provided you write a little script to give it the "tree" of URLs.

It is there : https://gitlab.com/BylonAkila/astreamfs

You have an example script for the provider 1fichier, which you can adapt to your own personal server. What the script does is only prepare all the arguments for astreamfs, so that the mount will show all your files and folders in the right place. It does so with the (in)famous "web scraping" technique, aka reading HTML pages and extracting relevant parts from that.

There are still two major constaints, obviously, compared to regular mounts like NFS or SSHFS:

• The http server needs to handle 'ranges' (most servers can do that)
• It is read-only!

Comments

[goldilocks]

Someone out there might have written a tool to simulate this but I doubt it. The way that HTTP servers typically present the information is not very conducive to it.

[Dave C]

As above. Also the HTTP browsing is just that - it browses and reads, doesn't support uploading. I would look at either sharing the folder(s) to mount either via SMB/CIFS or FTP. Alternatively look at WebDAV and I'm sure you can find a IIS implementation.

[goldilocks]

Live and learn. :)

[jwbensley]

@DaveC Well typically directory browsing doesn't support uploading, however I failed to mention that I wanted read only access. Although HTTP it's self does support uploading with the PUT method.

[jwbensley]

@HermannIngjaldsson what has that got to do with anything? :S As stated, I already have HTTP servers :)

[Dave C]

@javano Yes HTTP PUT is the standard for uploading but you'd need something server-side to receive the file and complete the upload (which the browser doesn't do). But as you're after read only...

[amphibient]

wouldn't the path portion of an HTTP URL be considered as corresponding to a directory structure under the root? also, many web servers offer a file listing if you request a URL ending with a directory, not a file. and then both of these two concepts could be translated into this model of mounting an HTTP server as a file system. now, as far as posting changes to such a mount, I am not sure about that but perhaps some web servers could be configured to do it much like FTP

[Stéphane Chazelas]

@amphibient, yes, what you call a file listing is what I call a generated HTML page with links to file. It still needs to be parsed and there's no standard on the format of those pages. On Apache alone there are several modules generating those indexes which can be tuned in many different ways. So there's no foolproof solution. As I said see lftp that tries to parse many of those listing pages.

[Admin]

There's also cadaver on Linux that can do that.