Verify if a URL exists

Question

I would like to verify if a URL exists without downloading. I am using below with curl:

if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if [[ $(wget ftp://ftp.somewhere.com/bigfile.gz) -O-]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

This works great if the URL doesn't exist. If it exists, it downloads the file. In my case, the files are really big and I do not want it to download. I just want to know if that URL exists.

Solution 1

You're close. The correct way to approach this is using the HEAD method.

With cURL:

if curl --head --silent --fail ftp://ftp.somewhere.com/bigfile.gz 2> /dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if wget -q --method=HEAD ftp://ftp.somewhere.com/bigfile.gz;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

Solution 2

I think, it is better to use the --spider argument with the wget tool. It also works on lightweight versions of the wget tool (in BusyBox).

Example:

URL="http://www.google.com"

if wget --spider "${URL}" 2>/dev/null; then
    echo "OK !"
else
    echo "ERROR ! Online URL ${URL} not found !"
fi

Solution 3

Try curl --head (HTTP FTP FILE) Fetch the headers only!

status=$(curl --head --silent ftp://ftp.somewhere.com/bigfile.gz | head -n 1)

if echo "$status" | grep -q 404
  echo "file does not exist"
else
  echo "file exists"
fi

Solution 4

if curl --output /dev/null --silent --head --fail "ftp://ftp.somewhere.com/bigfile.gz"
then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

View more solutions

Comments

[Admin]

If you are trying to save disk space, try redirecting output of the command to /dev/null.

[Admin]

See also: Stack Overflow: How to check if an URL exists with the shell and probably curl?

[user5359531]

what version of wget did you use for this? I get the error: wget: unrecognized option '--method=HEAD'

[darnir]

--method=HEAD was added in Wget 1.15 released over 5 years ago.

[user5359531]

thanks looks like we have version 1.14 on our CentOS 7 server (Build Date: 2016-06-03), that would explain why we dont have this feature

[darnir]

You could use the --spider option in that case.