wget how to download file from a web page that prompts you to click on "Agree" condition

Question

I'd like to download a file using the wget utility. I followed the instruction on "How do I use wget/curl to download from a site I am logged into?" and the download process works; however, the saved cookies expires after a certain time, so I'm unable to continue downloading the file the following day.

Here's the URL I'm trying to download the a file from:

https://frbservices.org/EPaymentsDirectory/FedACHdir.txt

The download page requires that I click the "Agree" button before I can proceed with the download.

Is there a way to include the "Agree" submission with the wget utility?

Thank you.

Solution

You can submit a form using wget, using the --post-data option. First, look at the form that page uses:

<form name="acceptedForm" id="acceptedForm" action="submitAgreement" method="post">
...
        <button id="agree_terms_use" name="agreementValue" type="submit" value="Agree">Agree</button>  &nbsp; 
        <button id="disagree_terms_use" name="agreementValue" type="submit" value="Do Not Agree">Do Not Agree</button>

Typically, the value of the action attribute is used to get the target URL, which becomes https://frbservices.org/EPaymentsDirectory/submitAgreement. The names of the form elements become the parameters. Then you need to save the cookies that you get from accepting the agreement. From this SO post, we can construct the command needed:

wget --post-data="agreementValue=Agree" https://frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies cookie.txt --keep-session-cookies --delete-after

Then, we can use these cookies to download the file:

wget --load-cookies=cookie.txt 'https://frbservices.org/EPaymentsDirectory/FedACHdir.txt?AgreementSessionObject=Agree'

Comments

[A.B.]

+1 And is this also possible with aria2c? ;)

[muru]

@A.B. I don't think aria2c has command line options to submit POST data. That will have to be done by wget or curl. Once that is done, you can use the cookie file with aria2c.

[Adam]

Thank you Muru! The steps you provided answered my question.