Command line URL fetch with JavaScript capabliity
you can use PhantomJS http://phantomjs.org
You can use it as below :
var page=require("webpage");
page.open("http://monster.com",function(status){
page.evaluate(function(){
/* your javascript code here
$.ajax("....",function(result){
phantom.exit(0);
}); */
});
});
un33k
Val Neekman is a Solutions Architect with a passion for Typescript-Angular, Python-Django, UX and Performant Software. He is the Principal Consultant at Neekware Inc.
Updated on June 23, 2022Comments
-
un33k almost 2 years
I use curl, in php and httplib2 in python to fetch URL.
However, there are some pages that use JavaScript (AJAX) to retrieve the data after you have loaded the page and they just overwrite a specific section of the page afterward.
So, is there any command line utility that can handle JavaScript?
To know what I mean go to: monster.com and try searching for a job.
You'll see that the Ajax is getting the list of jobs afterward. So, if I wanted to pull in the jobs based on my keyword search, I would get the page with no jobs.
But via browser it works.