How to get list of network requests done by HTML

Question

How can i get the list of network requests using Javascript done by the HTML, as seen in the chrome devtools.

For example:

This is the devtools for google.com. I want to, using javascript get all these request in a list. is this possible? if yes how?

Solution 1

Some browsers have implemented a version of the not-yet-standard, Resource Timing API where you can collect some of this information.

Some browsers may have some of this info available to browser extensions as part of their developer tools support, but that would require the installation of a custom extension, not something that could be done from a regular web page.

For very specific operations where you control the calling code or you know the calling code, it is possible to instrument some things. For example, if you know that all ajax calls go through one particular function, you can hook that function and it's completion handlers and monitor all ajax calls.

Solution 2

As I understand it, you can consult the list of requests via JavaScript. It is? "I do not know how."

But one solution that can help is this ...

You intercept all requisitions with the code below. If your JavaScript runs very early in loading the page you will be able to get most of the requests from the list.

See how cool this article.

XMLHttpRequest.prototype.realSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = function(value) {
    this.addEventListener("progress", function(){
        console.log("Loading. Here you can intercept...");
    }, false);
    this.realSend(value);
};

Solution 3

You can use Resource Timing API to get all relevant information (domain lookups, cache hits, redirects, etc.) about each resource being loaded on your website.

You can read about it here. There is also a bookmarklet that generates a page load waterfall using this API.

Resource Timing API is available in Chrome, Chromium, Chrome Mobile and IE10. Firefox team seems to be working on it.

Solution 4

I have written the code using the Resource Timing API

function captureNetworkRequest(e) {
    var capture_network_request = [];
    var capture_resource = performance.getEntriesByType("resource");
    for (var i = 0; i < capture_resource.length; i++) {
        if (capture_resource[i].initiatorType == "xmlhttprequest" || capture_resource[i].initiatorType == "script" || capture_resource[i].initiatorType == "img") {
            if (capture_resource[i].name.indexOf('www.demo.com OR YOUR URL') > -1) {
                capture_network_request.push(capture_resource[i].name)
            }
        }
    }
    return capture_network_request;
}

Solution 5

You could get the URLs of requests to be made when the page loads but retrieving any sort of statistics on load times is unrealistic. Query elements which make these kind of resource requests such as script, link or img.

For example:

var urls = Array.prototype.map.call(
    document.querySelectorAll("link, img, script, iframe"), // Elements which request external resources
    function(e) { // Loop over and return their href/src
        return e.href || e.src; 
    }
);

View more solutions

Comments

[Jasper]

Is this code part of a website or a Chrome Extension?

[Nachshon Schwartz]

It's part of a website, the reference to the chrome devtools was simply to explain what kind of info i want to get in the javascript...

[dandavis]

you can get some of them from the DOM, like images and css images, and css sheet urls, iframes, flash, etc, others like ajax are unreachable from js.

[epascarello]

You can get some of it, but not all of it.

[jfriend00]

Why the downvote? What is incorrect in this answer? Or what would make it a better answer?

[AdrianCooney]

Well, you'd of course have to repeat the requests if you wanted any real statistics about them.

[Matt]

This is correct. Any sort of inspection at this level would have to be done in an extension (at least in the case of Chrome).

[Nachshon Schwartz]

I meant redirects done by the elemnts themselves... like if the source of an image is a page that redirects to the image

[JohnnyJS]

I find your answer more useful than just saying, "It can't be done". Thanks.

[oldboy]

is it possible to get all the files on the network page? if youre creating a browser extension?

[jfriend00]

@oldboy - I have no idea. One would have to study the extension API to see what you get access to.

[oldboy]

still no way to do it without a browser extension? even with something like node?

[jfriend00]

@oldboy - It depends upon what you're really looking for. You can use node.js and get a parser like cheerio and parse the HTML to discover all the references to resources that are in the page. Or, you could even use something like puppeteer to run the page (including Javascript) and then examine the content. You won't be access the Chrome network page from node. Perhaps you should fully describe your problem and what you're trying to achieve in a new question?

[oldboy]

im looking to actually obtain all of the individual files that are served to the client. i cant even open a webpage within, say, an electron app's window, and then obtain (via somescript.js) the files that are served thereto? i created a question here. please feel free to share your thoughts and knowledge!! <3

[Xoundboy]

Wouldn't it be necessary to remove the event listener once the request ended?

[Sergio Cabral]

I understand that it is not necessary. Because after reaching the final status, the instance does not trigger any more events.

[Ethan O'Brien]

Use this.realSend.apply(this, arguments); to call with all arguments provided

[Vadorequest]

I'm not really interested in the extension, my interest is about watching over network requests, from my own extension or from an internal script. See stackoverflow.com/questions/72585423/… for more details