Flutter WebView: Retrieve ALL fully generated html from a rendered URL(Similar to "Inspect Element")

Question

I am trying to get the titles of the events from this webpage and put them into a list:

Since using HTTP did not get the fully generated html, I decided to try using flutter_webview_plugin to:

1. Load the desired URL on a hidden webview
2. Extract the fully generated HTML
3. Parse the HTML to get the titles I need

I am using this code to load the webview:

flutterWebviewPlugin.launch(
    "https://www.volunteermatch.org/search/?l=Long%20Grove,%20IL%2060047,%20USA",
    hidden: true);
flutterWebviewPlugin.onStateChanged.listen((viewState) async {
  if (viewState.type == WebViewState.finishLoad) {
    flutterWebviewPlugin
        .evalJavascript("document.documentElement.outerHTML")
        .then((html) {
      print(html);
      //some code to parse html
      flutterWebviewPlugin.close();
    });
  }
});

The problem is that even using JavaScript code to retrieve the HTML after the webview has finished loading does not give me the same HTML that going to "Inspect Element" on Safari would give, and none of the titles are in it either. What am I missing?

Solution

Well, now I feel silly.

Since I was running the app on my iPhone, I was using Xcode to check the output of print(html). For some reason, Xcode doesn't print out all of the html string:

Xcode just leaves a <...>(bottom right corner circled in red) and decides not to print the rest.

Comments

[Nilesh Senta]

It's look like you are scraping the web page. please check below post itnext.io/write-your-first-web-scraper-in-dart-243c7bb4d05

[Xcoder]

@NileshSenta Thanks for the response, I'll look into the link.

[Saifallak]

any luck using this ? i'm still getting the first html like http.get() not the full html i see in webview

[Xcoder]

@Saifallak Basically my issue was just an Xcode print issue, and the code I used actually did get all of the html.