How do I perform Web Scraping in Android?

Question

I want to scrape my website and then use the data from the website to populate elements in my app, my website has login pages and certain pages only open after the login has been done.

I started working with HtmlUnit as it is a headless browser and completed the custom api in a java IDE, later i tried to use the jar i generated from the java IDE and found that there are incompatibility issues with HtmlUnit and Android.

Can anyone propose a solution to this problem?

Edit : Since no one actually answered this question I am currently going with a work around using android's native WebView, settings its Visibility to invisible and then using javascript interfacing to a Java object, I can inject JS code to scrape any data.

Solution 1

Use Jsoup library for such purpose. Very handy and easy to use. Start with this answer and follow documents and other examples.

Solution 2

• Either you contribute to HtmlUnit to produce a version of HtmlUnit not using the missing dependencies from Android.
• Or you can use an alternative method like this one, as this seems to be the path someone else go before you.

If a real headless browser able to manage any recent web features, would exist, it would mean a team would have developed it and then invest much effort in it (in supporting existing and coming features) consistently.

Apart from Opera, Chrome, IE, and Firefox browsers, there is no such team. I would point out Chromium (CEF) as the most open and actively supported cross language wise. Try Cef for java