How can I call arguments and return the outputs from a Google cloud hosted function?

Question

I want to write a python function that allows me to scrape a website given a url, that will interface with a flutter mobile application. I have written the very basic code on python, but now I have no clue whatsoever on how to integrate it Google Cloud. Would really appreciate some walkthrough help.

My code model is as follows:

import timeago
import pytz
from bs4 import BeautifulSoup
from time import mktime
from datetime import datetime

def Parser(url):
     ......# URL PARSING
     ......
     ......
     return domain_name, page_title, clean_txt, timestamp
url = '******************'
file = Parser(url)
print (file[0], file [1], file [2], file [3])

I have tried looking at many examples online, and most of them use flask. I do not understand what flask is used for to be honest, and how to use it for Google Cloud Functions (or any cloud functions for that matter). As far as I understand, I should put my requirements.txt file as follows:

timeago
pytz
mktime
bs4
datetime

If so, then how should I define my imports (i.e. the from bs4 import BeautifulSoup, etc) statements?

Also, how can I call these functions to my flutter app code? I have the url for the cloud function, but don't know what to do with it. I tried going through the official documentation, but most of it is just "Hello World" code that is barely of any use to understand (for me), due to the lack of any input or output arguments.

Thanks and cheers

Solution

To address some of your queries:

I do not understand what flask is used for to be honest, and how to use it for Google Cloud Functions (or any cloud functions for that matter)

There are a few methods to trigger a Cloud Function, one of the more common ones is using HTTP Triggers.

For HTTP Triggers, Flask is needed to wrap around your actual function/business logic with a RESTful endpoint, so that your function can be invoked via an HTTP request (The latter action should be performed by your flutter code). Some of this is actually handled by Cloud Functions, and you mainly need to specify a wrapper method taking in a flask.Request object (essentially the payload of the above-mentioned HTTP request) as an argument to invoke your main business logic/function, wherein the flask.Request object will be the point where your arguments (e.g. url) are provided. You will then need to provide this wrapper method when deploying the cloud function (see below).

If so, then how should I define my imports (i.e. the from bs4 import BeautifulSoup, etc) statements?

The import statements work as usual as per your code snippet.

how can I call these functions to my flutter app code?

You will need to fire an HTTP request from your flutter app code to the URL of the cloud function.

To put things into context:

In your example above, you can wrap it around an entry point method with a flask.Request object like this:

import timeago
import pytz
from bs4 import BeautifulSoup
from time import mktime
from datetime import datetime
from flask import escape


def Parser(url):
     ......# URL PARSING
     ......
     ......
     return domain_name, page_title, clean_txt, timestamp


def invoke_parser(request):
    request_json = request.get_json(silent=True)
    file = Parser(escape(request_json['url']))
    print (file[0], file [1], file [2], file [3])

How your function has changed is that url has to be passed in the HTTP request body instead when invoking the function.

So, to deploy the function, use the command:

gcloud functions deploy [CLOUD FUNCTION NAME] \
--entry-point invoke_parser \
--runtime python37 \
--trigger-http

The --entry-point argument will be the wrapper method to be invoked (I have used invoke_parser as the wrapper method in my example).

To invoke it, fire an HTTP request (this is where the url argument to your Parser function can be provided) e.g.:

curl -X POST \
"https://YOUR_REGION-YOUR_PROJECT_ID.cloudfunctions.net/FUNCTION_NAME" \
-H "Content-Type:application/json" \
--data '{"url":"******************"}'

In your flutter code, you will need to fire an HTTP request equivalent to the above curl command

Comments

[LuckyStrike]

Okay so nested functions are accepted in GCF, and I just need to specify the function to run at the bottom of the function setup page then (I'm using the UI to define the function)?

[chaooder]

Yup the Function to execute option in the UI corresponds to the --entry-point argument in the gcloud CLI. You will need to provide the "wrapper function" here (in my above example it is the invoke_parser function, which wraps around your Parser function).

[LuckyStrike]

Thanks. That really helps a lot. Not to mention I finally figured out the flask package thanks to you.

[LuckyStrike]

How should the imports be implemented then? Where do I specify the from..... import..... statements?

[chaooder]

You can specify them as usual at the top of the python code

[chaooder]

Do help to accept the answer if it works for you :)

[LuckyStrike]

Just another quick question. A lot of solutions, including the example by GCP, use the escape function. What's that used for, since you haven't not used it in your solution?

[chaooder]

It is to convert special characters to HTML safe sequences (see tedboy.github.io/flask/generated/flask.escape.html). If your url has special characters please go-ahead to try adding it (no harm adding too anw).

[LuckyStrike]

Thanks. It's still not working yet (a few issues to iron out), but I guess with your answers, I'm figuring out GCF after so much trouble! Thanks a lot for your help. I've marked it as the accepted answer since my questions were answered.