How to efficiently check if a list of words is contained in a Spark Dataframe?
You should consider using pyspark sql module functions instead of writing a UDF
, there are several regexp
based functions:
First let's start with a more complete sample data frame:
df = sc.parallelize([["a","b","foo is tasty"],["12","34","blah blahhh"],["yeh","0","bar of yums"],
['haha', '1', 'foobar none'], ['hehe', '2', 'something bar else']])\
.toDF(["col1","col2","col_with_text"])
If you want to filter lines based on whether they contain one of the words in words_list
, you can use rlike
:
import pyspark.sql.functions as psf
words_list = ['foo','bar']
df.filter(psf.col('col_with_text').rlike('(^|\s)(' + '|'.join(words_list) + ')(\s|$)')).show()
+----+----+------------------+
|col1|col2| col_with_text|
+----+----+------------------+
| a| b| foo is tasty|
| yeh| 0| bar of yums|
|hehe| 2|something bar else|
+----+----+------------------+
If you want to extract the strings matching the regular expression, you can use regexp_extract
:
df.withColumn(
'extracted_word',
psf.regexp_extract('col_with_text', '(?=^|\s)(' + '|'.join(words_list) + ')(?=\s|$)', 0))\
.show()
+----+----+------------------+--------------+
|col1|col2| col_with_text|extracted_word|
+----+----+------------------+--------------+
| a| b| foo is tasty| foo|
| 12| 34| blah blahhh| |
| yeh| 0| bar of yums| bar|
|haha| 1| foobar none| |
|hehe| 2|something bar else| |
+----+----+------------------+--------------+
Kishintai
Updated on July 09, 2022Comments
-
Kishintai almost 2 years
Using PySpark dataframes I'm trying to do the following as efficiently as possible. I have a dataframe with a column which contains text and a list of words I want to filter rows by. So:
Dataframe would look like this
df: col1 col2 col_with_text a b foo is tasty 12 34 blah blahhh yeh 0 bar of yums
The list will be
list = [foo,bar]
And thus result will be:result: col1 col2 col_with_text a b foo yeh 0 bar
Afterwards not only identical string matching will be done but also tested for similarity by using SequenceMatcher or so. This is what I already tried:
def check_keywords(x): words_list = ['foo','bar'] for word in x if word == words_list[0] or word == words_list[1]: return x result = df.map(lambda x: check_keywords(x)).collect()
Unfortunately I was unsuccesfull, could someone help me out? Thanks in advance.
-
Be Chiller Too over 4 yearsHi, why are you using lookaheads in your regexps?