How to change case of whole column to lowercase?
Solution 1
I Got it (use Functions#lower
, see Javadoc)
import org.apache.spark.sql.functions.lower
String columnName="Category name";
src=src.withColumn(columnName, lower(col(columnName)));
src.show();
This replaced old column with new one retaining the whole Dataset.
+------+--------------------+
|ItemID| Category name|
+------+--------------------+
| ABC|brush & broom han...|
| XYZ|wheel brush parts...|
+------+--------------------+
Solution 2
Use lower
function from org.apache.spark.sql.functions
For instance:
df.select($"q1Content", lower($"q1Content")).show
The output.
+--------------------+--------------------+
| q1Content| lower(q1Content)|
+--------------------+--------------------+
|What is the step ...|what is the step ...|
|What is the story...|what is the story...|
|How can I increas...|how can i increas...|
|Why am I mentally...|why am i mentally...|
|Which one dissolv...|which one dissolv...|
|Astrology: I am a...|astrology: i am a...|
| Should I buy tiago?| should i buy tiago?|
|How can I be a go...|how can i be a go...|
|When do you use ...|when do you use ...|
|Motorola (company...|motorola (company...|
|Method to find se...|method to find se...|
|How do I read and...|how do i read and...|
|What can make Phy...|what can make phy...|
|What was your fir...|what was your fir...|
|What are the laws...|what are the laws...|
|What would a Trum...|what would a trum...|
|What does manipul...|what does manipul...|
|Why do girls want...|why do girls want...|
|Why are so many Q...|why are so many q...|
|Which is the best...|which is the best...|
+--------------------+--------------------+
Solution 3
first you should add the library by
import static org.apache.spark.sql.functions.lower;
then you need to put the lower
method at the right spot. here is an example:
.and(lower(df1.col("field_name")).equalTo("offeringname"))
I've read all answers here and then tried it myself, for some reason i was stuck with IntelliJ Idea for couple of minutes until I could make it understand (library wise). If you faced this glitch, just add the library by recommendations of IntelliJ as it will pop-up when something is unknown.
Good luck.
Shreeharsha
I like to encounter problems. I love solving problems I learn by implementing Linux is cool to me knows coding in Spark, java, J2ee, PHP,
Updated on August 11, 2020Comments
-
Shreeharsha almost 4 years
I want to Change case of whole column to Lowercase in Spark Dataset
Desired Input +------+--------------------+ |ItemID| Category name| +------+--------------------+ | ABC|BRUSH & BROOM HAN...| | XYZ|WHEEL BRUSH PARTS...| +------+--------------------+ Desired Output +------+--------------------+ |ItemID| Category name| +------+--------------------+ | ABC|brush & broom han...| | XYZ|wheel brush parts...| +------+--------------------+
I tried with
collectAsList()
andtoString()
, which is slow and complex procedure for very large dataset.I also found a method 'lower' but didnt get to know how to get it work in dasaset Please suggest me a simple or effective way to do the above. Thanks in advance