pyspark – IT Nursery

Spark performance for Scala vs Python

May 31, 2022 by IT Nursery

I prefer Python over Scala. But, as Spark is natively written in Scala, I was expecting my code to run faster in the Scala than the Python version for obvious reasons. With that assumption, I thought to learn & write the Scala version of some very common preprocessing code for some 1 GB of data. Data … Read more

How to change dataframe column names in pyspark?

May 18, 2022 by IT Nursery

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same doesn’t work in pyspark dataframes created using sqlContext. The only solution I could figure out to do this … Read more