How to show full column content in a Spark Dataframe?
I am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content: val df … Read more
apache-spark
What is the difference between map and flatMap and a good use case for each?
Can someone explain to me the difference between map and flatMap and what is a good use case for each? What does “flatten … Read more
Difference between DataFrame, Dataset, and RDD in Spark
I’m just wondering what is the difference between an RDD and DataFrame (Spark 2.0.0 DataFrame is a mere type alias for Dataset[Row]) in … Read more
Spark – repartition() vs coalesce()
According to Learning Spark Keep in mind that repartitioning your data is a fairly expensive operation. Spark also has an optimized version of … Read more