Recently spark version 2.1 was released and there is a significant difference between the 2 versions.
Spark 1.6 has DataFrame and SparkContext while 2.1 has Dataset and SparkSession.
Now the question arises how to write code so that both the versions of spark are supported.
Fortunately maven provides the feature of building your application with different profiles.
In this blog i will tell you guys how to make your application compatible with different spark versions.
Lets start by creating a empty maven project.
You can use the maven-archetype-quickstart for setting up your project.
Archetypes provide a basic template for your project and maven has a rich collection of these templates for all your needs.
Once the project setup is done we need to make 3 modules. Lets name them core, spark and spark2 and setting the artifact id of each module to their respective names.
View original post 557 more words