Eclipse Project with Scala Plugin, Maven and Spark

apache-sparkeclipsemavenscala

I have Eclipse Kepler, I have installed the Maven and Scala plugins. I create a new Maven project and add the dependency

groupId: org.apache.spark
artifactId: spark-core_2.10
version: 1.1.0

as per current doc at http://spark.apache.org/downloads.html, all is fine, the jars for Scala 2.10 are also added to the project. I then add the "Scala Nature" to the project, this adds Scala 2.11 and I end up with the following error

More than one scala library found in the build path (C:/Eclipse/eclipse-jee-kepler-SR2-win32-x86_64/plugins/org.scala-lang.scala-library_2.11.2.v20140721-095018-73fb460c1c.jar, C:/Users/fff/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar).
At least one has an incompatible version.
Please update the project build path so it contains only compatible scala libraries.

Is it possible to use Spark (from Maven) and Scala IDE Plugin together? Any ideas on how to fix this problem?

Thanks for your help. Regards

Best Answer

In short, yes, it is possible.

Spark is currently using Scala 2.10, and the latest Scala IDE is "cross published" for 2.10 and 2.11. You need to choose the 2.10-based version, which is 3.0.3.

However, the next major version, 4.0, which is in release candidate mode, has multi-version support. You can create a Scala project and select the Scala version you'd like to use (2.10 or 2.11). You could give that a try if you feel like it.

Related Solutions

Java – ‘Must Override a Superclass Method’ Errors after importing a project into Eclipse

Eclipse is defaulting to Java 1.5 and you have classes implementing interface methods (which in Java 1.6 can be annotated with @Override, but in Java 1.5 can only be applied to methods overriding a superclass method).

Go to your project/IDE preferences and set the Java compiler level to 1.6 and also make sure you select JRE 1.6 to execute your program from Eclipse.

Java – How to add local jar files to a Maven project

You can add local dependencies directly (as mentioned in build maven project with propriatery libraries included) like this:

<dependency>
    <groupId>com.sample</groupId>
    <artifactId>sample</artifactId>
    <version>1.0</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/src/main/resources/Name_Your_JAR.jar</systemPath>
</dependency>

Update

In new releases this feature is marked as deprecated but still working and not removed yet ( You just see warning in the log during maven start). An issue is raised at maven group about this https://issues.apache.org/jira/browse/MNG-6523 ( You can participate and describe why this feature is helpful in some cases). I hope this feature remains there!

If you are asking me, as long as the feature is not removed, I use this to make dependency to only one naughty jar file in my project which is not fit in repository. If this feature is removed, well, there are lots of good answers here which I can chose from later!

Best Answer

Related Solutions

Java – ‘Must Override a Superclass Method’ Errors after importing a project into Eclipse

Java – How to add local jar files to a Maven project

Related Topic