Java – Spark Java scala error

apache-sparkhadoopjavamavenscala

Hey I want to use spark in my Java Project :

I already add this dependency to my pom file :

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.4.0</version>
</dependency>

I tried this code :

import org.apache.spark.api.java.JavaSparkContext;

public class sparkSQL {
    public void query() {
        JavaSparkContext sc = new JavaSparkContext();
    }
}

I called this function in my main but I got this error :

Exception in thread "main" java.lang.NoClassDefFoundError: scala/Cloneable
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.SparkContext.(SparkContext.scala:111)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:56)
at realtimequeries.sparkSQL.query(sparkSQL.java:7)
at main.main(main.java:25)
Blockquote

Caused by: java.lang.ClassNotFoundException: scala.Cloneable
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
… 16 more

Blockquote

I don't understand why I got this error, because normally JavaSparkContext has been created for this utilisation :

A Java-friendly version of SparkContext that returns JavaRDDs and works with Java collections instead of Scala ones.

I already take a look at the pom of the spark-core_2.11 dependency I had and it's seems that we can find a scala dependency :

http://central.maven.org/maven2/org/apache/spark/spark-hive_2.10/1.4.0/spark-hive_2.10-1.4.0.pom

Did I miss something ? What I am doing wrong ?
Thanks in advance

Best Answer

The class scala.Cloneable is present in scala-library*.jar. This error is gone for me after adding scala-library to pom.xml

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.11.1</version>
</dependency>
Related Topic