Java – NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less

apache-sparkjavamavenscala

I have seen many answers related to this error, but all re-directing to scala versions etc. But I think my case is different.

I have a remote spark master-worker cluster set up with version 2.10. I was able to verify it through http://master-ip:8080 listing all worker nodes

From my application, I am trying to create SparkConf with Java 7 code. Following below is the code

sparkConf = new SparkConf(true)
                .set("spark.cassandra.connection.host", "localhost")
                .set("spark.cassandra.auth.username", "username")
                .set("spark.cassandra.auth.password", "pwd")
                .set("spark.master", "spark://master-ip:7077")
                .set("spark.app.name","Test App");

Following are the maven dependencies i added

<dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.10</artifactId>
            <version>2.0.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.1.0</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.validation</groupId>
                    <artifactId>validation-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

I get the below error

Caused by: java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
    at org.apache.spark.util.Utils$.getSystemProperties(Utils.scala:1710)
    at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:73)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:68)

Spark Version from one of the worker nodes

./spark-shell --version
Welcome to
     ____              __
    / __/__  ___ _____/ /__
   _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
     /_/

Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_91
Branch 
Compiled by user jenkins on 2016-12-16T02:04:48Z
Revision 
Url 
Type --help for more information.

Best Answer

It is related to Scala version.

Your cluster has Scala 2.10, but Spark dependency is

spark-core_2.11

which means Scala 2.11

Change it to 2.10 and will work

Related Solutions

Java – :: (double colon) operator in Java 8

Usually, one would call the reduce method using Math.max(int, int) as follows:

reduce(new IntBinaryOperator() {
    int applyAsInt(int left, int right) {
        return Math.max(left, right);
    }
});

That requires a lot of syntax for just calling Math.max. That's where lambda expressions come into play. Since Java 8 it is allowed to do the same thing in a much shorter way:

reduce((int left, int right) -> Math.max(left, right));

How does this work? The java compiler "detects", that you want to implement a method that accepts two ints and returns one int. This is equivalent to the formal parameters of the one and only method of interface IntBinaryOperator (the parameter of method reduce you want to call). So the compiler does the rest for you - it just assumes you want to implement IntBinaryOperator.

But as Math.max(int, int) itself fulfills the formal requirements of IntBinaryOperator, it can be used directly. Because Java 7 does not have any syntax that allows a method itself to be passed as an argument (you can only pass method results, but never method references), the :: syntax was introduced in Java 8 to reference methods:

reduce(Math::max);

Note that this will be interpreted by the compiler, not by the JVM at runtime! Although it produces different bytecodes for all three code snippets, they are semantically equal, so the last two can be considered to be short (and probably more efficient) versions of the IntBinaryOperator implementation above!

Scala – Cannot deploy local Spark job, worker fails with EndPointAssociationError

The reason you are getting the error is because Spark doesn't have spark-cassandra-connector on worker's classpath.

You can either use one of one-jar sbt plugins like sbt-assembly to create assembled jar including all dependencies. And use it to submit your job to Spark:

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  <assembled jar> \
  [application-arguments]

Or you can use Spark's Runtime Environment option spark.executor.extraClassPath and set classpath for your application globally.

Best Answer

Related Solutions

Java – :: (double colon) operator in Java 8

Scala – Cannot deploy local Spark job, worker fails with EndPointAssociationError

Related Topic