Java – map RDD to PairRDD in Scala

apache-sparkjavarddscala

I am trying to map RDD to pairRDD in scala, so I could use reduceByKey later. Here is what I did:

userRecords is of org.apache.spark.rdd.RDD[UserElement]

I try to create a pairRDD from userRecords like below:

val userPairs: PairRDDFunctions[String, UserElement] = userRecords.map { t =>
  val nameKey: String = t.getName()
  (nameKey, t)
}

However, I got the error:

type mismatch; found : org.apache.spark.rdd.RDD[(String, com.mypackage.UserElement)]
required: org.apache.spark.rdd.PairRDDFunctions[String,com.mypackage.UserElement]

What am I missing here? Thanks a lot!

Best Answer

You don't need to do that as it is done via implicits (explicitly rddToPairRDDFunctions). Any RDD that is of type Tuple2[K,V] can automatically be used as a PairRDDFunctions. If you REALLY want to, you can explicitly do what the implicit does and wrap the RDD in a PairRDDFunction:

val pair = new PairRDDFunctions(rdd)

Related Solutions

Java – How to efficiently iterate over each entry in a Java Map

Map<String, String> map = ...
for (Map.Entry<String, String> entry : map.entrySet()) {
    System.out.println(entry.getKey() + "/" + entry.getValue());
}

On Java 10+:

for (var entry : map.entrySet()) {
    System.out.println(entry.getKey() + "/" + entry.getValue());
}

Java – Create instance of generic type in Java

You are correct. You can't do new E(). But you can change it to

private static class SomeContainer<E> {
    E createContents(Class<E> clazz) {
        return clazz.newInstance();
    }
}

It's a pain. But it works. Wrapping it in the factory pattern makes it a little more tolerable.

Best Answer

Related Solutions

Java – How to efficiently iterate over each entry in a Java Map

Java – Create instance of generic type in Java

Related Topic