Scala – Converting String RDD to Int RDD

apache-sparkscala

I am new to scala..I want to know when processing large datasets with scala in spark is it possible to read as int RDD instead of String RDD

I tried the below:

val intArr = sc
              .textFile("Downloads/data/train.csv")
              .map(line=>line.split(","))
              .map(_.toInt)

But I am getting the error:

error: value toInt is not a member of Array[String]

I need to convert to int rdd because down the line i need to do the below

val vectors = intArr.map(p => Vectors.dense(p))

which requires the type to be integer

Any kind of help is truly appreciated..thanks in advance

Best Answer

As far as I understood, one line should create one vector, so it should goes like:

val result = sc
           .textFile("Downloads/data/train.csv")
           .map(line => line.split(","))
           .map(numbers => Vectors.dense(numbers.map(_.toInt)))

numbers.map(_.toInt) will map every element of array to int, so result type will be Array[Int]

Related Solutions

Scala – Implicit conversion from String to Int in scala 2.8

What is happening is that Java does not define a toInt method on String. In Scala, what defines that method is the class StringOps (Scala 2.8) or RichString (Scala 2.7).

On the other hand, there is a method toInt available on Int as well (through another implicit, perhaps?), so the compiler doesn't know if it is to convert the string to StringOps, through the defined implicit, or to Int, through your own implicit.

To solve it, call the implicit explicitly.

object Imp {
  implicit def string2Int(s: String): Int = augmentString(s).toInt

  def f(i: Int) = i

  def main(args: Array[String]) {
    val n: Int = f("666")
  }
}

Scala – How to find max value in pair RDD

Use Array.maxBy method:

val a = Array(("a",1), ("b",2), ("c",1), ("d",3))
val maxKey = a.maxBy(_._2)
// maxKey: (String, Int) = (d,3)

or RDD.max:

val maxKey2 = rdd.max()(new Ordering[Tuple2[String, Int]]() {
  override def compare(x: (String, Int), y: (String, Int)): Int = 
      Ordering[Int].compare(x._2, y._2)
})

Best Answer

Related Solutions

Scala – Implicit conversion from String to Int in scala 2.8

Scala – How to find max value in pair RDD

Related Topic