How to: Create a Temporaray table in Spark SQL

This article provides an example to create a temporary table for Spark SQL. 

Example)

-> import scala.math.random


//create a temp table records
-> case class Record(age: Int, married: Int)
-> val rdd = sc.parallelize((1 to 10000).map(i => Record((random*100).toInt, (random*2).toInt)))


//Assuming we are using hive on spark sql
-> val q = hiveContext.applySchema(rdd,rdd.schema)


// Any RDD containing case classes can be registered as a table. The schema of the table is automatically inferred using scala reflection.
-> q.registerTempTable("records")

->%sql

-> select age,count(*) from records where age <=${maxAge=30}group by age order by age

Have more questions? Submit a request

Comments

Powered by Zendesk