Reference: Spark Session vs Spark Context

Description

With the release of Spark 2.0.0 there is a new abstraction available to developers - the Spark Session - which can be instantiated and called upon just like the Spark Context that was previously available. The Spark Session encapsulates the existing Spark Context therefore existing functionality should not be affected and developers may continue using the Spark Context as desired. However the new Spark Session abstraction is preferred by the Spark community in Spark 2.0.0 on beyond therefore Qubole will be implementing this in the future.

Description

Users should feel free to use whichever option best suits their need when using Spark 2.0.0 and above - Spark Session or Spark Context. For the status of Spark Session development within the Qubole product please contact the support team.

Additional Reading

  • http://blog.madhukaraphatak.com/introduction-to-spark-two-part-1/
Have more questions? Submit a request

Comments

Powered by Zendesk