Reference: Spark Session vs Spark Context


With the release of Spark 2.0.0 there is a new abstraction available to developers - the Spark Session - which can be instantiated and called upon just like the Spark Context that was previously available. The Spark Session encapsulates the existing Spark Context therefore existing functionality should not be affected and developers may continue using the Spark Context as desired. However the new Spark Session abstraction is preferred by the Spark community in Spark 2.0.0 on beyond therefore Qubole will be implementing this in the future.


Users should feel free to use whichever option best suits their need when using Spark 2.0.0 and above - Spark Session or Spark Context. For the status of Spark Session development within the Qubole product please contact the support team.

Additional Reading

Have more questions? Submit a request


Powered by Zendesk