Package pyspark

source code

PySpark is the Python API for Spark.

Public classes:

• SparkContext Main entry point for Spark functionality.
• RDD A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
• Broadcast A broadcast variable that gets reused across tasks.
• Accumulator An "add-only" shared variable that tasks can only add values to.
• SparkConf For configuring Spark.
• SparkFiles Access files shipped with jobs.
• StorageLevel Finer-grained cache persistence levels.

Spark SQL:

• SQLContext Main entry point for SQL functionality.
• SchemaRDD A Resilient Distributed Dataset (RDD) with Schema information for the data contained. In addition to normal RDD operations, SchemaRDDs also support SQL.
• Row A Row of data returned by a Spark SQL query.

Hive:

• HiveContext Main entry point for accessing data stored in Apache Hive..