sql

Type Members

1. class AnalysisException extends Exception with Serializable

   

   :: DeveloperApi :: Thrown when a query fails to analyze, usually because the query itself is invalid.

   :: DeveloperApi :: Thrown when a query fails to analyze, usually because the query itself is invalid.

   Annotations

   @DeveloperApi()

2. class Column extends Logging

   

   A column that will be computed based on the data in a DataFrame.

   A column that will be computed based on the data in a DataFrame.

   A new column is constructed based on the input columns present in a dataframe:

   df("columnName")            // On a specific DataFrame.
   col("columnName")           // A generic column no yet associated with a DataFrame.
   col("columnName.field")     // Extracting a struct field
   col("`a.column.with.dots`") // Escape `.` in column names.
   $"columnName"               // Scala short hand for a named column.
   expr("a + 1")               // A column that is constructed from a parsed SQL Expression.
   lit("abc")                  // A column that produces a literal (constant) value.

   Column objects can be composed to form complex expressions:

   $"a" + 1
   $"a" === $"b"

   Since

   1.3.0

3. class ColumnName extends Column

   

   :: Experimental :: A convenient class used for constructing schema.

   :: Experimental :: A convenient class used for constructing schema.

   Annotations

   @Experimental()

   Since

   1.3.0

4. type DataFrame = Dataset[Row]

   

5. final class DataFrameNaFunctions extends AnyRef

   

   :: Experimental :: Functionality for working with missing data in DataFrames.

   :: Experimental :: Functionality for working with missing data in DataFrames.

   Annotations

   @Experimental()

   Since

   1.3.1

6. class DataFrameReader extends Logging

   

   Interface used to load a Dataset from external storage systems (e.g.

   Interface used to load a Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.read to access this.

   Since

   1.4.0

7. final class DataFrameStatFunctions extends AnyRef

   

   :: Experimental :: Statistic functions for DataFrames.

   :: Experimental :: Statistic functions for DataFrames.

   Annotations

   @Experimental()

   Since

   1.4.0

8. final class DataFrameWriter[T] extends AnyRef

   

   Interface used to write a Dataset to external storage systems (e.g.

   Interface used to write a Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.write to access this.

   Since

   1.4.0

9. class Dataset[T] extends Serializable

   

   A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations.

   A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each Dataset also has an untyped view called a DataFrame, which is a Dataset of Row.

   Operations available on Datasets are divided into transformations and actions. Transformations are the ones that produce new Datasets, and actions are the ones that trigger computation and return results. Example transformations include map, filter, select, and aggregate (groupBy). Example actions count, show, or writing data out to file systems.

   Datasets are "lazy", i.e. computations are only triggered when an action is invoked. Internally, a Dataset represents a logical plan that describes the computation required to produce the data. When an action is invoked, Spark's query optimizer optimizes the logical plan and generates a physical plan for efficient execution in a parallel and distributed manner. To explore the logical plan as well as optimized physical plan, use the explain function.

   To efficiently support domain-specific objects, an Encoder is required. The encoder maps the domain specific type T to Spark's internal type system. For example, given a class Person with two fields, name (string) and age (int), an encoder is used to tell Spark to generate code at runtime to serialize the Person object into a binary structure. This binary structure often has much lower memory footprint as well as are optimized for efficiency in data processing (e.g. in a columnar format). To understand the internal binary representation for data, use the schema function.

   There are typically two ways to create a Dataset. The most common way is by pointing Spark to some files on storage systems, using the read function available on a SparkSession.

   val people = spark.read.parquet("...").as[Person]  // Scala
   Dataset<Person> people = spark.read().parquet("...").as(Encoders.bean(Person.class)); // Java

   Datasets can also be created through transformations available on existing Datasets. For example, the following creates a new Dataset by applying a filter on the existing one:

   val names = people.map(_.name)  // in Scala; names is a Dataset[String]
   Dataset<String> names = people.map((Person p) -> p.name, Encoders.STRING)); // in Java 8

   Dataset operations can also be untyped, through various domain-specific-language (DSL) functions defined in: Dataset (this class), Column, and functions. These operations are very similar to the operations available in the data frame abstraction in R or Python.

   To select a column from the Dataset, use apply method in Scala and col in Java.

   val ageCol = people("age")  // in Scala
   Column ageCol = people.col("age"); // in Java

   Note that the Column type can also be manipulated through its various functions.

   // The following creates a new column that increases everybody's age by 10.
   people("age") + 10  // in Scala
   people.col("age").plus(10);  // in Java

   A more concrete example in Scala:

   // To create Dataset[Row] using SparkSession
   val people = spark.read.parquet("...")
   val department = spark.read.parquet("...")
   
   people.filter("age > 30")
     .join(department, people("deptId") === department("id"))
     .groupBy(department("name"), "gender")
     .agg(avg(people("salary")), max(people("age")))

   and in Java:

   // To create Dataset using SparkSession
   Dataset<Row> people = spark.read().parquet("...");
   Dataset<Row> department = spark.read().parquet("...");
   
   people.filter("age".gt(30))
     .join(department, people.col("deptId").equalTo(department("id")))
     .groupBy(department.col("name"), "gender")
     .agg(avg(people.col("salary")), max(people.col("age")));

   Since

   1.6.0

10. case class DatasetHolder[T] extends Product with Serializable

    

    A container for a Dataset, used for implicit conversions in Scala.

    A container for a Dataset, used for implicit conversions in Scala.

    To use this, import implicit conversions in SQL:

    import sqlContext.implicits._

    Since

    1.6.0

11. trait Encoder[T] extends Serializable

    

    :: Experimental :: Used to convert a JVM object of type T to and from the internal Spark SQL representation.

    :: Experimental :: Used to convert a JVM object of type T to and from the internal Spark SQL representation.

    Scala

    Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.

    import spark.implicits._
    
    val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)

    Java

    Encoders are specified by calling static methods on Encoders.

    List<String> data = Arrays.asList("abc", "abc", "xyz");
    Dataset<String> ds = context.createDataset(data, Encoders.STRING());

    Encoders can be composed into tuples:

    Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
    List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
    Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

    Or constructed from Java Beans:

    Encoders.bean(MyClass.class);

    Implementation

    • Encoders are not required to be thread-safe and thus they do not need to use locks to guard against concurrent access if they reuse internal buffers to improve performance.

    Annotations

    @Experimental() @implicitNotFound( ... )

    Since

    1.6.0

12. class ExperimentalMethods extends AnyRef

    

    :: Experimental :: Holder for experimental methods for the bravest.

    :: Experimental :: Holder for experimental methods for the bravest. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

    spark.experimental.extraStrategies += ...

    Annotations

    @Experimental()

    Since

    1.3.0

13. abstract class ForeachWriter[T] extends Serializable

    

    :: Experimental :: A class to consume data generated by a StreamingQuery.

    :: Experimental :: A class to consume data generated by a StreamingQuery. Typically this is used to send the generated data to external systems. Each partition will use a new deserialized instance, so you usually should do all the initialization (e.g. opening a connection or initiating a transaction) in the open method.

    Scala example:

    datasetOfString.write.foreach(new ForeachWriter[String] {
    
      def open(partitionId: Long, version: Long): Boolean = {
        // open connection
      }
    
      def process(record: String) = {
        // write string to connection
      }
    
      def close(errorOrNull: Throwable): Unit = {
        // close the connection
      }
    })

    Java example:

    datasetOfString.write().foreach(new ForeachWriter<String>() {
    
      @Override
      public boolean open(long partitionId, long version) {
        // open connection
      }
    
      @Override
      public void process(String value) {
        // write string to connection
      }
    
      @Override
      public void close(Throwable errorOrNull) {
        // close the connection
      }
    });

    Annotations

    @Experimental()

    Since

    2.0.0

14. class KeyValueGroupedDataset[K, V] extends Serializable

    

    :: Experimental :: A Dataset has been logically grouped by a user specified grouping key.

    :: Experimental :: A Dataset has been logically grouped by a user specified grouping key. Users should not construct a KeyValueGroupedDataset directly, but should instead call groupByKey on an existing Dataset.

    Annotations

    @Experimental()

    Since

    2.0.0

15. class RelationalGroupedDataset extends AnyRef

    

    A set of methods for aggregations on a DataFrame, created by Dataset.groupBy.

    A set of methods for aggregations on a DataFrame, created by Dataset.groupBy.

    The main method is the agg function, which has multiple variants. This class also contains convenience some first order statistics such as mean, sum for convenience.

    This class was named GroupedData in Spark 1.x.

    Since

    2.0.0

16. trait Row extends Serializable

    

    Represents one row of output from a relational operator.

    Represents one row of output from a relational operator. Allows both generic access by ordinal, which will incur boxing overhead for primitives, as well as native primitive access.

    It is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null.

    To create a new Row, use RowFactory.create() in Java or Row.apply() in Scala.

    A Row object can be constructed by providing field values. Example:

    import org.apache.spark.sql._
    
    // Create a Row from values.
    Row(value1, value2, value3, ...)
    // Create a Row from a Seq of values.
    Row.fromSeq(Seq(value1, value2, ...))

    A value of a row can be accessed through both generic access by ordinal, which will incur boxing overhead for primitives, as well as native primitive access. An example of generic access by ordinal:

    import org.apache.spark.sql._
    
    val row = Row(1, true, "a string", null)
    // row: Row = [1,true,a string,null]
    val firstValue = row(0)
    // firstValue: Any = 1
    val fourthValue = row(3)
    // fourthValue: Any = null

    For native primitive access, it is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null. An example of native primitive access:

    // using the row from the previous example.
    val firstValue = row.getInt(0)
    // firstValue: Int = 1
    val isNull = row.isNullAt(3)
    // isNull: Boolean = true

    In Scala, fields in a Row object can be extracted in a pattern match. Example:

    import org.apache.spark.sql._
    
    val pairs = sql("SELECT key, value FROM src").rdd.map {
      case Row(key: Int, value: String) =>
        key -> value
    }

17. class RowFactory extends AnyRef

    

18. class RuntimeConfig extends AnyRef

    

    Runtime configuration interface for Spark.

    Runtime configuration interface for Spark. To access this, use SparkSession.conf.

    Options set here are automatically propagated to the Hadoop configuration during I/O.

    Since

    2.0.0

19. class SQLContext extends Logging with Serializable

    

    The entry point for working with structured data (rows and columns) in Spark 1.x.

    The entry point for working with structured data (rows and columns) in Spark 1.x.

    As of Spark 2.0, this is replaced by SparkSession. However, we are keeping the class here for backward compatibility.

    Since

    1.0.0

20. abstract class SQLImplicits extends AnyRef

    

    A collection of implicit methods for converting common Scala objects into Datasets.

    A collection of implicit methods for converting common Scala objects into Datasets.

    Since

    1.6.0

21. final class SaveMode extends Enum[SaveMode]

    

22. class SparkSession extends Serializable with Logging

    

    The entry point to programming Spark with the Dataset and DataFrame API.

    The entry point to programming Spark with the Dataset and DataFrame API.

    In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:

    SparkSession.builder().getOrCreate()

    The builder can also be used to create a new session:

    SparkSession.builder()
      .master("local")
      .appName("Word Count")
      .config("spark.some.config.option", "some-value").
      .getOrCreate()

23. type Strategy = SparkStrategy

    

    Converts a logical plan into zero or more SparkPlans.

    Converts a logical plan into zero or more SparkPlans. This API is exposed for experimenting with the query planner and is not designed to be stable across spark releases. Developers writing libraries should instead consider using the stable APIs provided in org.apache.spark.sql.sources

    Annotations

    @DeveloperApi()

24. class TypedColumn[-T, U] extends Column

    

    A Column where an Encoder has been given for the expected input and return type.

    A Column where an Encoder has been given for the expected input and return type. To create a TypedColumn, use the as function on a Column.

    T

    The input type expected for this expression. Can be Any if the expression is type checked by the analyzer instead of the compiler (i.e. expr("sum(...)")).

    U

    The output type of this column.

    Since

    1.6.0

25. class UDFRegistration extends Logging

    

    Functions for registering user-defined functions.

    Functions for registering user-defined functions. Use SQLContext.udf to access this. Note that the user-defined functions must be deterministic.

    Since

    1.3.0