pyspark.sql.streaming.DataStreamReader.table#

DataStreamReader.table(tableName)[source]#

Define a Streaming DataFrame on a Table. The DataSource corresponding to the table should support streaming mode.

New in version 3.1.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters

tableNamestr: string, for the name of the table.

Returns

DataFrame

Notes

This API is evolving.

Examples

Load a data stream from a table.

>>> import tempfile
>>> import time
>>> _ = spark.sql("DROP TABLE IF EXISTS my_table")
>>> with tempfile.TemporaryDirectory(prefix="table") as d:
...     # Create a table with Rate source.
...     q1 = spark.readStream.format("rate").load().writeStream.toTable(
...         "my_table", checkpointLocation=d)
...
...     # Read the table back and print out in the console.
...     q2 = spark.readStream.table("my_table").writeStream.format("console").start()
...     time.sleep(3)
...     q1.stop()
...     q2.stop()
...     _ = spark.sql("DROP TABLE my_table")