pandera.api.pyspark.model.DataFrameModel¶

class pandera.api.pyspark.model.DataFrameModel(*args, **kwargs)[source]¶

Definition of a DataFrameSchema.

new in 0.16.0

See the User Guide for more.

Check if all columns in a dataframe have a column in the Schema.

Parameters:

check_obj – DataFrame object i.e. the dataframe to be validated.
head – Not used since spark has no concept of head or tail
tail – Not used since spark has no concept of head or tail
sample – validate a random sample of n% rows. Value ranges from 0-1, for example 10% rows can be sampled using setting value as 0.1. refer below documentation. https://spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.DataFrame.sample.html
random_state – random seed for the sample argument.
lazy – if True, lazily evaluates dataframe against all validation checks and raises a SchemaErrors. Otherwise, raise SchemaError as soon as one occurs.
inplace – if True, applies coercion to the object of validation, otherwise creates a copy of the data.

Returns:

validated DataFrame

Raises:

SchemaError – when DataFrame violates built-in or custom checks.

Example:

Calling schema.validate returns the dataframe.

>>> import pandera.pyspark as psa
>>> from pyspark.sql import SparkSession
>>> import pyspark.sql.types as T
>>> spark = SparkSession.builder.getOrCreate()
>>> data = [("Bread", 9), ("Butter", 15)]
>>> spark_schema = T.StructType(
...     [
...         T.StructField("product", T.StringType(), False),
...         T.StructField("price", T.IntegerType(), False),
...     ],
... )
>>> df = spark.createDataFrame(data=data, schema=spark_schema)
>>> schema_with_checks = psa.DataFrameSchema(
...     columns={
...         "product": psa.Column("str", checks=psa.Check.str_startswith("B")),
...         "price": psa.Column("int", checks=psa.Check.gt(5)),
...     },
...     name="product_schema",
...     description="schema for product info",
...     title="ProductSchema",
... )
>>> schema_with_checks.validate(df).take(2)

Methods

classmethod build_schema_(**kwargs)[source]¶

classmethod to_ddl()[source]¶

Recover fields of DataFrameModel as a PySpark DDL string.

Return type:: str
Returns:: String with current model fields, in compact DDL format.

classmethod to_structtype()[source]¶

Recover fields of DataFrameModel as a PySpark StructType object.

Return type:: StructType
Returns:: StructType object with current model fields.

classmethod validate(check_obj, head=None, tail=None, sample=None, random_state=None, lazy=True, inplace=False)[source]¶

Validate a DataFrame based on the schema specification.