pandera.api.pyspark.components.Column

class pandera.api.pyspark.components.Column(dtype=None, checks=None, nullable=False, coerce=False, required=True, name=None, regex=False, title=None, description=None, metadata=None)

Validate types and properties of DataFrame columns.

Create column validator object.

Parameters:

• dtype (Union[str, int, float, bool, type, DataType, Type, BooleanType, StringType, IntegerType, DecimalType, FloatType, DateType, TimestampType, DoubleType, ShortType, ByteType, LongType, BinaryType, None]) – datatype of the column. The datatype for type-checking a dataframe. If a string is specified, then assumes one of the valid pyspark string values: https://spark.apache.org/docs/latest/sql-ref-datatypes.html

• checks (Union[Check, List[Check], None]) – checks to verify validity of the column

• nullable (bool) – Whether or not column can contain null values.

• coerce (bool) – If True, when schema.validate is called the column will be coerced into the specified dtype. This has no effect on columns where dtype=None.

• required (bool) – Whether or not column is allowed to be missing

• name (Union[str, Tuple[str, …], None]) – column name in dataframe to validate.

• regex (bool) – whether the name attribute should be treated as a regex pattern to apply to multiple columns in a dataframe.

• title (Optional[str]) – A human-readable label for the column.

• description (Optional[str]) – An arbitrary textual description of the column.

• metadata (Optional[dict]) – An optional key value data.

Raises:

SchemaInitError – if impossible to build schema from parameters

Example:

>>> import pyspark as ps
>>> from pyspark.sql import SparkSession
>>> import pandera.pyspark as pa
>>>
>>>
>>> schema = pa.DataFrameSchema({
...     "column": pa.Column(str)
... })
>>> spark = SparkSession.builder.getOrCreate()
>>> schema.validate(spark.createDataFrame([{"column": "foo"},{ "column":"bar"}])).show()
    +------+
    |column|
    +------+
    |   foo|
    |   bar|
    +------+

See here for more usage details.

Attributes

BACKEND_REGISTRY
dtype	Get the pyspark dtype
properties	Get column properties.

Methods

__init__	Create column validator object.
get_regex_columns	Get matching column names based on regex column name pattern.
set_name	Used to set or modify the name of a column object.
validate	Validate a Column in a DataFrame object.
__call__	Alias for validate method.