universal_transfer_operator.datasets.file.types.parquet

Module Contents

Classes

ParquetFileTypes

Concrete implementation to handle Parquet file type
class universal_transfer_operator.datasets.file.types.parquet.ParquetFileTypes(path, normalize_config=None)

Bases: universal_transfer_operator.datasets.file.types.base.FileTypes

Concrete implementation to handle Parquet file type

Parameters:

  • path (str) –

  • normalize_config (dict | None) –

property name

get file type

export_to_dataframe(stream, columns_names_capitalization='original', **kwargs)

Read parquet file from one of the supported locations and return dataframe.

Parameters:

  • stream – file stream object

  • columns_names_capitalization – determines whether to convert all columns to lowercase/uppercase in the resulting dataframe

create_from_dataframe(df, stream)

Write parquet file to one of the supported locations.

Parameters:

  • df (pandas.DataFrame) – pandas dataframe

  • stream (io.TextIOWrapper) – file stream object

Return type:

None