Processors

parquet

Parquet Processor to export all substreams of a stream into separate apache parquet files.

Properties

Name	Default Value	Type	Description
adtfdat_file		string	The source ADTFDAT File. If specified and `session_meta_data` is empty, the description of the file will be interpreted and stored as a json-string within the `session-metadata` property.
compression	ZSTD	string	Selects the used compression format. The availability of these formats depends on the used arrow library.s
data_page_size	8192	unsigned	The data page size.
dictionary_encoding	true	bool	Whether or not to enable dictionary encoding.
dictionary_page_size	1048576	unsigned	The dictionary page size.
exclude_regex	.__[^.]*$	string	All elements matching the given regular expression will be ommited from the export.
ignore_unsupported_streams	false	bool	When `true`, unsupported streams are ignored.
invalid_characters_columnname	[]	string	Characters that should be replaced in parquet column names.
invalid_characters_filename	:[]	string	Characters that should be replaced in parquet filenames.
min_row_group_row_count	1000	unsigned	The minimum row count of a row group in bytes.
min_row_group_size	1038336	unsigned	The minimum size of a row group in bytes.
min_row_group_timespan_ns	2000000000	unsigned	The minimum timespan of a row group in nanoseconds.
session_meta_data		string	If specified the json string will be interpreted and stored within the `session-metadata` property.
thread_count	0	signed	Number of threads to use.
timestamp_mode	mb	string	Internal: When set to 'mb' three timestamp columns (`timestamp`, `gps_utc_timestamp` and `mb_mt_timestamp`) will be created. The first two will contain only NULLs and the last one the sample timestamps.

Plugin Attributes

Attribute	Value
Plugin Name	parquet_exporter.adtffileplugin
License	ADTF Subscription
Support Mail	suppo.nosp@m.rt@d.nosp@m.igita.nosp@m.lwer.nosp@m.k.net
Homepage URL	https://adtf.dev/

Usage

To use the supplied exporter, specify the plugin with the --plugin option of the adtf_dattool i.e.:

adtf_dattool --plugin /path/to/parquet_exporter/bin/parquet_exporter.adtffileplugin --export file_with_substreams.adtfdat --stream stream_with_substreams --processorid parquet --output data.parquet

Supported Types

The exporter currently supports the following stream types:

adtf/plaintype: single POD values
adtf/default: any type that contains a DDL based media description.
everything else is exported as a plain byte array.