Version History

1.8.1

• Drastically improve performance by switching to a column based read-scheme and implementing a custom row group buffering scheme
• Improve initialization performance by using re2 for regex matching

1.8.0

• Added option to exclude substream elements from the export based on a regular expression

1.7.3

• Fix NaN values not counting towards total count of values.

1.7.2

• Add validation for valid_element_count exceeding array_size
• Ignore case sensitivity in license check
• Use licme 0.6.0

1.7.1

• Add support for valid_element_count for optional non-array elements

1.7.0

• Set default filename extension to .parquet
• Use ADTF File Library 0.19

1.6.2

• Improve logging in case of errors

1.6.1

• Fix meta_data column of map file not containing timestamps

1.6.0

• Take valid_element_count into account to only store valid array elements in the parquet files
• Defer creation of parquet files until data needs to be written to them, preventing file creation for empty streams
• Generate plugin documentation from plugin meta information
• Fix plugin not throwing a license error, but failing silently

1.5.3

• Fix handling of emtpy structs

1.5.2

• Support all formats of PODs that are supported by DDL as well
• Fix array element names in map file

1.5.1

• Use cmake_toolchain_helper 1.1.1
• Use dw_cmake_utilities 1.0.0

1.5.0

• Split invalid_characters property into invalid_characters_filename and invalid_characters_columnname

1.4.0

• Add thread pool to process writing to multiple parquet files concurrently
• Processing time optimization of parquet exporting
• Add property for invalid characters
• Add adtfenvironment file to announce adtffileplugins and documentation for usage in ADTF Configuration Editor
• Use adtf_file_library 0.13.0
• Use dev_essential 1.3.4
• Use arrow 15.0.0

1.3.0

• Reported spec version has been bumped to 2.5
• Use adtf_file_library 0.12.0
• Use dev_essential 1.3.2

1.2.0

• Remove splitting of parquet files by column count

1.1.0

• Use zstd as default compression method (for both map and data files)
• Use 1MB data and dictionary page size for map files
• Use a single row group for the whole map file

1.0.0

• Initial version for exporting a set of substreams to parquet
• Split large substreams into parquet files named after their top level elements
• Store git commit SHA in file meta data
• Add md5 chesum to map file
• Option to use snappy and zlib for compression
• Use adtf file library 0.10.2
• Use dev_essential 1.2.1
• Use arrow 14.0.0

Known Issues / Restrictions

• none