Parquet local writer

info

Please read the getting started instructions before proceeding with this section.

Using Dockerized OLake

destination.json

{
  "type": "PARQUET",
  "writer": {
    "local_path": "/mnt/config"
  }
}

Build OLake

destination.json

{
  "type": "PARQUET",
  "writer": {
    "local_path": "./mnt/config"
  }
}

Key	Data Type	Example Value	Description & Possible Values
type	string	"PARQUET"	Specifies the output file format. Currently, only the Parquet format is supported.
writer.local_path	string	"./mnt/config"	The directory in user's local machine where Parquet files will be stored.
writer.max_file_size_mb	number	512	(Optional) Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset. Accepts a positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB).

Note: This configuration enables the Parquet local writer. For more details, check out the README section.

Data Partitioning

Refer to S3 docs for more details.