AWS S3 Writer

This component allows you to efficiently write your database data into Amazon S3 in Parquet format. For more background details, please refer to the README. Before proceeding, make sure you have completed the getting started instructions.

Quick Start Guide

info

OLake UI is live (beta)! You can now use the UI to configure your AWS S3 Destination, manage and configure various catalogs. Check it out at OLake UI regarding how to setup using Docker Compose and running it locally.

Use OLake UI for AWS S3

Create an S3 Destination in OLake UI

Follow the steps below to get started with S3 Destination using the OLake UI (assuming the OLake UI is running locally on localhost:8000):

1. Navigate to Destinations Tab.
2. Click on + Create Destination.
3. Select AWS S3 as the Destination type from Connector drop down.
4. Fill in the required connection details in the form. For details regarding the connection details, refer to the S3 Destination Configuration docs section on the right side of UI.
5. Click on Create ->
6. OLake will test the destination connection and display the results. If the connection is successful, you will see a success message. If there are any issues, OLake will provide error messages to help you troubleshoot.

This will create a S3 destination in OLake, now you can use this destination in your Jobs Pipeline to sync data from any Database to AWS S3.

Edit S3 Destination in OLake UI

To edit an existing S3 destination in OLake UI, follow these steps:

1. Navigate to the Destinations Tab.
2. Locate the S3 destination you want to edit from Active destination or Inactive destination tabs or using the search destination bar.
3. Click on the Edit button next to the Destinations from the Actions tab (3 dots).
4. Update the connection details as needed in the form and Click on Save Changes.

caution

Editing a destination can break pipeline.

You will see a notification saying "Due to editing, the jobs are going to get affected".

Editing this destination will affect the following jobs that are associated with this destination and as a result will fail immediately. Do you still want to edit the destination?

5. OLake will test the updated destination connection once you hit confirm on the destination Editing Caution Modal. If the connection is successful, you will see a success message. If there are any issues, we will provide error messages to help you troubleshoot.

Jobs Associated with S3 Destination

In the Destination Edit page, you can see the list of jobs that are associated with this destination. You can also see the status of each job, whether it is running, failed, or completed and can pause the job from the same screen as well.

Delete S3 Destination in OLake UI

To delete an existing S3 Destination in OLake UI, follow these steps:

1. Navigate to the Destination Tab.
2. Locate the destination you want to delete from Active Destinations or Inactive Destinations tabs or using the search destination bar.
3. Click on the Delete button next to the destinations from the Actions tab (3 dots).

4. A confirmation dialog will appear asking you to confirm the deletion.
5. Click on Delete to confirm the deletion.

This will remove the S3 Destination from OLake.

note

You can also delete a Destination from the Destination Edit page by clicking on the Delete button at the bottom of the page.

Use OLake CLI for AWS S3

Its a simple 3 step process:

1. Create a source config file and lets name it source.json,
2. Create another config file named destination.json and
3. Run the discover and sync commands to fetch the schema and start syncing the data respectively.

info

1. source.json - holds the source database information like host, port, username, password, database name, etc.
2. destination.json - holds the iceberg destination configurations.

Now, depending upon from where (source) to where (destination) you would like to sync the data, you can choose the below configurations.

1. PostgreSQL to Iceberg | Postgres Source Config
2. MongoDB to Iceberg | MongoDB Source Config
3. MySQL to Iceberg | MySQL Source Config

Now that you have the source configuration set, lets move on to S3 destination configuration.

4. Run Sync Commands:
   To replicate data from the source database to S3, you need to run the sync commands. The sync command will read the data from the source database and write it to the S3.

   • Discover Command: <DISCOVER_COMMAND>
   • Sync Command: <SYNC_COMMAND>
   • Sync with State Command: <SYNC_WITH_STATE_COMMAND>

Refer to respective Database docs to use the command for discover schema and sync the data.

• MongoDB Discover and sync command
• Postgres Discover and sync command
• MySQL Discover and sync command

A sample disover & sync command would look like this:

docker run --pull=always  \
  -v /Users/USERNAME/Desktop/projects/OLAKE_DIRECTORY:/mnt/config \
  olakego/source-mongodb:latest \
  discover \
  --config /mnt/config/source.json

info

The olakego/source-mongodb is the OLake image for MongoDB source. You can replace it with the respective source image for PostgreSQL (source-postgres) or MySQL (source-mysql) or can build one locally.