Skip to main content

Permissions Required

1. AWS (Amazon Web Services) IAM Policy​

To successfully sync your source data with S3, ensure that your AWS IAM policy grants at least minimal permissions for listing, creating, reading, and writing to the S3 bucket. Below is a sample IAM policy.

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3BucketReadWrite",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucket*",
                "s3:*Object"
            ],
            "Resource": [
                "arn:aws:s3:::{S3_BUCKET_NAME}",
                "arn:aws:s3:::{S3_BUCKET_NAME}/*"
            ]
        },
        {
            "Sid": "ListAllBuckets",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

2. GCS (Google Cloud Storage) IAM Policy​

To successfully sync source data using GCS, ensure that GCS IAM policy grants at least minimal permission for listing, creating, reading, and writing to S3 bucket.

Using GCloud CLI​

  • Create a service account in GCS. Install Google Cloud SDK (gcloud CLI). Initialize the SDK using gcloud init command.

  • Login to GCS account, use gcloud auth login command.

  • Set the preferred project ID, use gcloud config set project Project-ID. Replace 'Project-ID' with your actual project id.

  • Create minimal set of IAM role required for the sync to GCS bucket.

    bash
    gcloud iam roles create storage.customAccess \
    --project="$(gcloud config get-value project)" \
    --file=/user-preferred-directory/gcs_policy.yaml

  • Sample gcs_policy.yaml with minimal set of permissions required.

    gcs_policy.yaml
    title: Custom Storage Access
    description: Equivalent to AWS S3 policy (List/Get bucket, full object access, list all buckets)
    stage: GA
    includedPermissions:
    - storage.buckets.getIamPolicy
    - storage.buckets.get
    - storage.buckets.list
    - storage.objects.list
    - storage.objects.get
    - storage.objects.create
    - storage.objects.delete
    - storage.objects.update

  • Replace 'Project-ID' and 'gcs-based-service-account' with actual GCS project id and service account.

    gcs_rolebinding.json
    {
    "bindings": [
            {
                "role": "projects/Project-ID/roles/storage.customAccess",
                "members": [
                "serviceAccount:gcs-based-service-account.com"
                ]
            }
        ]
    }

  • Bind the role to service account.

    bash
    gcloud storage buckets set-iam-policy \
    gs://<bucket-name> \
    /user-preferred-directory/gcs_rolebinding.json

  • Create a GCS bucket and HMAC keys for the service account for whom the above role is defined.

Using GCP​

  • Create a project.

  • Creating service account - permissions can be given then and there as well.

    Google Cloud create service account form, with fields for name, description, and optional permissions.

  • Create a GCS bucket and HMAC keys for the service account

info

For more information on HMAC keys, refer to this GCP HMAC Keys.

3. Azure Blob Storage Permissions​

To sync source data to Azure, the identity or credential OLake Go uses must be able to list, read, write, and delete blobs in the target container. The container must already exist. Azure does not use an IAM policy document. Access depends on the authentication type.

1. Managed identity​

For a Virtual Machine managed identity or an AKS workload identity, assign the built-in Storage Blob Data Contributor role to that identity, scoped to the container. The Contributor role manages the storage account but does not grant access to blob data.

Using Azure CLI​

bash
az role assignment create \
  --assignee-object-id <PRINCIPAL_OBJECT_ID> \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT>/blobServices/default/containers/<CONTAINER>"

Using Azure Portal​

  • Go to Storage accounts → <STORAGE_ACCOUNT> → Containers → <CONTAINER>.
  • Open Access Control (IAM) → Add → Add role assignment.
  • Select Storage Blob Data Contributor, then select the identity.

To turn the identity on, follow Configure managed identities on Azure virtual machines or Use a Microsoft Entra Workload ID with Azure Kubernetes Service.

2. SAS token​

A SAS token scoped to the container needs Read, Add, Create, Write, Delete, and List (racwdl).

3. Shared key or Connection String​

A storage account key, used as Shared Key or as AccountKey in a connection string, grants access to every container in the storage account and cannot be scoped to one container. A connection string that contains SharedAccessSignature is a SAS and can be scoped to one container.