Connect Google Cloud Storage to Rig

    Give one email address read access to a bucket and Rig loads the files in it into your warehouse. Exports that land in Cloud Storage become a queryable table that grows as new files arrive, with no key to create or rotate.

    Requirements
    Permission to manage access on the bucket
    Time required
    5 mins

    How the connection works

    Rig provisions a service account for your workspace: a Google identity owned by Rig with no access to anything until you give it some. You grant it read access on one bucket, and Rig can then list and read the objects in that bucket and nothing else in your project.

    The grant sits in your own bucket permissions, so you can see it, audit it and remove it at any time without involving us. Removing it stops the next sync.

    Before you start

    • Permission to change the bucket's access, such as the Storage Admin role. If someone else administers the project, they need to make the grant.
    • A workspace owner, or a role with the Manage connections permission, in Rig. If the Connectors tab says “Data ingestion for this workspace is managed by Rig”, ask an owner or Rig support to add the source.
    • At least one file already in the bucket. Rig checks it can read a matching file when you connect, which is also what catches a mistyped bucket name.
    • Files that share a layout. Every file a connection matches lands in the same table, so point each connection at one kind of export.
    • Some organisations restrict which identities can be granted access to their resources. If yours does, an admin will need to allow the service-account address before the grant will save.

    Generate your Rig service account

    The service account is generated once for your whole workspace, not once per connector. If you have already generated one for another Google source, such as Drive or Sheets, reuse that same email address and skip to the next section.

    1. In Rig, open Connections and go to Sources & actions.
    2. In the Google service account card at the top of the page, click Generate service account.
    3. Copy the email address Rig gives you. It ends in .iam.gserviceaccount.com.

    Give Rig read access to the bucket

    1. In the Google Cloud console, open Cloud Storage → Buckets and click the bucket.
    2. Open the Permissions tab and click Grant access.
    3. Paste the service-account email as the new principal and give it the Storage Object Viewer role. Rig only ever reads.
    4. Save.

    Grant on the bucket, not the project
    Storage Object Viewer granted at project level would let Rig read every bucket in the project. Granting it on the bucket keeps Rig to the one you chose.

    Connect it in Rig

    1. In Sources & actions, pick Google Cloud Storage and stay on the Share to Rig (no OAuth) tab.
    2. Enter the Bucket URL, including any folder the files sit in, for example gs://acme-exports/finance/.
    3. Set the File glob pattern for the files you want (it defaults to *.csv), for example *.csv or orders_*.parquet. Use **/*.csv to include subfolders.
    4. Name the Destination table name the files will land in.
    5. Click I've shared it: check & connect. Rig lists the bucket with the service account and connects only if it can read a matching file.

    If your security team would rather use a service account of their own, the Manual / API key tab takes its JSON key in Service account key (JSON) instead. It needs the same Storage Object Viewer grant on the bucket.

    How files land in the warehouse

    Every file the pattern matches is loaded into the one destination table. A folder of daily exports with the same columns becomes a single table holding every day, rather than a table per file.

    Each sync reads only the files that are new or changed since the last one, so a growing folder costs the same to sync tomorrow as it does today. CSV, Parquet, JSONL and Excel files load as rows, and PDFs, Word documents and text files load as searchable text.

    Add new files rather than overwriting old ones
    A file rewritten in place is read again in full and its rows are added a second time. Exports that write a new file each run, such as a date in the filename, load cleanly. If rows do repeat across files, fill in Primary key (optional) when you connect, so a row delivered again replaces its earlier version. Adding it later does not remove duplicates already loaded.

    Troubleshooting

    • "Nothing is shared to … yet": the grant did not land, or it is on a different bucket. Check the bucket's Permissions tab lists the service-account address with Storage Object Viewer.
    • No files match: the bucket URL or the pattern is off, or the bucket is still empty. Check the bucket name for typos, and that the pattern matches a file that is already there.
    • Duplicate rows: a file is being overwritten in place. Write each export to a new file instead, or set a primary key when you connect.

    What people use this for

    • Turn a folder of scheduled exports into one table
    • Join partner and vendor feeds to warehouse data
    • Make archived files queryable without a pipeline

    More ideas for using Google Cloud Storage

    Was this guide helpful?