Connect CSV / Parquet (S3) to Rig

    Rig reads the files in an S3 bucket or folder and loads every match into one table you name. Each sync reads only files that are new or changed, so a folder of daily exports builds up into a single table.

    Before you start

    • You need an AWS access key for an IAM user that can list and read the bucket. Read access is all Rig needs.
    • Know the bucket address and, if the files sit in a folder, its path, such as s3://my-bucket/exports/.
    • Files that go into one connection should share a layout. Use a second connection for a different kind of export.
    • Google Cloud Storage has its own card, so use that for gs:// buckets.

    Step 1: Create a read-only access key

    1. In the AWS console, open IAM and choose or create a user for Rig.
    2. Give that user a policy that allows s3:ListBucket on the bucket and s3:GetObject on its objects.
    3. Open the user's security credentials, create an access key and copy the access key ID and the secret access key. AWS shows the secret once.

    Step 2: Connect it in Rig

    Adding a source needs a workspace owner, or a role with the Manage connections permission, on a workspace where you manage your own connections. If the Connectors tab says “Data ingestion for this workspace is managed by Rig”, ask an owner or Rig support to add the source.

    1. In Rig, open Connections, choose the Connectors tab and go to Sources & actions.
    2. Search for CSV / Parquet (S3), open the card and click Connect.
    3. Bucket or directory URL: the bucket and folder, such as s3://my-bucket/exports/.
    4. File glob pattern: which files to read, such as *.csv, exports/*.parquet or **/*.jsonl. It defaults to *.csv.
    5. Destination table name: the table every matching file loads into.
    6. Primary key (optional): the column, or comma-separated columns, that identify a row. Set it now if rows can arrive twice, because adding it later leaves existing duplicates in place.
    7. Latest-version column (optional): with a primary key, the column that decides which copy wins when several versions of a row arrive together, such as updated_at.
    8. AWS access key ID (s3:// only): the access key ID you created.
    9. AWS secret access key (s3:// only): the matching secret access key.
    10. Click Continue. Rig checks the credentials and lists the streams it can reach, so choose the ones you want and click Connect.
    11. The source then syncs on its schedule, which is nightly unless you change it. To load data straight away, click Sync on its row; Test pulls a small sample first if you want to check the connection.

    What Rig pulls in

    CSV / Parquet (S3) lands in its own csv_filesystem schema in your warehouse:

    • One table: every file matching the glob loads into the table you named. CSV, Parquet and JSONL files load as rows, and a PDF loads as one row per page.

    Good to know

    • Rig keeps track of each file's modification date, so a re-sync reads only files that are new or changed since the last run.
    • Without a primary key, a file that is rewritten has its rows added again. Exports that write a new file each run load cleanly.
    • Rig parses files with DuckDB, whose CSV reader copes with byte-order marks, quoted commas and ragged trailing columns.
    • To disconnect CSV / Parquet (S3), click the bin icon on the source's row under Sources & actions. Only whoever connected the source or a workspace owner can do this, and data already synced stays in the warehouse.

    What people use this for

    • Turn a folder of exports into one table
    • Keep the latest version of each record
    • Join partner and vendor feeds to your data

    More ideas for using CSV / Parquet (S3)

    Was this guide helpful?