Connect Apify to Rig

    Sync your Apify actors, their runs and the scraped items each run produced into Rig. Every item is tagged with the run and actor that made it, so you can trace a row back to what scraped it and when.

    Before you start

    • You need an Apify account that can see the actors you want to sync.
    • Decide whether to sync every actor or only some. You can list actor IDs to limit the sync.

    Step 1: Copy your Apify API token

    1. Sign in to the Apify Console.
    2. Open Settings then API & Integrations.
    3. Copy your Personal API token.

    Step 2: Connect it in Rig

    Adding a source needs a workspace owner, or a role with the Manage connections permission, on a workspace where you manage your own connections. If the Connectors tab says “Data ingestion for this workspace is managed by Rig”, ask an owner or Rig support to add the source.

    1. In Rig, open Connections, choose the Connectors tab and go to Sources & actions.
    2. Search for Apify, open the card and click Connect.
    3. API token: paste your personal API token.
    4. Actor IDs: optional. Enter actor IDs separated by commas to sync only those, or leave it blank to sync every actor the token can see.
    5. Click Continue. Rig checks the credentials and lists the streams it can reach, so choose the ones you want and click Connect.
    6. The source then syncs on its schedule, which is nightly unless you change it. To load data straight away, click Sync on its row; Test pulls a small sample first if you want to check the connection.

    What Rig pulls in

    Apify lands in its own apify schema in your warehouse:

    • Actors: the actors visible to your token.
    • Runs: each run, with its actor, status and timing.
    • Dataset items: the rows each run scraped, tagged with the run and actor that produced them.

    Good to know

    • Each actor decides its own item columns, so items from different actors share a table without sharing every column. The run and actor tags tell you which columns belong to which actor.
    • Rig reads items through each run's dataset. Apify's dataset list only shows named datasets, and run datasets are unnamed.
    • A finished run's dataset does not change, so each nightly sync only fetches runs newer than the last one. Apify allows 60 requests a minute.
    • To disconnect Apify, click the bin icon on the source's row under Sources & actions. Only whoever connected the source or a workspace owner can do this, and data already synced stays in the warehouse.

    What people use this for

    • Use scraped data next to your own
    • Monitor scraping runs
    • Track how scraped values change over time

    More ideas for using Apify

    Apify documentation

    Was this guide helpful?