Fivetran Connector SDK allows Real-time, efficient data replication to your destination of choice.

Overview

This repository contains a collection of example custom connectors using Fivetran Connector SDK, demonstrating how to build custom data connectors in Python and deploy them as an extension of Fivetran.

Additionally, this repository provides guidance on using AI to help you code an SDK connector quickly.

Why Connector SDK?

Fivetran Connector SDK allows you to code a custom data connector using Python and deploy it as an extension of Fivetran. Fivetran automatically manages running Connector SDK connections on your scheduled frequency and manages the required compute resources, eliminating the need for a third-party provider.

Connector SDK provides native support for many Fivetran features and relies on existing Fivetran technology. It also eliminates timeout and data size limitations seen in AWS Lambda.

Requirements

• Python ≥3.9 and ≤3.12
• Operating System:
  • Windows 10 or later
  • MacOS 13 (Ventura) or later

Getting started

See Setup guide to get started.

Examples

There are several examples available under /examples:

Quickstart Examples

• hello

  • This is the simplest, append-only example.

• simple_three_step_cursor

  • This is an emulated source, without any calls out to the internet.

• configuration

  • This example shows how to use secrets.

• multiple_code_files

  • This example shows how you can write a complex connector comprising multiple .py files.

• using_pd_dataframes

  • This example shows the use of Pandas DataFrames to manipulate data prior to sending to Fivetran.

• large_data_set

  • This example shows how to handle large data from API responses with pagination and without pagination.

• weather

  • This is a realistic example, using a public API, fetching data from NOAA.

• complex_configuration_options

  • This example shows how to cast configuration fields to LIST, INTEGER, BOOLEAN, and DICT for use in connector code.

• base_64_encoding_decoding

  • This example shows how to use base64 encoding and decoding in your connector code.

Common Patterns for Connectors

• Authentication

  • api_key
    • This is a simple example of how to work with API Key authentication for a REST API.
  • certificate
    • using_base64_encoded_certificate
      • It is an example of using base64-encoded strings for certificate-based authentication. The script includes functions to decode the certificate and key and use them to authenticate API requests.
    • retrieve_from_aws
      • It is an example of how to retrieve the certificate from AWS S3 bucket and use it for certificate-based authentication.
  • http_basic
    • This is a simple example of how to work with HTTP BASIC authentication for a REST API.
  • http_bearer
    • This is a simple example of how to work with HTTP BEARER authentication for a REST API.
  • oauth2_with_token_refresh
    • It is an example of using OAuth 2.0 client credentials flow, and the refresh of Access token from the provided refresh token.
    • Refer to the OAuth Refresh flow in its readme.md.
  • session_token
    • This is a simple example of how to work with Session Token authentication for a REST API.

• Cursors

  • marketstack
    • This code retrieves different stock tickers and the daily price for those tickers using Marketstack API. Refer to Marketstack's documentation.
  • multiple_tables_with_cursors
    • The parent-child relationship between tables from incremental API endpoints, with the complex cursor.

• Export

  • csv
    • This is a simple example of how to work with .CSV file response for a REST API of export type.

• hashes

  • This example shows how to calculate a hash of fields to be used as primary key. This is useful in scenarios where the incoming rows do not have any field suitable to be used as a Primary Key.

• Pagination

  • keyset
    • This is a simple example of how to work with key-based pagination for a REST API.
  • next_page_url
    • This is a simple example for how to work with next-page-url pagination for a REST API.
  • offset_based
    • This is a simple example of how to work with offset-based pagination for a REST API.
  • page_number
    • This is a simple example for how to work with page-number-based pagination for a REST API.

• priority_first_sync_for_high_volume_initial_syncs

  • A priority-first sync (pfs), is very helpful for high-volume historical syncs. It is a sync strategy that prioritises fetching the most recent data first so that fresh data is ready for you to use more quickly. This is a simple example of how you could implement the priority-first sync strategy in a connector.py file for your connection.

• records_with_no_created_at_timestamp

  • This example shows how to work with records where the source does not provide a created_at (or equivalent) field. It is useful when it's desired to keep track of when the record was first observed.

• specified_types

  • This example declares a schema and upserts all data types.

• three_operations

  • This example shows how to use upsert, update and delete operations.

• unspecified_types

  • This example upserts all data types without specifying a schema.

• update_and_delete

  • This example shows how to handle composite primary keys while using update and delete operations with a PostgreSQL database as the data source.

Source Examples

• aws_athena/using_boto3

  • This is an example of how we can sync data from AWS Athena by using Connector SDK using Boto3.

• aws_athena/using_sqlalchemy

  • This is an example of how we can sync data from AWS Athena by using Connector SDK using SQLAlchemy with PyAthena.

• aws_dynamo_db_authentication

  • This is an example of how we can connect and sync data from AWS DynamoDB by using Connector SDK.

• Common Patterns

  • key_based_replication
    • This example shows key-based replication from database sources. Replication keys are columns that are used to identify new and updated data for replication. When you set a table to use Incremental Replication, you’ll also need to define a replication key for that table.

• gcp_pub_sub

  • This example shows how to sync data from Google Cloud Pub/Sub using the Connector SDK.

• hubspot

  • This example shows how to connect and sync specific event type Event data from Hubspot using Connector SDK

• newsapi

  • This is a simple example of how to sync data from NewsAPI using Connector SDK.

• oauth2_and_accelo_api_connector_multithreading_enabled

  • This example shows how to sync data from the Accelo API. It uses OAuth 2.0 Client Credentials flow authentication, rate limiting, and multithreading, which allows to make API calls in parallel to pull data faster. You need to provide your Accelo OAuth credentials for this example to work.
  • Refer to the Multithreading Guidelines in api_threading_utils.py.

• redshift

  • This is an example to show how to sync records from Redshift by using Connector SDK. You need to provide your Redshift credentials for this example to work.

• s3_csv_validation

  • This is an example of how to read .CSV file from Amazon S3 and validate the data. You need to provide your AWS S3 credentials for this example to work.

• smartsheets

  • This is an example of how we can sync Smartsheets sheets by using Connector SDK. You need to provide your Smartsheets api_key for this example to work.

• sql_server

  • This example uses pyodbc to connect to SQL Server for syncing data using Connector SDK. You need to provide your SQL Server credentials for this example to work.

• toast

  • This is an example of how we can sync Toast data using the Connector SDK. You would need to provide your Toast credentials for this example to work.

• veeva_vault_using_basic_auth

  • This example shows how to authenticate to Veeva Vault using basic authentication and sync records from Veeva Vault. You need to provide your Veeva Vault credentials for this example to work.

Example Workflows

• github
  • This is an example of a GitHub workflow to deploy a hello connector after a push to the hello directory. It uses GitHub secrets and environment variables to create a configuration.json file that is included in the deployment.

AI and Connector SDK

• Readme
  • This is an introduction to using AI tools to leverage Connector SDK
• claude_20250228
  • This example contains the code produced by Claude AI to build a custom connector using our Connector SDK. See our blog article for details.

Support

Learn how we support Fivetran Connector SDK.

Additional considerations

We provide examples to help you effectively use Fivetran's Connector SDK. While we've tested the code provided in these examples, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples.

Note that API calls made by your Connector SDK connection may count towards your service’s API call allocation. Exceeding this limit could trigger rate limits, potentially impacting other uses of the source API.

It's important to choose the right design pattern for your target API. Using an inappropriate pattern may lead to data integrity issues. We recommend that you review all our examples carefully to select the one that best suits your target API. Keep in mind that some APIs may not support patterns for which we currently have examples.

As with other new connectors, SDK connectors have a 14-day trial period during which your usage counts towards free MAR. After the 14-day trial period, your usage counts towards paid MAR. To avoid incurring charges, pause or delete any connections you created to run these examples before the trial ends.

Maintenance

This repository is actively maintained by Fivetran Developers. Reach out to our Support team for any inquiries.