📝 Refactor connector breaking changes documentation (#68098)
This commit is contained in:
Aaron ("AJ") Steers
committed by
GitHub
GitHub
parent
5b3de4f456
commit
e09ee0e04f
1
.github/pr-welcome-community.md
vendored
1
.github/pr-welcome-community.md
vendored
@@ -5,6 +5,7 @@ Thank you for your contribution from **{{ .repo_name }}**! We're excited to have
|
||||
### Helpful Resources
|
||||
|
||||
- [PR Guidelines](https://docs.airbyte.com/contributing-to-airbyte): Check our guidelines for contributions.
|
||||
- [Breaking Changes Guide](https://docs.airbyte.com/platform/connector-development/connector-breaking-changes): Guide to breaking changes, migration guides, and upgrade deadlines.
|
||||
- [Developing Connectors Locally](https://docs.airbyte.com/platform/connector-development/local-connector-development): Learn how to set up your environment and develop connectors locally.
|
||||
- If you enable [BYO Connector Credentials](https://docs.airbyte.com/platform/connector-development/local-connector-development#managing-connector-secrets) within your fork, you can view your test results [here](https://github.com/{{ .repo_name }}/actions/workflows/run-connector-tests-command.yml):
|
||||
[](https://github.com/{{ .repo_name }}/actions/workflows/run-connector-tests-command.yml)
|
||||
|
||||
1
.github/pr-welcome-internal.md
vendored
1
.github/pr-welcome-internal.md
vendored
@@ -4,6 +4,7 @@ Here are some helpful tips and reminders for your convenience.
|
||||
|
||||
### Helpful Resources
|
||||
|
||||
- [Breaking Changes Guide](https://docs.airbyte.com/platform/connector-development/connector-breaking-changes) - Breaking changes, migration guides, and upgrade deadlines
|
||||
- [Developing Connectors Locally](https://docs.airbyte.com/platform/connector-development/local-connector-development)
|
||||
- [Managing Connector Secrets](https://docs.airbyte.com/platform/connector-development/local-connector-development#managing-connector-secrets)
|
||||
- [On-Demand Live Tests](https://github.com/airbytehq/airbyte/actions/workflows/live_tests.yml)
|
||||
|
||||
@@ -16,7 +16,7 @@ If you need support along the way, visit the [Slack channel](https://airbytehq.s
|
||||
|
||||
1. **Pick the technology and build**. The first step in creating a new connector is to choose the tools you’ll use to build it. For _most_ cases, you should start in Connector Builder.
|
||||
2. **Publish as a custom connector**. After building and testing your connector, you’ll need to publish it. This makes it available in your workspace. At that point, you can use the connector you’ve built to move some data!
|
||||
3. **Contribute back to Airbyte**. If you want to contribute what you’ve built to the Airbyte Cloud and OSS connector catalog, follow the steps provided in the [contribution guide for submitting new connectors](../contributing-to-airbyte/submit-new-connector.md).
|
||||
3. **Contribute back to Airbyte**. If you want to contribute what you've built to the Airbyte Cloud and OSS connector catalog, follow the steps provided in the [contribution guide for submitting new connectors](./submit-new-connector.md).
|
||||
|
||||
## Connector development options
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Managing Breaking Changes in Connectors
|
||||
|
||||
Whenever possible, changes to connectors should be implemented in a non-breaking way. This allows users to upgrade to the latest version of a connector without additional action required on their part. Assume that _every_ breaking changes creates friction for users and should be avoided except when absolutely necessary.
|
||||
|
||||
When it is not possible to make changes in a non-breaking manner, additional **breaking change requirements** include:
|
||||
|
||||
1. A **Major Version** increase. (Or minor in the case of a pre-1.0.0 connector in accordance with Semantic Versioning rules)
|
||||
2. A [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) in the `releases` section of the `metadata.yaml` file
|
||||
3. A migration guide which details steps that users should take to resolve the change
|
||||
4. An Airbyte Engineer to complete the [Connector Breaking Change Release Playbook](https://docs.google.com/document/u/0/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit) (internal link) before merging the PR.
|
||||
|
||||
## Types of Breaking Changes
|
||||
|
||||
A breaking change is any change that requires users to take action before they can continue to sync data. The following are examples of breaking changes:
|
||||
|
||||
- **Spec Change** - The configuration required by users of this connector have been changed and syncs will fail until users reconfigure or re-authenticate. This change is not possible via a Config Migration
|
||||
- **Schema Change** - The type of property previously present within a record has changed
|
||||
- **Stream or Property Removal** - Data that was previously being synced is no longer going to be synced.
|
||||
- **Destination Format / Normalization Change** - The way the destination writes the final data or how normalization cleans that data is changing in a way that requires a full-refresh.
|
||||
- **State Changes** - The format of the source’s state has changed, and the full dataset will need to be re-synced
|
||||
- **Full Downstream Rewrite** - Very rarely, a change is so sigificant that it requires a full rewrite of downstream SQL transformations or BI dashboards. In these cases, consider forking the connector as a "Gen 2" version instead of making a breaking change that would fully break users' downstream pipelines.
|
||||
- Example: Migration from legacy JSON-only "raw" tables to normalized typed columns in a destination tables. These historic changes were so significant that they would break all downstream SQL transformations and BI dashboards. A "gen-2" approach in these cases gives users the ability to run both "Gen 1" and "Gen 2" in parallel, migrating only after they have had a chance to adapt their code to the new data models.
|
||||
|
||||
### Avoiding Breaking Changes with Migrations
|
||||
|
||||
In some cases, breaking changes can be avoided through automated migrations:
|
||||
|
||||
- **Config Migrations** - For low-code connectors, config migrations can automatically transform old configurations to new formats, avoiding the need for users to manually reconfigure. See the [low-code config migration model](https://github.com/airbytehq/airbyte-python-cdk/blob/8158f0d2a07c0480d25581359d54c8f9d30dbb29/airbyte_cdk/sources/declarative/declarative_component_schema.yaml#L4021-L4045) for technical reference.
|
||||
|
||||
- **State Migrations** - For low-code connectors, state migrations can automatically transform old state formats to new formats, avoiding the need for a full refresh. See the [low-code state migration model](https://github.com/airbytehq/airbyte-python-cdk/blob/8158f0d2a07c0480d25581359d54c8f9d30dbb29/airbyte_cdk/sources/declarative/declarative_component_schema.yaml#L1579-L1587) for technical reference.
|
||||
|
||||
When considering spec or state changes, evaluate whether a migration can eliminate the need for a breaking change before proceeding with a major version bump.
|
||||
|
||||
### Schema Change Considerations: Data Type Compatibility
|
||||
|
||||
When changing data types in schemas, consider type compatibility to determine whether the change is truly breaking:
|
||||
|
||||
- **Widening that is breaking downstream:** Changes like `float` to `str` or `datetime` to `str` widen the type but may break downstream processes that expect numeric or date values.
|
||||
|
||||
- **Widening that is non-breaking:** Changes like `int` to `bigint` or `int` to `double` widen the type in a compatible way that preserves data semantics.
|
||||
|
||||
- **Non-compatible type changes:** Changes between fundamentally incompatible types (e.g., `datetime` to `float` or vice versa) are breaking and should be avoided if at all possible, as they change the semantic meaning of the data.
|
||||
|
||||
When in doubt, treat type changes as breaking unless you can verify that all downstream use cases will handle the new type correctly.
|
||||
|
||||
## Defining the Scope of Breaking Changes
|
||||
|
||||
Some legitimate breaking changes may not impact all users of the connector. For example, a change to the schema of a specific stream only impacts users who are syncing that stream.
|
||||
|
||||
The breaking change metadata allows you to specify narrowed scopes, and specifically _which streams_ are specifically affected by a breaking change. See the [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) documentation for supported scopes. If a user is not using the affected streams and therefor are not affected by a breaking change, they will not see any in-app messaging or emails about the change.
|
||||
|
||||
## Migration Guide Documentation Requirements
|
||||
|
||||
Your migration guide must be created as a separate file at `docs/integrations/{sources|destinations}/{connector-name}-migrations.md`. The guide should be detailed and user-focused, addressing the following for each breaking change version:
|
||||
|
||||
- **WHAT** - What changed: Specifically, what is fixed or better for the user after this change?
|
||||
- **WHY** - Why did we make this change? (API improvements, upstream deprecation, bug fixes, performance improvements).
|
||||
- **WHO** - Which users are affected? Be specific about streams, sync modes, or configuration options that are impacted.
|
||||
- **STEPS** - Exact steps users must take to migrate, including when to take them (before/after upgrade, before/after first sync).
|
||||
|
||||
Your migration guide can be as long as necessary and may include images, code snippets, SQL examples, and compatibility tables to help users understand and execute the migration.
|
||||
|
||||
### Examples of Good Migration Guides
|
||||
|
||||
Review these examples to understand the expected format and level of detail:
|
||||
|
||||
- [HubSpot Migration Guide](/integrations/sources/hubspot-migrations) - Clear version sections with affected stream identification
|
||||
- [Google Ads Migration Guide](/integrations/sources/google-ads-migrations) - Table-based field change documentation
|
||||
- [Stripe Migration Guide](/integrations/sources/stripe-migrations) - Detailed sync mode and cursor field changes
|
||||
- [Snowflake Destination Migration Guide](/integrations/destinations/snowflake-migrations) - Use case-based migration paths
|
||||
|
||||
## Breaking Change Metadata Requirements
|
||||
|
||||
When adding a `breakingChanges` entry to your connector's `metadata.yaml` file, you must provide two critical fields:
|
||||
|
||||
### Message Field
|
||||
|
||||
The `message` field is a short summary shown in-app to all users of the connector. This message should:
|
||||
|
||||
- Be concise but informative (users should know if they're affected and what action is needed).
|
||||
- Identify which users are affected.
|
||||
For example: "users syncing the `campaigns` stream"
|
||||
- Summarize the action required (detailed steps go in the migration guide).
|
||||
- Link to the migration guide for full details.
|
||||
|
||||
The platform uses this message to send automated emails to affected users, so clear communication is critical.
|
||||
|
||||
### Upgrade Deadline Field
|
||||
|
||||
The `upgradeDeadline` field specifies the date by which users should upgrade (format: `YYYY-MM-DD`). When setting this deadline:
|
||||
|
||||
- **Minimum timelines:**
|
||||
- Source connectors: At least 7 days (2 weeks recommended for simple changes)
|
||||
- Destination connectors: At least 1 month (destinations have broader impact on data pipelines)
|
||||
|
||||
- **Rationale:** The deadline should provide enough time for users to review the migration guide, test in staging environments, and execute the migration steps.
|
||||
|
||||
- **Exception: Immediate upstream breakage:** In the case of immediate upstream breaking changes, such as an already-removed upstream API endpoint, the deadline can be present-day or even in the past - with the rationale that users' connections are _already_ broken without the fix and therefor need the upgrade applied immediately.
|
||||
|
||||
- **Automated notifications:** The platform automatically emails users when a breaking change is released and sends reminders as the deadline approaches.
|
||||
|
||||
## Related Topics
|
||||
|
||||
- [Semantic Versioning for Connectors](../contributing-to-airbyte/resources/pull-requests-handbook.md#semantic-versioning-for-connectors) - Guidelines for determining Major/Minor/Patch version changes
|
||||
- [Connector Metadata File](https://docs.airbyte.com/connector-development/connector-metadata-file/) - Technical reference for `breakingChanges` metadata format
|
||||
- [Pull Request Title Convention](../contributing-to-airbyte/resources/pull-requests-handbook.md#pull-request-title-convention) - How to format PR titles (use 🚨 emoji for breaking changes)
|
||||
- [QA Checks](../contributing-to-airbyte/resources/qa-checks.md) - Automated quality checks including breaking change requirements
|
||||
- [Connector Breaking Change Release Playbook](https://docs.google.com/document/u/0/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit) - Internal process for Airbyte Engineers (requires access)
|
||||
@@ -202,3 +202,7 @@ When working on PRs from forks, maintainers can apply `/format-fix` to help expe
|
||||
Note:
|
||||
|
||||
- Slash commands may only be executed by maintainers, and they run with the context and the permissions from the main repo.
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
When making breaking changes to a connector, please be sure to follow the [breaking change process](./connector-breaking-changes.md) as outlined in our contribution guidelines.
|
||||
|
||||
@@ -19,15 +19,15 @@ The token must be a (classic) token with repo scope enabled. Follow these direct
|
||||
2. In the left sidebar, click `Developer settings`.
|
||||
3. In the left sidebar, under Personal access tokens, click `Tokens (classic)`.
|
||||
|
||||
|
||||
|
||||
|
||||
4. Select Generate new token, then click `Generate new token (classic)`.
|
||||
|
||||
|
||||
|
||||
|
||||
5. Add the `workflow` scope. Click `Generate Token`.
|
||||
|
||||
|
||||
|
||||
|
||||
6. Make sure to copy your personal access token. You won’t be able to see it again! Paste the personal access token to the Airbyte UI, and click "Contribute" to contribute the connector.
|
||||
|
||||
@@ -47,9 +47,9 @@ This will enable our team to make sure your contribution does not overlap with e
|
||||
|
||||
1. Make sure your connector passes `airbyte-ci connectors test` tests. [Here's a guide on how to run them](../connector-development/testing-connectors/README.md).
|
||||
2. Make sure you include the README, documentation, and an icon for your connector. Without them, one of the CI checks will fail.
|
||||
3. Follow the [pull request convention](./resources/pull-requests-handbook.md#pull-request-title-convention)
|
||||
3. Follow the [pull request convention](../contributing-to-airbyte/resources/pull-requests-handbook.md#pull-request-title-convention)
|
||||
4. Wait for a review from a community maintainer or our team. We generally look for the following criteria:
|
||||
- Does this PR cover authentication, pagination, and incremental syncs where applicable?
|
||||
- Does this PR cover authentication, pagination, and incremental syncs where applicable?
|
||||
- Does the PR add reasonable list of streams?
|
||||
- If the connector uses custom Python components, did you write unit tests?
|
||||
5. Provide a sandbox account. For some APIs, we'll need a sandbox account that we'll ask for. We'll then set it up in our CI and use it to test this connector in the future.
|
||||
@@ -54,10 +54,15 @@ Do not submit a pull request using the master branch from your forked repository
|
||||
The team will not be able to run integration tests and your pull request will be closed.
|
||||
:::
|
||||
|
||||
Guidelines to common code contributions:
|
||||
:::tip
|
||||
It is generally preferrable to submit pull requests from a personal fork instead of an organization fork. This is because GitHub does not allow maintainers to commit directly to branches in organization forks. If you submit from a personal fork, Airbyte maintainers can apply suggested fixes directly, which can significantly speed up the review and approval process.
|
||||
:::
|
||||
|
||||
- [Contribute a code change to an existing connector](change-cdk-connector.md)
|
||||
- [Contribute a new connector](submit-new-connector.md)
|
||||
Guidelines for connector contributions included in the [Connector Development Guide](../connector-development/README.md):
|
||||
|
||||
- [Contribute a New Connector](../connector-development/submit-new-connector.md) - Guide to submitting a new connector to Airbyte.
|
||||
- [Developing Connectors Locally](../connector-development/local-connector-development.md) - Guide to setting up your local environment for connector development.
|
||||
- [Breaking Changes in Connectors](../connector-development/connector-breaking-changes.md) - Guide to breaking changes and version updates.
|
||||
|
||||
## Documentation contributions
|
||||
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
# Changes to Python CDK or Low-Code Connector
|
||||
|
||||
## Contribution Process
|
||||
|
||||
### Open an issue, or find a similar one
|
||||
|
||||
Before jumping into the code:
|
||||
|
||||
1. Check if the improvement you want to make or bug you want to fix is already captured in an [existing issue](https://github.com/airbytehq/airbyte/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Fconnectors+-label%3Aneeds-triage+label%3Acommunity)
|
||||
2. If you don't find an existing issue, either
|
||||
- [Report a Connector Bug](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fbug%2Carea%2Fconnectors%2Cneeds-triage&projects=&template=1-issue-connector.yaml), or
|
||||
- [Request a New Connector Feature](https://github.com/airbytehq/airbyte/discussions/categories/connector-ideas-and-features)
|
||||
|
||||
This will enable our team to make sure your contribution does not overlap with existing works and will comply with the design orientation we are currently heading the product toward. If you do not receive an update on the issue from our team, please ping us on [Slack](https://slack.airbyte.io)!
|
||||
|
||||
|
||||
### Code your contribution
|
||||
|
||||
1. To contribute to a connector, fork the [Connector repository](https://github.com/airbytehq/airbyte).
|
||||
2. Open a branch for your work
|
||||
3. Code the change
|
||||
4. Write a unit test for each custom function you added or changed
|
||||
5. Ensure all tests, including connector acceptance tests, pass
|
||||
6. Update the `metadata.yaml` following the [guidelines](./resources/pull-requests-handbook.md#semantic-versioning-for-connectors)
|
||||
7. Update the changelog entry in documentation in `docs/integrations/<connector-name>.md`
|
||||
8. Make sure your contribution passes our [QA checks](./resources/qa-checks.md)
|
||||
|
||||
:::info
|
||||
There is a README file inside each connector folder containing instructions to run that connector's tests locally.
|
||||
:::
|
||||
|
||||
:::warning
|
||||
Pay attention to breaking changes to connectors. You can read more [here](#breaking-changes-to-connectors).
|
||||
:::
|
||||
|
||||
### Open a pull request
|
||||
|
||||
1. Rebase master with your branch before submitting a pull request.
|
||||
2. Open the pull request.
|
||||
3. Follow the [title convention](./resources/pull-requests-handbook.md#pull-request-title-convention) for Pull Requests
|
||||
4. Link to an existing Issue
|
||||
5. Update the [description](./resources/pull-requests-handbook.md#descriptions)
|
||||
6. Wait for a review from a community maintainer or our team.
|
||||
|
||||
### Review process
|
||||
|
||||
When we review, we look at:
|
||||
|
||||
- Does the PR solve the issue?
|
||||
- Is the proposed solution reasonable?
|
||||
- Is it tested? \(unit tests or integration tests\)
|
||||
- Is it introducing security risks?
|
||||
- Is it introducing a breaking change?
|
||||
Once your PR passes, we will merge it 🎉.
|
||||
|
||||
## Breaking Changes to Connectors
|
||||
|
||||
Often times, changes to connectors can be made without impacting the user experience. However, there are some changes that will require users to take action before they can continue to sync data. These changes are considered **Breaking Changes** and require:
|
||||
|
||||
1. A **Major Version** increase. (Or minor in the case of a pre-1.0.0 connector in accordance with Semantic Versioning rules)
|
||||
2. A [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) in the `releases` section of the `metadata.yaml` file
|
||||
3. A migration guide which details steps that users should take to resolve the change
|
||||
4. An Airbyte Engineer to follow the [Connector Breaking Change Release Playbook](https://docs.google.com/document/u/0/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit) before merging.
|
||||
|
||||
### Types of Breaking Changes
|
||||
|
||||
A breaking change is any change that will require users to take action before they can continue to sync data. The following are examples of breaking changes:
|
||||
|
||||
- **Spec Change** - The configuration required by users of this connector have been changed and syncs will fail until users reconfigure or re-authenticate. This change is not possible via a Config Migration
|
||||
- **Schema Change** - The type of property previously present within a record has changed
|
||||
- **Stream or Property Removal** - Data that was previously being synced is no longer going to be synced.
|
||||
- **Destination Format / Normalization Change** - The way the destination writes the final data or how normalization cleans that data is changing in a way that requires a full-refresh.
|
||||
- **State Changes** - The format of the source’s state has changed, and the full dataset will need to be re-synced
|
||||
|
||||
### Limiting the Impact of Breaking Changes
|
||||
|
||||
Some of the changes listed above may not impact all users of the connector. For example, a change to the schema of a specific stream only impacts users who are syncing that stream.
|
||||
|
||||
The breaking change metadata allows you to specify narrowed scopes that are specifically affected by a breaking change. See the [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) documentation for supported scopes.
|
||||
@@ -8,7 +8,7 @@ TypeScript using React.
|
||||
## Submitting Code
|
||||
|
||||
If you would like to submit code to Airbyte, please follow the [Pull Request Handbook](resources/pull-requests-handbook.md)
|
||||
guide when creating Github Pull Requests. When you are ready to submit code, use the [Submit a New Connector](submit-new-connector.md) document to make
|
||||
guide when creating Github Pull Requests. When you are ready to submit code, use the [Submit a New Connector](../connector-development/submit-new-connector.md) document to make
|
||||
sure that the process can go as smoothly as possible.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -136,6 +136,8 @@ const buildAConnector = {
|
||||
],
|
||||
},
|
||||
"connector-development/local-connector-development",
|
||||
"connector-development/submit-new-connector",
|
||||
"connector-development/connector-breaking-changes",
|
||||
"connector-development/connector-specification-reference",
|
||||
"connector-development/partner-certified-destinations",
|
||||
"operator-guides/using-custom-connectors",
|
||||
@@ -157,8 +159,6 @@ const contributeToAirbyte = {
|
||||
},
|
||||
items: [
|
||||
"contributing-to-airbyte/issues-and-requests",
|
||||
"contributing-to-airbyte/change-cdk-connector",
|
||||
"contributing-to-airbyte/submit-new-connector",
|
||||
"contributing-to-airbyte/developing-locally",
|
||||
"contributing-to-airbyte/writing-docs",
|
||||
{
|
||||
|
||||
@@ -480,6 +480,16 @@
|
||||
"source": "/platform/next/embedded-setup",
|
||||
"destination": "/embedded",
|
||||
"statusCode": 301
|
||||
},
|
||||
{
|
||||
"source": "/platform/contributing-to-airbyte/change-cdk-connector",
|
||||
"destination": "/platform/connector-development/connector-breaking-changes",
|
||||
"statusCode": 301
|
||||
},
|
||||
{
|
||||
"source": "/platform/contributing-to-airbyte/submit-new-connector",
|
||||
"destination": "/platform/connector-development/submit-new-connector",
|
||||
"statusCode": 301
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user