1
0
mirror of synced 2025-12-20 02:23:30 -05:00
Commit Graph

1266 Commits

Author SHA1 Message Date
girarda
b0df19185b 🤖 Bump patch version of Python CDK 2024-03-29 01:52:26 +00:00
Alexandre Girard
b27ddfe19e connector-builder: return full url-encoded URL instead of separating parameters (#36680) 2024-03-28 18:49:35 -07:00
lazebnyi
a2847fd1b5 🤖 Bump minor version of Python CDK 2024-03-28 22:46:13 +00:00
Serhii Lazebnyi
c3c87ea1a5 follow up to #36294: allow migrate sub stream state with custom partition router (#36590) 2024-03-28 23:39:26 +01:00
dependabot[bot]
5d48bafc6b Bump black from 22.1.0 to 24.3.0 in /airbyte-cdk/python (#36568)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-27 20:17:29 -07:00
Alexandre Girard
634db576dc Python CDK: rename a unit test (#36556) 2024-03-27 17:00:38 -07:00
brianjlai
e7f2fc3365 🤖 Bump patch version of Python CDK 2024-03-27 20:50:10 +00:00
Brian Lai
624415d111 [airbyte-cdk] emit source recordCount as float instead of integer (#36560)
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2024-03-27 16:02:30 -04:00
Ella Rohm-Ensing
aba305435e airbyte cdk: simplify datamodel-gen scripts (#36530) 2024-03-27 19:33:16 +00:00
Ella Rohm-Ensing
3f91b64d53 check cdk: don't use connector test runenr (#36564) 2024-03-27 19:13:17 +00:00
Ella Rohm-Ensing
bd2a814b13 delete unused cdk scripts (#36496) 2024-03-27 10:59:03 -07:00
Ella Rohm-Ensing
0c367680b0 Fix E721 errors in the CDK (#36490) 2024-03-26 18:05:33 +00:00
erohmensing
c2fc35719b 🤖 Bump patch version of Python CDK 2024-03-26 15:54:59 +00:00
Ella Rohm-Ensing
d58466c0bb CDK: fix empty extras (#36485) 2024-03-26 15:49:47 +00:00
girarda
d2031bcfee 🤖 Bump minor version of Python CDK 2024-03-26 00:18:07 +00:00
Alexandre Girard
118a864ea2 low-code: Add string filter (#36393)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-03-25 17:02:31 -07:00
erohmensing
1ba7582bcb 🤖 Bump minor version of Python CDK 2024-03-25 23:52:15 +00:00
Ella Rohm-Ensing
195408eb15 Airbyte CDK: move from gradle to poetry (#36250) 2024-03-25 18:46:57 -05:00
girarda
7023bd12f9 🤖 Bump minor version of Python CDK 2024-03-25 23:13:28 +00:00
Alexandre Girard
28591c3481 per partition state and custom state migrations (#36294) 2024-03-25 16:05:55 -07:00
roman-yermilov-gl
991dc084dd 🤖 Bump minor version of Python CDK 2024-03-22 06:14:07 +00:00
Roman Yermilov [GL]
242dd6a425 Airbyte CDK: request options allowed to be an array (#36357) 2024-03-22 07:07:44 +01:00
Brian Lai
728c92cba1 [airbyte-cdk] Stream should not extract state using legacy get_updated_state if no cursor (#36342) 2024-03-20 23:47:23 -04:00
girarda
d52f6b22ee 🤖 Bump minor version of Python CDK 2024-03-20 21:09:57 +00:00
Alexandre Girard
01324ed1f0 bump CDK minimum python version to 3.9 (#36324) 2024-03-20 14:04:20 -07:00
Anatolii Yatsuk
157be91cb1 File-based CDK: Add skip_wrong_number_of_fields_error parameter for CSV parser (#36237)
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
2024-03-20 22:49:49 +02:00
maxi297
b581db626a 🤖 Bump patch version of Python CDK 2024-03-20 13:55:28 +00:00
Maxime Carbonneau-Leclerc
2f34f084e4 [ISSUE #6548] make all fields nullable except from pk and cursor field (#36201) 2024-03-20 09:48:38 -04:00
Alex Birdsall
44f784e200 Remove most_recent_record arg from Cursor.close_slice (#36216) 2024-03-18 18:28:50 -07:00
artem1205
abb1d69f9e 🤖 Bump patch version of Python CDK 2024-03-18 17:59:32 +00:00
Artem Inzhyyants
240aa0180d Airbyte CDK (low code): add refresh_token_error handler to DeclarativeOauth2Authenticator (#36058)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2024-03-18 18:52:58 +01:00
girarda
36ae0cdd96 🤖 Bump minor version of Python CDK 2024-03-18 17:21:43 +00:00
Alexandre Girard
875e5dfacb low-code: Allow developers to use a custom schema loader (#36040) 2024-03-18 09:42:16 -07:00
ambirdsall
3e4d184d8f 🤖 Bump minor version of Python CDK 2024-03-15 22:38:42 +00:00
Alex Birdsall
a6a1b3c0c3 Base datetime cursor state off latest observed record (#35843) 2024-03-15 15:06:43 -07:00
girarda
8a60518db9 🤖 Bump patch version of Python CDK 2024-03-15 16:50:58 +00:00
Alexandre Girard
5508719699 🐛 low-code: remove debug sleep (#36187) 2024-03-15 09:45:07 -07:00
brianjlai
7c0684dd2d 🤖 Bump patch version of Python CDK 2024-03-13 19:12:51 +00:00
Tobias Macey
f67938993e [airbyte-cdk] Fix tab delimiter configuration in CSV file type (#35901) 2024-03-13 13:46:32 -03:00
erohmensing
797ac00b29 🤖 Bump minor version of Python CDK 2024-03-12 21:58:55 +00:00
girarda
75385c71d7 🤖 Bump patch version of Python CDK 2024-03-12 19:26:10 +00:00
Alexandre Girard
15b954546f raise exception with the full class name if a class for a custom comp… (#35868) 2024-03-12 12:18:38 -07:00
brianjlai
c073e647f9 🤖 Bump patch version of Python CDK 2024-03-11 23:23:42 +00:00
Brian Lai
b1f3b8a36f [airbyte-cdk] entrypoint wrapper should use per-stream state not legacy format (#35976) 2024-03-11 19:09:08 -04:00
brianjlai
0f2685be4f 🤖 Bump minor version of Python CDK 2024-03-09 00:29:49 +00:00
Ella Rohm-Ensing
2ac5248387 Emit record counts in state messages for concurrent streams (#35907)
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2024-03-08 19:08:59 -05:00
Ella Rohm-Ensing
a4dca3b45b CDK: assert >0 state messages per read (fix tests) (#35906)
<!--
Thanks for your contribution! 
Before you submit the pull request, 
I'd like to kindly remind you to take a moment and read through our guidelines
to ensure that your contribution aligns with the type of contributions our project accepts.
All the information you need can be found here:
   https://docs.airbyte.com/contributing-to-airbyte/

We truly appreciate your interest in contributing to Airbyte,
and we're excited to see what you have to offer! 

If you have any questions or need any assistance, feel free to reach out in #contributions Slack channel.
-->

## What
* After https://github.com/airbytehq/airbyte/pull/35905, we should be emitting a state message with every successful sync. However there are a few tests that were too lenient and weren't actually _successful_ syncs. This PR fixes those cases and adds validation that we emit at least one state message per successful sync. 

## How
* Add an assertion that we get at least 1 state message for a successful sync 
* Fix some tests that previously "output 0 expected records" but actually errored silently - do not run them as read tests
* Fix a test that failed silently due to lack of support for multi-format
* Add a new test for syncs that output 0 records successfully

## 🚨 User Impact 🚨
None - test changes


## Pre-merge Actions
*Expand the relevant checklist and delete the others.*

<details><summary><strong>New Connector</strong></summary>

### Community member or Airbyter

- **Community member?** Grant edit access to maintainers ([instructions](https://docs.github.com/en/github/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests))
- Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run `./gradlew :airbyte-integrations:connectors:<name>:integrationTest`.
- Connector version is set to `0.0.1`
    - `Dockerfile` has version `0.0.1`
- Documentation updated
    - Connector's `README.md`
    - Connector's `bootstrap.md`. See [description and examples](https://docs.google.com/document/d/1ypdgmwmEHWv-TrO4_YOQ7pAJGVrMp5BOkEVh831N260/edit?usp=sharing)
    - `docs/integrations/<source or destination>/<name>.md` including changelog with an entry for the initial version. See changelog [example](https://docs.airbyte.io/integrations/sources/stripe#changelog)
    - `docs/integrations/README.md`

### Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

- Create a non-forked branch based on this PR and test the below items on it
- Build is successful
- If new credentials are required for use in CI, add them to GSM. [Instructions](https://docs.airbyte.io/connector-development#using-credentials-in-ci).

</details>

<details><summary><strong>Updating a connector</strong></summary>

### Community member or Airbyter

- Grant edit access to maintainers ([instructions](https://docs.github.com/en/github/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests))
- Unit & integration tests added


### Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

- Create a non-forked branch based on this PR and test the below items on it
- Build is successful
- If new credentials are required for use in CI, add them to GSM. [Instructions](https://docs.airbyte.io/connector-development#using-credentials-in-ci).

</details>

<details><summary><strong>Connector Generator</strong></summary>

- Issue acceptance criteria met
- PR name follows [PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook)
- If adding a new generator, add it to the [list of scaffold modules being tested](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connector-templates/generator/build.gradle#L41)
- The generator test modules (all connectors with `-scaffold` in their name) have been updated with the latest scaffold by running `./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds` then checking in your changes
- Documentation which references the generator is updated as needed

</details>

<details><summary><strong>Updating the Python CDK</strong></summary>

### Airbyter

Before merging:
- Pull Request description explains what problem it is solving
- Code change is unit tested
- Build and my-py check pass
- Smoke test the change on at least one affected connector
   - On Github: Run [this workflow](https://github.com/airbytehq/airbyte/actions/workflows/connectors_tests.yml), passing `--use-local-cdk --name=source-<connector>` as options
   - Locally: `airbyte-ci connectors --use-local-cdk --name=source-<connector> test`
- PR is reviewed and approved
      
After merging:
- [Publish the CDK](https://github.com/airbytehq/airbyte/actions/workflows/publish-cdk-command-manually.yml)
   - The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
   - Write a thoughtful changelog message so we know what was updated.
- Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
   - This step is optional if the change does not affect the connector builder or declarative connectors.

</details>
2024-03-08 14:21:46 -08:00
Ella Rohm-Ensing
acbdc2d6e1 Introduce FinalStateCursor to emit state messages at the end of full refresh syncs (#35905)
Co-authored-by: brianjlai <brian.lai@airbyte.io>
2024-03-08 16:58:26 -05:00
girarda
7d3a5f86d8 🤖 Bump patch version of Python CDK 2024-03-07 16:27:35 +00:00
Alexandre Girard
4a808ee178 🐛 follow up to #35471: update the cartesian stream slicer (#35865) 2024-03-07 08:20:02 -08:00