1
0
mirror of synced 2026-01-08 12:03:02 -05:00
Commit Graph

11815 Commits

Author SHA1 Message Date
Alexandre Girard
97a353d5c5 Run mypy on airbyte-cdk as part of the build pipeline and fix typing issues in the file-based module (#27790)
* Try running only on modified files

* make a change

* return something with the wrong type

* Revert "return something with the wrong type"

This reverts commit 23b828371e.

* fix typing in file-based

* format

* Mypy

* fix

* leave as Mapping

* Revert "leave as Mapping"

This reverts commit 908f063f70.

* Use Dict

* update

* move dict()

* Revert "move dict()"

This reverts commit fa347a8236.

* Revert "Revert "move dict()""

This reverts commit c9237df2e4.

* Revert "Revert "Revert "move dict()"""

This reverts commit 5ac1616414.

* use Mapping

* point to config file

* comment

* strict = False

* remove --

* Revert "comment"

This reverts commit 6000814a82.

* install types

* install types in same command as mypy runs

* non-interactive

* freeze version

* pydantic plugin

* plugins

* update

* ignore missing import

* Revert "ignore missing import"

This reverts commit 1da7930fb7.

* Install pydantic instead

* fix

* this passes locally

* strict = true

* format

* explicitly import models

* Update

* remove old mypy.ini config

* temporarily disable mypy

* format

* any

* format

* fix tests

* format

* Automated Commit - Formatting Changes

* Revert "temporarily disable mypy"

This reverts commit eb8470fa3f.

* implicit reexport

* update test

* fix mypy

* Automated Commit - Formatting Changes

* fix some errors in tests

* more type fixes

* more fixes

* more

* .

* done with tests

* fix last files

* format

* Update gradle

* change source-stripe

* only run mypy on cdk

* remove strict

* Add more rules

* update

* ignore missing imports

* cast to string

* Allow untyped decorator

* reset to master

* move to the cdk

* derp

* move explicit imports around

* Automated Commit - Formatting Changes

* Revert "move explicit imports around"

This reverts commit 56e306b72f.

* move explicit imports around

* Upgrade mypy version

* point to config file

* Update readme

* Ignore errors in the models module

* Automated Commit - Formatting Changes

* move check to gradle build

* Any

* try checking out master too

* Revert "try checking out master too"

This reverts commit 8a8f3e373c.

* fetch master

* install mypy

* try without origin

* fetch from the script

* checkout master

* ls the branches

* remotes/origin/master

* remove some cruft

* comment

* remove pydantic types

* unpin mypy

* fetch from the script

* Update connectors base too

* modify a non-cdk file to confirm it doesn't get checked by mypy

* run mypy after generateComponentManifestClassFiles

* run from the venv

* pass files as arguments

* update

* fix when running without args

* with subdir

* path

* try without /

* ./

* remove filter

* try resetting

* Revert "try resetting"

This reverts commit 3a54c424de.

* exclude autogen file

* do not use the github action

* works locally

* remove extra fetch

* run on connectors base

* try bad  typing

* Revert "try bad  typing"

This reverts commit 33b512a3e4.

* reset stripe

* Revert "reset stripe"

This reverts commit 28f23fc6dd.

* Revert "Revert "reset stripe""

This reverts commit 5bf5dee371.

* missing return type

* do not ignore the autogen file

* remove extra installs

* run from venv

* Only check files modified on current branch

* Revert "Only check files modified on current branch"

This reverts commit b4b728e654.

* use merge-base

* Revert "use merge-base"

This reverts commit 3136670cbf.

* try with updated mypy

* bump

* run other steps after mypy

* reset task ordering

* run mypy though

* looser config

* tests pass

* fix mypy issues

* type: ignore

* optional

* this is always a bool

* ignore

* fix typing issues

* remove ignore

* remove mapping

* Automated Commit - Formatting Changes

* Revert "remove ignore"

This reverts commit 9ffeeb6cb1.

* update config

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-07-13 16:55:48 -07:00
Duy Nguyen
792878b253 [Source-Postgres] Handle empty table when syncing via Ctid -> cursor-based (#28298)
* Only build cursorbased stream state with cursor value when table has data

* Add statetype and version to empty state

* only put entry in map once
2023-07-13 17:34:12 -05:00
Ben Church
67a01d6be2 Add explicit registry to metadata file for langchain and gainsight (#28296)
* DNC

* Fix metadata files

* Revert "DNC"

This reverts commit 85ba0cedba.
2023-07-13 15:15:41 -05:00
Brian Lai
8e835963c1 [file-based cdk] spec schema improvements and fixes (#28263)
* fix spec schema incompatibility with ui and improve spec documentation and titles

* fix schema to account for latest changes pulled from main

* tests

* remove duplicate test
2023-07-13 15:14:05 -04:00
Duy Nguyen
633c939d46 [Source-postgres] Set default cursor value for cdc mode (#27442)
* use LSN as default cursor for postgres CDC

* Fixed static constant

* Set lsn default cursor value for postgres sync

* Bumped metadata and dockerfile versions

* Disable acceptance backwards compatibility discovery test as this is a breaking change

---------

Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
2023-07-13 11:12:34 -05:00
Catherine Noll
48843cf807 File-based CDK: handle user-input schema (#28052) 2023-07-13 11:59:42 -04:00
Denys Davydov
c816d14fc1 Source Google Ads: add new streams (#28246)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* add new streams

* upd changelog

* update CAT config

* update expected records
2023-07-13 16:53:08 +03:00
btkcodedev
c6c3ee11b8 Source Twilio: New streams UserConversations with parent Users (#27221)
* New stream UserConversations with parent Users

* Update Docs

* EOF streams.py

* formatting fixes

* Add account_sid as primary key

* bump version

* fix merge errors

* fix catalog and expected records

---------

Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Daryna Ishchenko <darina.ishchenko17@gmail.com>
2023-07-13 16:09:42 +03:00
Arsen Losenko
1f3bd5632a Source Pipedrive: update expected records (#28272) 2023-07-13 15:14:30 +03:00
Arsen Losenko
3f8c87d5e0 🐛 Source Hubspot: update expected records (#28040)
* Source Hubstpo: update expected records

* Update contacts, email_events, goals expected records

* Update CAT config

* Update goals expected records
2023-07-13 15:14:21 +03:00
Arsen Losenko
94fd3f5c29 Source Sentry: update expected records (#28271) 2023-07-13 15:13:44 +03:00
Roman Yermilov [GL]
aea5a9ec8e Source Amazon Ads: fix Nonetype error when recordId is missing (#28155)
* Source Amazon Ads: fix Nonetype error when recordId is missing

* Source Amazon Ads: bump version and update changelog

* Source Amazon Ads: unittest for recordId generation
2023-07-13 15:10:41 +04:00
Artem Inzhyyants
a0e152aa55 🚨🚨 Source PayPal Transactions: Rename field in Balances schema (#27916)
* Source PayPal Transactions: update spec

* Source PayPal Transactions: update docs

* Source PayPal Transactions: update test config

* Source PayPal Transactions: update test config

* Source PayPal Transactions: update test config

* Source PayPal Transactions: update balances schema

* Source PayPal Transactions: update test config

* Source PayPal: bump version

---------

Co-authored-by: Augustin <augustin@airbyte.io>
2023-07-13 11:23:47 +02:00
Anatolii Yatsuk
507ad698f3 Source Monday: Add incremental sync for Items and Boards stream (#27944)
* Add incremental sync for the Boards and Items streams using Activity logs

---------

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-07-13 12:13:47 +03:00
Brian Lai
f0951ffbd8 [file-based cdk] file based spec boilerplate backed by pydantic models (#28139)
* file based spec operation backed by pydantic models

* pr feedback to clean up various config and the test scenarios

* fix tests after rebase
2023-07-12 19:42:50 -04:00
Davin Chia
188782e386 Revert delete insert test. (#28253)
Unfortunately other tests depend on this.

---------

Co-authored-by: davinchia <davinchia@users.noreply.github.com>
2023-07-12 16:30:33 -07:00
Ben Church
4c6f14a016 Disable gradle build cache YOLO (#28248) 2023-07-12 17:39:35 -05:00
Alexandre Girard
40e62fbcb4 Implement parquet parser (#28064)
* Implement parquet parser

* move comment

* comments

* Automated Commit - Formatting Changes

* cleanup

* Update

* remove superfluous method

* update

* format

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-07-12 13:40:30 -07:00
pnavarromuttdata
9f12062a35 🎉 Source Facebook Marketing - add action report time enum to adinsights class (#26345)
* chore: bump version

* feat: new action_report_time spec into custom_insights stream

* feat: airbyte-config new action_report_time enum spec into custom_insights stream FBMarketing

* feat: new action_report_time tests

* fix: update allOf instead of array json spec

* chore: update docs source connector

* update pr id into version history doc

* update metadata.yaml

* adding action report time into cloud registry and oss registry

* fix formatting of python files

* fix: update type action_report_time

* fix: update title action report time

* fix: format python files with flake8

* fix: update version to minor version

* fix: remove action report time default value in test_get_custom_insights_stream

* fix: keep old config for some unit tests

* fix: put missing action breakdowns

* fix: remove default value variable and set into class argument directly

* chore: bump version

* Delete changes from cloud_registry

* Removed action_report_time from oss registry

---------

Co-authored-by: Juan Marchese <juan.marchese@muttdata.ai>
Co-authored-by: sajarin <sajarindider@gmail.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
2023-07-12 20:50:09 +02:00
Davin Chia
c76235c468 Remove Snowflake insert tests that are no longer applicable. (#28245)
We no longer support the insert method. Confirmed in https://airbytehq-team.slack.com/archives/C03C4AVJWG4/p1688711004709439.
2023-07-12 11:36:48 -07:00
Christo Grabowski
56a7f07a92 📝 Docs: Source S3 documentation update (#28229)
* add detailed setup steps for s3 bucket

* complete s3 setup steps

* compress versioned setup steps

* update S3 Provider Settings section

* update CSV and Parquet sections

* update file format settings section

* final edits/fixes

* maintain typecase of True/False

* Update docs/integrations/sources/s3.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/s3.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/s3.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/s3.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add example to escape character field description

---------

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-07-12 14:27:58 -04:00
Joe Reuter
b52e88a977 🎉 New Destination Vector Database (powered by LangChain) (#26184)
* basic version

* polish

* iterate

* keep working

* fix spec

* wip

* improve destination

* basic unit tests

* move embedding dimensionality into embedder

* improve several things

* adjust documentation

* remove unnecessary call

* add some debug information

* fix local destination

* various small fixes

* bring tests into order

* document and add batching to pinecone

* checklist

* improve performance a bit and add test

* fix formatting

* fix metadata

* install C++ 11 on python base

* no more alpine for ci-connector-ops

* remove hard-to-run test

* more documentation

* better documentation

* add icon

* some small adjustments

* review comments

* format

* review comments

---------

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
2023-07-12 20:06:59 +02:00
Roman Yermilov [GL]
6f89af01d7 Source Google Ads: disable logging (#28230)
* Source Google Ads: disable logging

* Source Google Ads: bump version, update changelog
2023-07-12 21:05:25 +04:00
btkcodedev
6792753b86 New Source: Gainsight-px [Low Code CDK] (#26998)
* Initial commit

* Update source_definitions.yaml

* Update source_specs.yaml

* change manifest

* fix manifest

* fix schema

* fix conflict

* fix conflict

* fix schema, refactor manifest, add new streams

* add streams to docs

* add stream users

* add new stream survey_response

* update docs

* EOF, formatting

* Add new stream, Update docs

* Add new streams

* Refactor Schemas

* Change gainsight to gainsight-px

* Add spec

* Change spec

* rewrite spec

* Fix Schemas, metadata

* Add pagination, fix schemas

* Add icon

* Update gainsight-px.md

* Update supportUrl to documentationUrl

Co-authored-by: Mal Hancock <mhancock@archangelic.space>

* Rename to icon.svg

* fix metedata.yaml

---------

Co-authored-by: Mal Hancock <mhancock@archangelic.space>
2023-07-12 09:48:20 -07:00
Arsen Losenko
ead6bcfbea 🐛 Source Twilio: update expected records (#28216)
* Source Twilio: update expected records

* Add channel_metadata property to stream conversation_messages
2023-07-12 19:27:47 +03:00
Arsen Losenko
a6d9fb01f1 Source Stripe: update expected records (#28201) 2023-07-12 19:26:05 +03:00
Christo Grabowski
a0f74e825b 📝 Docs: Files Source setup steps update (#28090)
* add connector setup steps

* add links to Azure Blob storage guides

* add instructions for azure blob

* expand SSH/SCP/SFTP configuration field section

* complete update of setup steps

* change limited tag for local filesystem to OSS

* minor fixes to file source page
2023-07-12 08:31:24 -07:00
Christo Grabowski
cb1081e680 📝 Docs: update HubSpot source page (#28022)
* docs: hubspot source page cleanup

* rework setup steps

* add branching paths for cloud vs oss authentication

* merge engagement-screen-notes sections

* move step skipping note for Cloud users to top of setup

* update in-app setup steps to match docs

* remove stream scopes list from inapp guide

* added stream list back to inapp guide

* remove sentence implying start date is optional

* small touch-ups and fixes

* fix typo

* modify engagements note and add env-specific sections

* add recommended tags to authentication methods
2023-07-12 08:31:04 -07:00
Joe Reuter
537b8c44a7 Connector builder: Extend compatibility guide (#28145)
* extend comp guide

* fix
2023-07-12 16:39:33 +02:00
Arsen Losenko
7a9b01f185 Source Zendesk Support: update expected records (#28197) 2023-07-12 14:52:54 +03:00
Arsen Losenko
83faee7e4f Source Intercom: update expected records (#28196) 2023-07-12 14:52:44 +03:00
Arsen Losenko
17a54030a7 Source Trello: update expected records (#28194) 2023-07-12 14:52:33 +03:00
Artem Inzhyyants
3fb737c257 🐛 Source Zendesk Support: use cursor pagination only (#28096)
* Source Zendesk Support: use cursor pagination only

* Source Zendesk Support: fix logger

* Source Zendesk Support: bump version + docs update

* Source Zendesk Support: Fix unit tests

* Source Zendesk Support: Fix request params

* Source Zendesk Support: Fix unit tests

* Source Zendesk Support: update expected_records
2023-07-12 10:29:37 +02:00
Duy Nguyen
f92d833d8d [Source-Postgres] Switch over to user-defined-cursor after CTID initial sync (#27809)
* Separated Standard vs Xmin categorization
Added StandardCtidUtilsTest
Added versions to StandardStatus

* WIP incremental read test

* Added PostgresStandardStateManager to write state_type key to stream state
Added sync to verify state messages and cursor records on running multiple syncs of the same streams

* Isolate tables used for ctid test and enable checkpointing records to 1 for just CTID test

* Moved ctid cursor test to its own file CtidEnabledPostgresSourceTest

* Gracefully handle case where there is no statetype
Update all "standard" to "cursorBased"
2023-07-11 19:30:40 -05:00
Ben Church
679d475cae Break resources into dependency trees (#28166) 2023-07-11 14:25:13 -07:00
markortleb-dice
8ff7e58077 Source Salesloft: Added new streams (call data records, call dispositions, call sentiments, custom fields, meetings, searches) (#27505)
* adding new Salesloft streams

* change the version to 1.1.1

* increment the patch version

* fixing typo in meetings

* fix another typo

* making corrections requested by reviewer

* increment version in salesloft metadata to 1.2.0

* reformating files

* fix expected records

* Source Salesloft: change Call Dispositions, Call Sentiments, and Custom Fields to full refresh. Use created_at instead of updated_at for timestamp field for Meetings.

* changing meetings cursor field to created_at

* Source Salesloft: Convert meetings to a non-incremental stream, since there is no viable cursor field option.

* fix schemas and update tests

---------

Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
2023-07-11 13:49:07 -07:00
simonbutt-datatonic
af4b0d7244 Update google-analytics-data-api.md (#28081)
Updating the "Full Refresh - Overwrite" link in `google-analytics-data-api.md` as currently it sends to a page not found link (https://docs.airbyte.com/understanding-airbyte/glossary#full-refresh-sync).
2023-07-11 15:01:41 -05:00
Marcos Marx
ff868ee983 Source Google Sheets: fix several reported vulnerabilities (#28085)
* Fixes several reported vulnerabilities,
total - 25, critical - 1, high - 17, medium - 6, low - 1
Upgraded alpine from 3.15 to 3.18
CVE-2022-37434
CVE-2022-42898
CVE-2022-28391
CVE-2022-1271
CVE-2022-40674
CVE-2023-29491
CVE-2022-1304
CVE-2023-2650
CVE-2023-0464
CVE-2023-0215
CVE-2022-4450

* Updated Changelog with PR number.

* Leave pip installed as its required by our pipeline

---------

Co-authored-by: Mauricio A <mauricio.alarcon@betterup.co>
Co-authored-by: Ben Church <ben@airbyte.io>
2023-07-11 15:16:21 -03:00
Arsen Losenko
9cd5ee701c 🐛 Source Github: update schema for Organizations stream (#28144)
* Source Github: update schema for Organizations stream

* Update changelog
2023-07-11 21:13:36 +03:00
Arsen Losenko
98d175c7a9 Source Faker: update expected records (#28146) 2023-07-11 21:13:19 +03:00
Maxime Carbonneau-Leclerc
0ca1c4bd05 🐛 Source Apple Search Ads: manifest.yaml duplicate key error (#28153)
* Fix source-apple-search-ads manifest.yaml duplicate key error

* version bump
2023-07-11 12:50:20 -05:00
maxi297
07da56914f 🤖 Bump patch version of Airbyte CDK 2023-07-11 17:43:50 +00:00
Maxime Carbonneau-Leclerc
df2a6e50bb Issue 21014/oauth requests (#27973)
* [ISSUE #27494] fix type issue caused by connector builder logging

* [ISSUE #21014] log request/response for oauth as 'global_requests'

* formatcdk

* [ISSUE #21014] support DeclarativeOauth2Authenticator as well

* [ISSUE #21014] improving message grouper tests

* formatcdk

* Test solution with logic in MessageRepository (#27990)

* Test solution with logic in MessageRepository

* Solution without creating a new ModelToComponentFactory

* [ISSUE #21014] adding tests

* [ISSUE #21014] add title and description to global requests

* Revert "Solution without creating a new ModelToComponentFactory"

This reverts commit f17799ecff.

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* [ISSUE #21014] do not break on log appender conflict

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* formatcdk

* [ISSUE #21014] moving is_global to is_auxiliary
2023-07-11 13:37:38 -04:00
Eduard Tudenhoefner
2a69ee5eff Destination Iceberg: Bump Iceberg from 1.0.0 to 1.1.0 (#23201)
* Destination Iceberg: Fix compilation

* Iceberg Destination: Bump Iceberg from 1.0.0 to 1.1.0

* add icon and changelog entry

* force aws dependencies used by iceberg

* fix tests as MINIO runs without SSL

* move icon

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2023-07-11 11:50:18 -05:00
Anatolii Yatsuk
4bd0358c69 🐛 Source Monday: Add new streams and update old (#27410)
Add new streams: Tags, Workspaces. Add new fields for existing streams.

---------

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-07-11 19:07:33 +03:00
Denys Davydov
0e11a6fc3b 🚨 🚨 Source Recharge: fix stream schema (#27612)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #1879 source recharge: fix schema

* #1879 source recharge: upd changelog

* upd CAT config

* source recharge: upd CAT config

---------

Co-authored-by: Augustin <augustin@airbyte.io>
2023-07-11 19:00:36 +03:00
Catherine Noll
07286f7069 File-based CDK: implement schemaless option (#28063) 2023-07-11 11:52:47 -04:00
Christo Grabowski
9217c1221b Source Close.com: add human readable titles to api-key and `start_d… (#27950)
* Source Close.com: add human readable titles to `api-key` and `start_date` fields 

- Added `api-key` title: `API Key`
- Added `start_date` title: `Replication Start Date`

* update version to 0.4.1 in metadata.yaml

* update version to 0.4.1 in Dockerfile

* update close-com.md changelog to 0.4.1

* Update airbyte-integrations/connectors/source-close-com/source_close_com/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

---------

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-07-11 11:48:26 -04:00
Marcos Marx
57a7a03705 change discourse to github discussion (#28152) 2023-07-11 11:50:32 -03:00
Brian Lai
f79aa72d64 refactor config validation_policy to not store policies on the config (#28097) 2023-07-10 20:27:02 -04:00