1
0
mirror of synced 2026-01-02 12:02:47 -05:00
Commit Graph

1490 Commits

Author SHA1 Message Date
Yevhenii
85851b7e0f 🐛 Hotfix Hubspot: add the length of separator symbol to check (#7393)
* hotfix for split_properties function
2021-10-26 23:27:04 +03:00
terencecho
1b1b52d6c9 🎉 New Source: Strava (#7151)
* Initial commit for new source: Strava

* Update readme and invalid_config

* Update PR number in docs

* Update docs/integrations/README

* Annotae spec with airbyte_secret

* Change starting_after input to start_date and add try/catch connection check

* Add connector to connector index

* Address PR commends and separate streams

* Set start_date as a required field

* Remove nexted objects from schema

* Remove schema TODO.md

* Fix schema

* Edit activites schema to allow for null values

* Fix formatting

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Fix formatting

* Update source_definitions.yaml

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-10-26 17:06:12 -03:00
Eugene
46a249e5c9 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) (#7327)
Source-Clickhouse: added support for connection via ssh tunnel
2021-10-26 21:39:18 +03:00
Jenny Brown
22f8c391c9 Clarified wording on Save Actions. (#7358) 2021-10-26 13:05:54 -05:00
Igli
19f2541a03 🎉 New Source: Lemlist (#7062)
* add source lemlist

* update change log

* apply default formating

* add comment, delete unnecessary files

* remove unnecessary things

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-10-26 14:16:49 -03:00
Alasdair Brown
58b569da18 🎉 Source Amazon SQS: New connector (#6937)
* Initial commit, working source with static Creds

* Typo in example queue url

* Adds auto delete of messages after read

* Adds visibility timeout

* remove insecure comments from AWS IAM Key spec

* explicitly set supported sync modes

* explicit sync mode should be lower case

* Adds unit tests for check, discover, read

* remove incremental acceptance test block

* remove incremental from conf catalog sample

* remove test requirement moto from main req

* align int catalog sample with sample_files

* fixing catalog configs

* acceptance testing config

* adds expected records txt

* automated formatting changes

* remove expected records block from acpt test

* Adds Docs page

* Ammends formatting on readme

* Adds doc link to summary

* Improve error handling & debug logging

* Adds bootstrap.md

* Add a todo suggestion for batch output

* Adds SQS to integrations readme list

* lower case properties

* removed unused line

* uses enum for aws region

* updates sample configs to use lowercase

* required props to lower case

* add missed property to lowercase

* gradle formatting

* Fixing issues from acceptance tests

* annotate secrets in spec.json with airbyte_secret

* Adds explicit warnings about data less when using Delete Message option
2021-10-26 13:23:10 -03:00
Yevhenii
928aa6280c 🐛 Source Hubspot: Fix issue with getting 414 HTTP error for streams (#6954)
* Source Hubspot: Fix issue with getting 414 HTTP error for streams

* update code and schemas

* bump version
2021-10-26 15:37:07 +03:00
John Lafleur
722f82d17a Creating license-faq (#7153)
* Creating license-faq

* Updating links in Licenses and more details on examples

* Create examples.md

* Update SUMMARY.md

* Update README.md

* Fix license FAQ asset.

* Update license-faq.md

* Update license-faq.md

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@gmail.com>
2021-10-26 22:25:02 +11:00
LiRen Tu
779c39c088 Copy job attempt state to configs database (#7219)
* Add migration to create latest state table

* Log migration name

* Expose db variables to airbyte-db

* Implement migration

* Fix migration test

* temp

* Rebase on master

* Save state in temporal (#7253)

* Copy state to airbyte_configs table

* Add standard sync state

* Move state methods to config repository

* Add unit tests

* Fix unit tests

* Register standard sync state in migration

* Add comment

* Use config model instead of json node

* Add comments

* Remove unnecessary method

* Fix migration query

* Remove unused config database

* Move persist statement and log the call

* Update dev doc

* Add unit tests for sync workflow

Co-authored-by: Charles <giardina.charles@gmail.com>
2021-10-25 17:08:08 -07:00
Marcos Marx
5a52d20979 Source Recharge: change schema for discount and order (#7295) 2021-10-25 14:10:55 -03:00
irynakruk
dbfd6cae88 Source MongoDB v2: Fixed nested document parsing (#7160)
* Fixed nested document parsing: added parsing of nested documents and arrays, added data types test
2021-10-25 09:16:34 -04:00
Christophe Duong
27df558a89 🎉 BigQuery destinations with partitionned/clustered keys (#7240)
* [ #5959 ][ #2579 ] Add support of partitioned tables by _airbyte_emitted_at field (#7141)

Co-authored-by: Andrés Bravo <andresbravog@gmail.com>
2021-10-25 12:41:04 +02:00
Eugene
9a0b8dac6c 🎉Source Clickhouse: added ssl support and "strict-encrypt" connector (#7127)
* [Ticket 6435] Source-ClickHouse: added ssl support

* [Ticket 6716] Source-ClickHouse: added secure only connector
2021-10-25 13:33:25 +03:00
Jared Rhizor
7fcbc5c027 Bump Airbyte version from 0.30.22-alpha to 0.30.23-alpha (#7320)
Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>
2021-10-24 23:28:26 -07:00
irynakruk
1ae5cf42c3 🐛 Source Snowflake: Fixed parsing of extreme values for FLOAT and NUMBER data types (#7257)
* Fixed parsing of extreme values for FLOAT and NUMBER data types
2021-10-22 15:48:55 -04:00
Brian Bugh
a9e01db8b3 Fix incorrect code for Snowflake GCS (#7287)
This was confusing, as `io_airbyte_test_staging` looks like a literal string and is not mentioned in the explanation above. Slack support context: https://app.slack.com/client/T01AB4DDR2N/C01MFR03D5W/thread/C01MFR03D5W-1634852461.058400
2021-10-22 16:22:29 -03:00
Baz
9a435ec15b 🎉 Source Zuora: support of Unlimited option for Data Query (#7093) 2021-10-22 17:01:00 +03:00
Arthur Galuza
cb6408af8c 🎉 Source Facebook Pages: bump FB API version (#7128)
* Upd FB API version

* Upd changelog

* Exclude fields with extra permissions required
2021-10-22 14:35:50 +03:00
VitaliiMaltsev
0c932749d4 🎉 Redshift Source and Destination set SSL as default option (#7234)
* Redshift Source and Restination set SSL as default option

* add changelog

* remove SSL test| add more documentation

* bump new version

* bump new version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-10-22 12:28:54 +03:00
Marcos Marx
ad6223d817 Publish PR 6962: New Source Salesloft (#7274)
* Generated connector source code

* Added basic users stream

* Added people stream

* Added cadences and cadence_memberships streams

* Updated unit tests

* Fixed docker build

* Updated connector to use OAuth

* address some checklist items

* review comments

* small fix

* add salesloft into airbyte-config

* correct docs

Co-authored-by: lgomezm <luis@calixa.io>
Co-authored-by: prasrvenkat <pras@calixa.io>
2021-10-22 02:31:37 -03:00
Prasanna Ram Venkatachalam
115af277e2 🎉 New Source: Salesloft (#6962)
* Generated connector source code

* Added basic users stream

* Added people stream

* Added cadences and cadence_memberships streams

* Updated unit tests

* Fixed docker build

* Updated connector to use OAuth

* address some checklist items

* review comments

Co-authored-by: lgomezm <luis@calixa.io>
2021-10-22 02:00:16 -03:00
Bo Lu
fac6840cc8 🎉 New Source: OneSignal (#6998)
* add source onesignal

* update PR number in change log

* change source define cursor and sync mode

* get correct max cursor time across stream slices

* code improvement as code review advices

* format code

* remove unused code

* remove TODOs

Co-authored-by: Maksym Pavlenok <antixar@gmail.com>
2021-10-22 00:38:50 -03:00
Chris Wu
17961fccfc Source Jira: Make recently added configs optional for backwards compatibility (#7222)
* Make start_date optional

* Make projects optional

* Bump version
2021-10-21 22:46:27 -03:00
Jared Rhizor
24ff741f56 Bump Airbyte version from 0.30.21-alpha to 0.30.22-alpha (#7264)
Co-authored-by: Phlair <Phlair@users.noreply.github.com>
2021-10-21 23:21:24 +01:00
Marcos Marx
a82d04f688 Docs: correct youtube video link for CDK Speedrun(#7256) 2021-10-21 17:32:48 -03:00
Marcos Marx
0585a5d52d Publish PR 6930: google dir fix members (#7218)
* Google Directory: Fix crash if there's a group don't have members

* bump version

Co-authored-by: Xiangxuan Liu <xiangxuan.liu@rightcapital.com>
2021-10-21 17:32:21 -03:00
George Claireaux
d660661b64 add google sheets oauth flow to server + fix auth rootObject type in protocol (#7131)
* adding google sheets oauth flow to server

* fix oauth type in protocol yaml

* bump sheets version in definitions

* added GDrive scope

* update sheets to master changes

* update protocol incl. cdk

* protocol typing for oauth rootobject

* format
2021-10-21 18:28:50 +01:00
Lake Mossman
bd5d4ee598 Fix more typos in the docs (#7225)
* Fix more typos in the docs

* fix another typo

* add "to"

* make "example" plural

* remove extra words

* add period

* remove extra "the"

* use `source_defined_cursor` instead of `cursor_field` for `AirbyteStream`

* Use ConfiguredAirbyteCatalog instead of AirbyteCatalog
2021-10-21 10:08:28 -07:00
Alexander Tsukanov
561fabea9a 🐛 BUG-4307: Fix incorrect handling of LONG RAW data type (#7125) 2021-10-21 19:01:44 +03:00
Arthur Galuza
dfdbda8cff Source Klaviyo: update schema validation in SAT (#6952)
* Disable schema validation for configured_catalog

* Upd pydantic-generated schema

* Upd comment

* Bump version, upd changelog
2021-10-21 13:05:53 +03:00
LiRen Tu
e87604eccb Update doc about m1 (#7233) 2021-10-20 23:59:02 -07:00
Abhi Vaidyanatha
7ae250deed Fix WooCommerce source page. (#7228) 2021-10-20 18:56:17 -07:00
Benoit Moriceau
78d6d6af1f Update scaling documentation (#7177)
* Update scaling documentation

* RM unvated changes
2021-10-20 16:55:54 -07:00
Hai To
fae4d00c8f Source Hubspot: Add new marketing emails (with statistics) stream (#5840)
* Add new marketing emails (with statistics) stream

* Update docs and changelog
2021-10-20 20:37:48 -03:00
Lake Mossman
a7ddd167f6 Fix typo (#7193)
* Fix typo

* fix another typo

* fix another typo
2021-10-20 14:21:10 -07:00
Jared Rhizor
1affb14827 even more link fixes (#7148)
* even more link fixes

* add remaining exchangerates items
2021-10-20 12:56:55 -07:00
irynakruk
265986b5bc Create a secure-only MongoDb destination (#6945)
* Added mongodb destination strict encrypt
2021-10-20 09:04:30 -04:00
Maksym Pavlenok
059645fad4 🎉 Source File: separate secure fork (#6768)
* create a fork for the source-file connector

* update docs and tests

* update docs and tests

* add secrets

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* refactoring after reviews

* correction of doc

* update spec file



Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-10-20 12:28:26 +03:00
LiRen Tu
e3be3d2ad0 Update M1 doc and fix doc links (#7205)
* Add note about jdk 14 vs 17 for m1

* Fix doc links
2021-10-19 22:31:16 -07:00
Marcos Marx
6c0a17304a Bump Source Jira Connector (#7202)
* Create separate cache file per stream and add projects and start_date config

* Add project id to issue record

* Add projectId to issue schema

* Add BoardIssues stream

* Add SprintIssues stream

* Add Epics stream and deduplicate state code

* Add EpicIssues stream and additional fields for Issues stream

* Add story points to sprint issues

* Add new streams to test catalog

* Update gitignore

* Rename cache boolean and fix test catalog

* Fix streams that depend on Issues stream

* Fix sprint_issues stream

* Add more fields to issues stream and format

* Add option to expand issue changelogs

* Remove epic_issues stream

* Expand project descriptions

* Show rendered fields for epics

* Include project key

* Include project key in issues stream

* Address comments

* Use CDK caching

* Remove extra changes

* Fix sprints stream reading from non-scrum boards

* Format

* small fix and bump version

* bump version

Co-authored-by: Chris Wu <chris@faros.ai>
2021-10-19 23:55:39 -03:00
Jared Rhizor
c2eabc182d m1 pr cleanups (#7197) 2021-10-19 14:53:47 -07:00
Abhi Vaidyanatha
35304291f9 Change Airbyte Spec doc links to permalinks. (#7019) 2021-10-19 11:36:16 -07:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104)
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
2021-10-19 11:20:21 -07:00
George Claireaux
1d3a17a8fb 🎉 Source S3 - memory & performance optimisations + advanced CSV options (#6615)
* memory & performance optimisations

* address comments

* version bump

* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions

* updated to use the latest airbyte-cdk

* updated docs

* bump source-s3 to 0.1.6

* remove unneeded lines

* Use the all dep ami for python builds.

* ec2-instance-id should be ec2-image-id

* ec2-instance-id should be ec2-image-id

Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-19 16:50:51 +01:00
Yevhenii
25110c167d 🎉 Google Directory: Add organizations and phones to users schema (#7167)
* Google Directory: Add organizations and phones to users schema

https://developers.google.com/admin-sdk/directory/v1/guides/manage-users#json-response

* bump version, add changelogs to google-directory.md

Co-authored-by: Xiangxuan Liu <9570112+nauxliu@users.noreply.github.com>
2021-10-19 16:59:00 +03:00
Yevhenii
b18acd6014 🎉 Connector Facebook-Marketing: update insights streams with custom entries for fields, breakdowns and action_breakdowns (#7158)
* Connector Facebook-Marketing: update streams with custom streams

* update: remove custom streams, add new custom insights from config

* update: add new model for InsightConfig, remove old imports

* fix: format to source file and streams file

* update Changelog

* update: add to check a validation to insights entries, update documentation and fix to resolve  in spec schema

* fix: fix import logger from entrypoint, change to python logger

* fix: change error message from check custom insights entries, fix logger import

Co-authored-by: vladimir <vladimir.remar@gmail.com>
2021-10-19 15:08:28 +03:00
Jared Rhizor
2d32df1b20 fix more doc links (#7146) 2021-10-18 18:21:21 -07:00
Jared Rhizor
726c3b909b Bump Airbyte version from 0.30.20-alpha to 0.30.21-alpha (#7145)
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
2021-10-18 18:19:49 -07:00
Tuan Nguyen
1de7f4c365 Minor typos in doc (#7106)
* Update google-ads.md

* Update docs/integrations/sources/google-ads.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-10-18 11:32:34 -03:00
oleh.zorenko
cf04b8bb8e 🎉 Source GitLab: allow all domains to be used as api_url (#7108)
* Remove `pattern` field from `api_url` in spec file

* Bump connector's version + update docs
2021-10-18 16:24:36 +03:00