1
0
mirror of synced 2025-12-25 02:09:19 -05:00
Commit Graph

3520 Commits

Author SHA1 Message Date
LiRen Tu
e3be3d2ad0 Update M1 doc and fix doc links (#7205)
* Add note about jdk 14 vs 17 for m1

* Fix doc links
2021-10-19 22:31:16 -07:00
Marcos Marx
6c0a17304a Bump Source Jira Connector (#7202)
* Create separate cache file per stream and add projects and start_date config

* Add project id to issue record

* Add projectId to issue schema

* Add BoardIssues stream

* Add SprintIssues stream

* Add Epics stream and deduplicate state code

* Add EpicIssues stream and additional fields for Issues stream

* Add story points to sprint issues

* Add new streams to test catalog

* Update gitignore

* Rename cache boolean and fix test catalog

* Fix streams that depend on Issues stream

* Fix sprint_issues stream

* Add more fields to issues stream and format

* Add option to expand issue changelogs

* Remove epic_issues stream

* Expand project descriptions

* Show rendered fields for epics

* Include project key

* Include project key in issues stream

* Address comments

* Use CDK caching

* Remove extra changes

* Fix sprints stream reading from non-scrum boards

* Format

* small fix and bump version

* bump version

Co-authored-by: Chris Wu <chris@faros.ai>
2021-10-19 23:55:39 -03:00
Chris Wu
1fcd6c4671 Source Jira: Add Board, Epic, and Sprint-related streams and improve request caching (#6621)
* Create separate cache file per stream and add projects and start_date config

* Add project id to issue record

* Add projectId to issue schema

* Add BoardIssues stream

* Add SprintIssues stream

* Add Epics stream and deduplicate state code

* Add EpicIssues stream and additional fields for Issues stream

* Add story points to sprint issues

* Add new streams to test catalog

* Update gitignore

* Rename cache boolean and fix test catalog

* Fix streams that depend on Issues stream

* Fix sprint_issues stream

* Add more fields to issues stream and format

* Add option to expand issue changelogs

* Remove epic_issues stream

* Expand project descriptions

* Show rendered fields for epics

* Include project key

* Include project key in issues stream

* Address comments

* Use CDK caching

* Remove extra changes

* Fix sprints stream reading from non-scrum boards

* Format
2021-10-19 23:20:15 -03:00
Jared Rhizor
c2eabc182d m1 pr cleanups (#7197) 2021-10-19 14:53:47 -07:00
LiRen Tu
afde653cde Restore platform option in m1-specific docker compose file (#7188)
* Revert "remove platform option temporarily to fix master build (#7184)"

This reverts commit d9e7bdc3ee.

* Test if docker compose works in github action

* Use a dedicated docker compose file for m1

* Restore docker-compose command

* Update comments

* Fix docker compose filename

* Add issue link
2021-10-19 14:21:18 -07:00
LiRen Tu
d9e7bdc3ee Remove platform option temporarily to fix master build (#7184) 2021-10-19 13:43:11 -07:00
Abhi Vaidyanatha
35304291f9 Change Airbyte Spec doc links to permalinks. (#7019) 2021-10-19 11:36:16 -07:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104)
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
2021-10-19 11:20:21 -07:00
Shahil Mawjee
b79984ab7d Fixed broken link in Source Bigquery spec (#7165) 2021-10-19 13:32:24 -03:00
Jared Rhizor
dc575f6361 fix kube build again (#7174) 2021-10-19 09:29:40 -07:00
George Claireaux
1d3a17a8fb 🎉 Source S3 - memory & performance optimisations + advanced CSV options (#6615)
* memory & performance optimisations

* address comments

* version bump

* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions

* updated to use the latest airbyte-cdk

* updated docs

* bump source-s3 to 0.1.6

* remove unneeded lines

* Use the all dep ami for python builds.

* ec2-instance-id should be ec2-image-id

* ec2-instance-id should be ec2-image-id

Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-19 16:50:51 +01:00
Yevhenii
25110c167d 🎉 Google Directory: Add organizations and phones to users schema (#7167)
* Google Directory: Add organizations and phones to users schema

https://developers.google.com/admin-sdk/directory/v1/guides/manage-users#json-response

* bump version, add changelogs to google-directory.md

Co-authored-by: Xiangxuan Liu <9570112+nauxliu@users.noreply.github.com>
2021-10-19 16:59:00 +03:00
Andrés Bravo
abf0159778 🎉 Add configurable dbt parameter to destination-bigquery (#7118)
* Add configurable dbt  parameter to destination-bigquery

* Update airbyte-integrations/connectors/destination-bigquery/Dockerfile
2021-10-19 14:51:37 +02:00
Yevhenii
b18acd6014 🎉 Connector Facebook-Marketing: update insights streams with custom entries for fields, breakdowns and action_breakdowns (#7158)
* Connector Facebook-Marketing: update streams with custom streams

* update: remove custom streams, add new custom insights from config

* update: add new model for InsightConfig, remove old imports

* fix: format to source file and streams file

* update Changelog

* update: add to check a validation to insights entries, update documentation and fix to resolve  in spec schema

* fix: fix import logger from entrypoint, change to python logger

* fix: change error message from check custom insights entries, fix logger import

Co-authored-by: vladimir <vladimir.remar@gmail.com>
2021-10-19 15:08:28 +03:00
Davin Chia
214b158bde Add labels to EC2 build machines. Refactor actions to reduce duplicate start ec2 runner actions. (#7119)
As the title.

I also took the chance to compose the actions, instead of injecting it everywhere.
2021-10-19 18:02:18 +08:00
Achmad Syarif Hidayatullah
1441bf944b 🎉 New Source: Appsflyer (#6497) 2021-10-19 09:40:30 +03:00
Jared Rhizor
2d32df1b20 fix more doc links (#7146) 2021-10-18 18:21:21 -07:00
Jared Rhizor
726c3b909b Bump Airbyte version from 0.30.20-alpha to 0.30.21-alpha (#7145)
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
v0.30.21-alpha
2021-10-18 18:19:49 -07:00
Jared Rhizor
41db058eb2 loosen definition of object specs in SecretsHelper (#6654)
* address missing secretshelper case

* fix test

* migration changes

* fix build

* format

* switch secrets migration to platform-only build
2021-10-18 17:47:18 -07:00
Jared Rhizor
7901524131 exclude archive from link checker (#7140) 2021-10-18 14:46:27 -07:00
Jared Rhizor
0f93224769 kube build fixes (#7139)
* upgrade ec2-github-runner

* remaining build fixes

* fix build
2021-10-18 14:46:17 -07:00
Mario Molina
8d8eda9744 Fix Airbyte version env var (#7100) 2021-10-18 10:51:14 -07:00
Charles
334d6d6d17 copy correct docs dir to webapp (#7129) 2021-10-18 09:26:40 -07:00
vitaliizazmic
741001ae92 🐛 Airbyte CDK: fixing integration test failing
* Airbyte CDK native logger #1279 - fix import logger error

* Airbyte CDK native logger #1279 - source-paypal-transaction: change level from "WARN" to logging.WARN in self.logger.log

* Airbyte CDK native logger #1279 - source-s3: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - source-zuora: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - fix get logger
2021-10-18 18:46:55 +03:00
Tuan Nguyen
1de7f4c365 Minor typos in doc (#7106)
* Update google-ads.md

* Update docs/integrations/sources/google-ads.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-10-18 11:32:34 -03:00
Dmytro
04989bcad5 Asana oAuth backend. (#7049) 2021-10-18 16:39:43 +03:00
oleh.zorenko
cf04b8bb8e 🎉 Source GitLab: allow all domains to be used as api_url (#7108)
* Remove `pattern` field from `api_url` in spec file

* Bump connector's version + update docs
2021-10-18 16:24:36 +03:00
Sherif A. Nada
62c230ad42 Enable freshdesk in cloud (#7089) 2021-10-18 05:43:00 -07:00
andriikorotkov
ad2b6dbff1 🐛 fixed OOM error when splitting a stream into several files (#7074)
* UPDATED DESTINATION JDBC

* fixed remarks

* fixed remarks

* added new test for testing largeNumberRecords

* updated test

* fixed remarks

* Separate staging filename generator and add tests

Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2021-10-18 15:00:18 +03:00
oleh.zorenko
b6c165c39f Run format command (#7105) 2021-10-18 14:32:19 +03:00
Iakov Salikov
c0c8f90954 fix: hide search control if select no searchable (#6970)
* fix: hide search control if select no searchable
2021-10-18 18:15:34 +07:00
Maksym Pavlenok
4054d8b45f 🐛 Source Zendesk Support: incorrect spec file (#7097)
* remove oauth settings from spec file

* update changelog

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
2021-10-18 11:56:05 +03:00
oleh.zorenko
f1ddfca1c5 🐛 Source Twilio: update schemas + turn on transformer for all streams (#7034)
* Update schemas + turn on transformer for all streams

* Update docs

* Transform RFC2822 datetime into ISO8601

* Bump connector's version + update `accounts` schema
2021-10-18 07:05:07 +03:00
Charles
f194f354c7 move basic normalization to connectors_base build (#7068) 2021-10-15 17:26:08 -07:00
Samy Pessé
d365f85654 Fix invalid YAML frontmatter (#7088) 2021-10-15 21:25:58 -03:00
Charles
8ddfd90c9c add docs on how to set up ide to automatically add final (#7083) 2021-10-15 16:42:11 -07:00
Charles
ba44f700b9 add final for params, local variables, and fields (#7084) 2021-10-15 16:41:04 -07:00
Maksym Pavlenok
592f7fe2b2 🐛 Source ZenDesk Support: fix 404 responses for the ticket_comments stream (#6513)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add log message for 404 errors

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove oauth logic from this PR

* remove oauth logic from this PR

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-16 01:20:41 +03:00
Jared Rhizor
1b5922896b upgrade to Gradle 7.2 (#7070)
* upgrade to gradle 7.2

* get build working with gradle 7.2

* fix build

* fix connector build?

* EXCLUDE distTar and switch includes to excludes

* switch back to includes

* temporarily try to adjust properties

* don't know why this was modified

* revert spotless upgrade

* upgrade gradle node (better arm support it says)

* add back in Xss since we are getting StackOverflowError without it

* revert bad spotless fmt

* fix acceptance tests duplicates strategy

* add zip for jdbc includes

* check if this helps for java connectors

* remove accidental commit

* remove redundant

* fix warning explosion

* oops

* override gradle.properties

* remove set-env
2021-10-15 14:03:30 -07:00
Charles
c129e02ffa refactor confusing methods (#6641) 2021-10-15 13:05:29 -07:00
Charles
d154f8b278 update docs to avoid long run times for formatting (#7064) 2021-10-15 13:04:42 -07:00
Maksym Pavlenok
713fe61efd 🎉 New Source: TikTok Marketing (#5887)
* create a new connector folder

* add base classes

* add schemas

* add ads/ad_groups streams

* update tests and docs

* add a bootstrap file

* update the base domain name for API

* update the base domain name for API

* update Dockerfile libs

* remove unused comments

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* fix UI error with input parameters

* fix problem with updated state

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/spec.py

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add an unit test

* update README.md

* bump version

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-15 19:24:14 +03:00
Benoit Moriceau
ec240edc46 Fix the content type retrieval (#7013)
* Fix the content type retrieval

* Re-add the original MIME type

* Fix CS

* Update logic and add a test

* Auto format

* PR comments
2021-10-15 08:59:23 -07:00
Subodh Kant Chaturvedi
e70e84084f Remove job submitter from scheduleJobsPool (#7044) 2021-10-15 16:26:43 +08:00
Marcos Marx
825e487bb1 enable campaign_metrics stream (#6856) 2021-10-14 21:45:56 -03:00
Iakov Salikov
00760168c3 fix: nav overflow-y (#6953)
* fix: nav overflow-y

* fix: MainContainer mini-height
2021-10-15 03:59:35 +07:00
Baz
cfae84cbb0 🎉 Source Shopify: support oauth flow (#6951) 2021-10-14 22:37:30 +03:00
Christophe Duong
c4620559d7 🎉 Refactor Normalization docker images and upgrade to use dbt 0.21.0 (#6959)
* Split normalization docker images for some connectors with specifics dependencies

* Regenerate (#7003)
2021-10-14 20:29:16 +02:00
Yuhui Shi
d9e8a74d7e Fix column title order in sync catalog (#6862)
* Fix column title order in sync catalog

* Fix formatting issue
2021-10-14 10:18:33 -07:00
Vadym
3adf811726 CDK: add option to source to skip config validation on read/discover (#6978)
* Add connector check_config_against_spec parameter

* Bump CDK version.
Add changelog record.

* Always enable spec check on check cmd

* Update airbyte-cdk/python/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-14 18:31:00 +03:00