1
0
mirror of synced 2025-12-31 15:03:11 -05:00
Commit Graph

6265 Commits

Author SHA1 Message Date
Alexander Marquardt
e4d3d60ca8 🎉 New Source: Webflow (#13617)
* Added webflow code

* Updated readme

* Updated README

* Added webflow to source_definitions.yaml

* Enhanced documentation for the Webflow source connector

* Improved webflow source connector instructions

* Moved Site ID to before API token in Spec.yaml (for presentation in the UI)

* Addressed comments in PR.

* Changes to address requests in PR review

* Removed version from config

* Minor udpate to spec.yaml for clarity

* Updated to pass the accept-version as a constant rather than parameter

* Updated check_connection to hit the collections API that requires both site id and the authentication token.

* Fixed the test_check_connection to use the new check_connection function

* Added a streams test for generate_streams

* Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK

* Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog

* Fixed a typo in a comment

* Renamed some classes to be more intuitive

* Renamed class to be more intuitive

* Minor change to an internal method name

* Made _get_collection_name_to_id_dict staticmethod

* Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests".
This was caused by Mocked settings from test_source.py leaking into test_streams.py

* format: add double quotes and remove unused import

* readme: remove semantic version naming of connector in build commands

* Updated spec.yaml

* auto-bump connector version

* format files

* add changelog

* update dockerfile

* auto-bump connector version

Co-authored-by: sajarin <sajarindider@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-06-22 08:44:27 -03:00
Tuhai Maksym
aa28d448d8 12708: Add an option to use encryption with staging in Redshift Destination (#13675)
* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated
2022-06-22 14:31:04 +03:00
George Claireaux
dd2d5d0129 in case runners fail to spin up, this needs to run on github-hosted (#13996) 2022-06-22 11:45:00 +01:00
Tuhai Maksym
0d870bd37b 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)
* 6339: debug info

* 6339: not using 'USE' on Azure SQL servers

* 6339: cleanup

* 6339: cleanup2

* 6339: cleanup3

* 6339: versions/changelogs updated

* 6339: merge from master (consolidation issue)

* 6339: dev connector version (for testing in airbyte cloud)

* 6339: code review implementation

* 6339: apply formatting
2022-06-22 13:08:45 +03:00
Serhii Chvaliuk
f69a78c9a4 🎉 Source Github: break point added for workflows_runs stream (#13926)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-22 11:49:04 +03:00
Subodh Kant Chaturvedi
7cd02b0544 skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) 2022-06-22 13:57:44 +05:30
Peter Hu
bcc20b4db1 publish oss for cloud (#13978)
workflow to publish oss artifacts that cloud needs to build against
use docker buildx to create arm images for local development
2022-06-21 14:30:42 -07:00
Brian Lai
be01b476ce Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472)
* write out new request options provider and refactor components and parts of the YAML config

* fix formatting

* pr feedback to consolidate body_data_provider to simplify the code

* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Topher Lubaway
7acf0c54df Adds test for new workflow (#13986)
* Adds test for new workflow

* Adds airbyte repo

* remove testing line
2022-06-21 14:58:27 -05:00
Octavia Squidington III
8c96a5e7f5 Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984)
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
v0.39.23-alpha
2022-06-21 12:46:58 -07:00
Jonathan Pearlin
6ca4017d28 Per-stream state support for Postgres source (#13609)
* WIP Per-stream state support for Postgres source

* Fix failing test

* Improve code coverage

* Make global the default state manager

* Add legacy adapter state manager

* Formatting

* Include legacy state for backwards compatibility

* Add global state manager

* Implement Global/CDC state handling

* Fix test issues

* Fix issue with updated method signature

* Handle empty state case in global state manager

* Adjust to protocol changes

* Fix failing acceptance tests

* Fix failing test

* Fix unmodifiable list issue

* Fix unmodifiable exception

* PR feedback

* Abstract global state manager selection

* Handle conversion between different state types

* Handle invalid conversion

* Rename parameter

* Refactor state manager creation

* Fix failing tests

* Fix failing integration tests

* Add CDC test

* Fix failing integration test

* Revert change

* Fix failing integration test

* Use per-stream for postgres tests

* Formatting

* Correct stream descriptor validation

* Correct permalink

* PR feedback
2022-06-21 15:35:40 -04:00
Adam
a14fbda19f 📄 Postgres source: fix CDC setup order in docs (#13949)
* postgres source: fix CDC setup order docs

* Update docs/integrations/sources/postgres.md

Co-authored-by: Liren Tu <tuliren@gmail.com>
2022-06-21 11:52:06 -07:00
Parker Mossman
9403c28b50 Parker/temporal cloud (#13243)
* switch to temporal cloud client for now

* format

* use client cert/key env secret instead of path to secret

* add TODO comments

* format

* add logging to debug timeout issue

* add more logging

* change workflow task timeout

* PR feedback: consolidate as much as possible, add missing javadoc

* fix acceptance test, needs to specify localhost

* add internal-use only comments

* format

* refactor to clean up TemporalClient and prepare it for future dependency injection framework

* remove extraneous log statements

* PR feedback

* fix test

* return isInitialized true in test
v0.39.22-alpha
2022-06-21 11:37:27 -07:00
Octavia Squidington III
5105c00a5d Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979)
Co-authored-by: Phlair <Phlair@users.noreply.github.com>
2022-06-21 19:37:05 +01:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
Edmundo Ruiz Ghanem
bb68bcd6a7 🪟 🔧 Add auto-fixable linting rules to webapp (#13462)
* Add new eslint rules that fit with our code style and downgrade rules to warn

* allowExpressions in fragment eslint rule

* Enable function-component-definition in eslint and fix styles

* Cleanup lint file

* Fix react/function-component-definition warnings manually

* Add more auto-fixable rules and fix

* Fix functions that require usless returns

* Update array-type rule to array-simple

* Fix eslint errors manually
disable assignmentExpression for arrays in prefer-destructuring rule

* Auto fix new linting issues after rebase
2022-06-21 13:18:38 -04:00
Amruta Ranade
c42576440d Update salesforce.md
Fixed broken link
2022-06-21 10:55:40 -04:00
Vladimir
73c17396e3 🪟 🔧 Add eslint rules for CSS modules (#13952)
* add eslint-plugin-css-modules rules

* Fixes:
- turn on eslint css modules rule as error
- remove unused styles

* add warning message if styled components is used

* Revert "add warning message if styled components is used"

This reverts commit 4e92b8b211.

* replace rule severity with words
2022-06-21 17:21:54 +03:00
Denis Davydov
e8a5c7fffb Source Marketo: process fail during creation of an export job (#13930)
* #9322 source Marketo: process fail during creation of an export job

* #9322 source marketo: upd changelog

* #9322 source marketo: fix unit test

* #9322 source marketo: fix SATs

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 16:15:03 +03:00
Edmundo Ruiz Ghanem
de0cf89fb3 🪟 Updated email invitation flow that enables invited users to set name and create password (#12788)
* First pass accepting email link invitation
* Update Auth service with signInWithEmailLink calls
* Add AcceptEmailInvite component
* Update FirebaseActionRoute to handle sign in mode
* Rename ResetPasswordAction to FirebseActionRoute

* Add create password setp to AcceptEmailInvite component

* Remove continueURL from invite fetch

* Update accept email invite for user to enter both email and password together

* Set name during email link signup

* Update AcceptEmailInvite to send name
* Add updateName to UserService
* Update AuthService to set name during sign up

* Remove steps from AcceptEmailInvite component
Remove setPassword from AuthService

* Add header and title to accept invite page

* Move invite error messages to en file

* For invite link pages, show login link instead of sign up

* Disable name update on sign in via email lnk

* Resend email invite when the invite link is expired

* Fix status message in accept email invite page

* Re-enable set user's name during sign up email invite

* Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name

* Update comments on GoogleAuthService signInWithEmailLink

* Add newsletter and accept terms checkboxes to accept email invite component
* Extract signup form from signup page
* Extract fields from signup form
* Update accept email invite component to use field components from signup form
* Ensure that sign up button is disable until form is valid and security checkbox is checked

* Make error status text color in accept email link red

* Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces

* Add coment around continueUrl param usage in UserService

* Remove usless default case in GoogleAuthService
2022-06-21 08:42:52 -04:00
Anna Lvova
32b5ed736e 🐛 Source: Typeform - Update schema for Responses stream (#13935)
* Upd responses schema

* Upd docs

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 15:38:30 +03:00
Denis Davydov
a46686e317 #13958 Source Stripe: fix configured catalogs (#13959) 2022-06-21 15:20:02 +03:00
Eugene
393397290f 🎉Source-mysql: aligned datatype test (#13945)
* [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing
2022-06-21 14:05:11 +03:00
George Claireaux
b9f79ccdf0 remove test action (#13944) 2022-06-20 17:45:10 +01:00
George Claireaux
546d633337 newaction (#13942) 2022-06-20 17:07:20 +01:00
Octavia Squidington III
689064e373 Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938)
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
v0.39.21-alpha
2022-06-20 18:04:15 +02:00
Baz
efa6042e4c commented out non-relevant tests (#13940) 2022-06-20 18:53:09 +03:00
George Claireaux
30c5d2d9a7 🎉 New Source Dockerhub (#13931)
* init

* implement working source + tests

* add docs

* add docs

* fix bad comments

* Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml

* Update airbyte-integrations/connectors/source-dockerhub/Dockerfile

* Update airbyte-integrations/connectors/source-dockerhub/.dockerignore

* Apply suggestions from code review

* Update docs/integrations/sources/dockerhub.md

* Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py

Co-authored-by: George Claireaux <george@airbyte.io>

* address @Phlair's feedback

* address @Phlair's feedback

* each record is now a Docker image rather than response page

* format

* fix unit tests

* fix acceptance tests

* add icon, definition and generate seed spec

* add requests to requirements

Co-authored-by: sw-yx <shawnthe1@gmail.com>
2022-06-20 15:13:29 +01:00
Yevhen Sukhomud
de057533fb Reverted changes in SshBastionContainer (#13934) 2022-06-20 19:14:30 +07:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753)
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-20 14:56:33 +03:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
Andrii Leonets
8f602aee96 Source MsSql : incr ver to include changes #13854 (#13887)
* incr version

* put PR id

* docker ver
2022-06-20 11:54:40 +03:00
Yevhen Sukhomud
bc03fa4e16 13548 Fixed integration tests source-tidb Mac OS (#13927) 2022-06-20 15:12:58 +07:00
Yevhen Sukhomud
174f15d0c0 13546 Fix integration tests source-postgres Mac OS (#13872)
* 13546 Fix integration tests source-postgres Mac OS
2022-06-20 15:09:01 +07:00
Serhii Chvaliuk
4b116f4889 0.2.5 -> 0.2.6 (#13924)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-20 09:01:39 +03:00
Rajakavitha Kodhandapani
ec8e5bccf2 Docs: fixed the broken links (#13915) 2022-06-19 19:32:49 -03:00
Serhii Chvaliuk
49d181a198 Normalization: Fix incorrect jinja2 macro json_extract_array call (#13894)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-19 13:13:49 +03:00
Ganpat Agarwal
b338014b83 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633)
* start and end date for finacial stream should not be more than 180 days apart

* improve unit tests

* make changes to start date for finance stream

* update tests

* lint changes

* update version to 0.2.22 for source-amazon-seller-partner
2022-06-18 17:27:04 +05:30
Benoit Moriceau
5852989267 Common code to deserialize a state message in the new format (#13772)
* Common code to deserialize a state message in the new format

* PR comments and type changed to typed

* Format

* Add StateType and StateWrapper objects to the model

* Use state wrapper instead of Either

* Switch to optional

* PR comments

* Support array legacy state

* format

Co-authored-by: Jimmy Ma <jimmy@airbyte.io>
2022-06-17 19:01:33 -07:00
Benoit Moriceau
74d16ccb09 Do not hide unexpected errors in the check connection (#13903)
* Do not hide unexpected errors in the check connection

* Fix test
2022-06-17 16:46:42 -07:00
Amruta Ranade
9c1fbea3dd deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) 2022-06-17 15:58:07 -04:00
Amruta Ranade
37b7470139 Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) 2022-06-17 14:32:30 -04:00
Jonathan Pearlin
abe3024511 Update airbyte-protocol.md (#13892)
* Update airbyte-protocol.md

* Fix typo

* Fix prose

* Add protocol reviewers for protocol documentation

* Remove duplicate
2022-06-17 14:06:53 -04:00
Charles
e89846cba9 update api for per stream (#13835) 2022-06-17 10:59:41 -07:00
Evan Tahler
e208d2c3a2 Display connection state in connection setting page (#13394)
* Display Connection State in Setting page

* memoize callback

* rendering and confirmaton

* setState API

* Input validation

* remove JSON step

* rename apiMethod to `updateState`

* test and adjust route

* skip if sync is running

* prevent state update when sync is running

* code editor component

* errors fixed

* scss style

* make linter happy

* Back to monaco editor

* Remove ability to edit state

* Adjust FE code

* Fix CSS problem

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>

* just use PRE to render state for now

Co-authored-by: Tim Roes <tim@airbyte.io>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
2022-06-17 10:52:33 -07:00
Edmundo Ruiz Ghanem
828efb6ddb Update connection update calls to use central utility to ensure connection update has all data (#13564)
* Update connection updates with build update utility
* Add buildConnectionUpdate utility
* Update components that update the connection to use utility when necessary

* Use conection name when saving connection from replication view to prevent override from refreshed catalog

* Improve connection check on ReplicationView onSubmit function
2022-06-17 13:46:22 -04:00
Augustin
72af00a649 octavia-cli: fix workspace not having anonymous_data_collection property (#13869) 2022-06-17 17:48:40 +02:00
George Claireaux
e63458fabb Publishing java connectors with better formatted stacktrace in Trace messages (#13852)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Fix md and lg spacing scss vars (#13886)

* Delete TEMP-testing-command.yml

* auto-bump connector version

* Delete publish-command.yml

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* Update _variables.scss

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
2022-06-17 16:15:24 +01:00
Edmundo Ruiz Ghanem
e03adaa0a6 Fix md and lg spacing scss vars (#13886) 2022-06-17 10:06:26 -04:00
Denis Davydov
a2a4ab5553 #13888 source instagram: fix build (#13889) 2022-06-17 17:03:56 +03:00