1
0
mirror of synced 2026-01-02 21:02:43 -05:00
Commit Graph

710 Commits

Author SHA1 Message Date
Benoit Moriceau
89b7740507 Revert "Revert "Remove configuration API (#18998)" (#19440)" (#19443)
* Revert "Revert "Remove configuration API (#18998)" (#19440)"

This reverts commit 3d0a9deb2c.

* Re add parameters needed by cloud
2022-11-15 15:09:46 -08:00
Jimmy Ma
3d0a9deb2c Revert "Remove configuration API (#18998)" (#19440)
This reverts commit 41f3c0aded.
2022-11-15 09:46:46 -08:00
Benoit Moriceau
41f3c0aded Remove configuration API (#18998)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* extract workspace api

* Extract source definition specification api

* Remove configuration API
2022-11-15 08:27:15 -08:00
Michael Siega
bfa3b5cb6c only persist secrets if we ever hydrated them in workspace webhook config handling (#19352) 2022-11-14 17:42:57 +01:00
Benoit Moriceau
b3db914265 Change where a connection is deleted (#19096)
* Tmp

* Move when the deletion is performed

* Re-enable disable test

* PR comments

* Use cancel

* rename

* Fix test and version check position

* Log exception
2022-11-09 13:05:27 -08:00
Benoit Moriceau
77d22c5e1b Bmoric/extract source definition specification api (#18997)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* extract workspace api

* Extract source definition specification api
2022-11-08 14:40:14 -08:00
Benoit Moriceau
1cdf1ba9e8 Bmoric/extract workspace api (#18996)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* extract workspace api
2022-11-08 10:15:57 -08:00
Benoit Moriceau
b05a5b2a6a Bmoric/extract webbackend api (#18988)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* Fix build
2022-11-08 08:42:32 -08:00
Jimmy Ma
c2b13b6ebc Check disabled connections after protocol update (#18990)
* Add support to clear unsupported protocol flag to StandardSyncPersistence

* Fix internal records scope

* Plug clearUnsupportedProtocolVersionFlag on connector update

* Update UPDATED_AT when clearing protocol flags
2022-11-07 17:25:12 -08:00
Benoit Moriceau
a16ecd6097 Bmoric/extract state api (#18980)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* Add missing binder

* fix type
2022-11-07 14:39:43 -08:00
Benoit Moriceau
29676e1cfa Extract source definition api (#18977)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path
2022-11-07 12:01:13 -08:00
Benoit Moriceau
22efa070c6 Bmoric/extract source api (#18944)
* Extract Operation API

* Extract scheduler API

* Format

* extract source api
2022-11-04 16:54:35 -07:00
Anne
1d809a7539 Add new workspace api endpoint (#18983)
* Add new workspace api endpoint
2022-11-04 14:20:07 -07:00
Benoit Moriceau
15143f7238 Fix oauth controller (#18981) 2022-11-04 13:34:05 -07:00
Davin Chia
e5c3f4bd0d Remove unused job persistence methods. (#18952)
Closes #12823.

Also remove unused dump methods.
2022-11-04 11:29:43 -07:00
Davin Chia
7bb3241781 Progress Bar API Changes. (#18953)
API changes to support the progress bar.

- The eventual idea is for the save_stats route to be called by the workers during replication. Workers will save stats for a job id and attempt number.
- Make modifications to the /jobs/list and the /jobs/get_debug_info routes to also return estimated bytes/records.
We need both estimated metadata, as well as running states to calculate progress bar and throughput.


- add the save_stats route. This is the route that will be called by workers. I've done my best to reuse existing openapi bodies to reduce duplication.
- add the estimatedRecords and estimatedBytes fields to the AttemptStats body. This is part of the AttemptRead and the AttemptStreamStats objects. This eventually filters up to the jobs/list and jobs/get_debug_info objects. This also adds these to all the endpoints that were previously returning stats information. I think the duplicated data is a small issue and don't think it's worth splitting out a new api objects, though I will gladly do so if folks feel strongly.
minor changes to the AttemptApiController to support the new route.
- I've stubbed out the handlers for now since the backend is not yet implemented.
2022-11-04 10:05:07 -07:00
Benoit Moriceau
4e905cabfd Bmoric/extract scheduler api (#18941)
* Extract Operation API

* Extract scheduler API

* Format
2022-11-03 15:43:55 -07:00
Benoit Moriceau
baf6466a19 Extract Operation API (#18928)
* Extract Operation API

* Format
2022-11-03 14:46:52 -07:00
Benoit Moriceau
6b21802f2b Extract open API (#18879)
* Extract open API

* Format

* Fix PMD
2022-11-02 17:44:36 -07:00
Benoit Moriceau
589f6efbfc Extract OAuth API (#18818)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* extract jobs api

* fix test

* format

* Extract logs api

* Add missing declaration

* Fix build

* Tmp

* format and PR comments

* Extract notification API

* re-organize tags

* Extract all Oauth

* Fix PMD
2022-11-02 15:37:15 -07:00
Anne
d26e5bc248 DiscoverSchema endpoints calculates diff and breaking change (#18571)
* update discover schema endpoint to calculate diff
2022-11-02 14:10:07 -07:00
Jimmy Ma
35ceb67fae Validate protocol version on connector update (#18639)
* Add helper class to check protocol version range

* Check ProtocolVersion when modifying a destination definition

* Check ProtocolVersion when modifying a source definition

* Format

* Add UnsupportedProtocolVersion exception

* Rewrite AirbyteProtocolVersionRange as a record

* Format

* Rename exception
2022-11-02 13:44:05 -07:00
Benoit Moriceau
0937b4cd84 Bmoric/extract notification api (#18812)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* extract jobs api

* fix test

* format

* Extract logs api

* Add missing declaration

* Fix build

* Tmp

* format and PR comments

* Extract notification API

* Fix PMD
2022-11-02 12:03:24 -07:00
Benoit Moriceau
3b74857534 Bmoric/extract logs api (#18621)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* extract jobs api

* fix test

* format

* Extract logs api

* Add missing declaration

* Fix build

* format and PR comments

* Fix PMD
2022-11-02 11:42:12 -07:00
Benoit Moriceau
417481e081 Bmoric/extract jobs api (#18524)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* extract jobs api

* fix test

* format

* Add missing declaration
2022-11-02 11:16:41 -07:00
Octavia Squidington III
b299688f3b Bump Airbyte version from 0.40.17 to 0.40.18 (#18827)
Co-authored-by: grishick <grishick@users.noreply.github.com>
2022-11-01 19:35:23 -07:00
Charles
87a0a2a27c hide ConfigPersistence inside ConfigRepository to discourage use (#18803) 2022-11-01 16:51:09 -07:00
Charles
20ab047ae0 Remove the bulk actions from ConfigPersistence (#18800) 2022-11-01 16:50:03 -07:00
Anne
0c183334bf add nonBreakingChangePreference and notifySchemaChanges (#18636)
* add nonBreakingChangePreference and notifySchemaChanges
2022-10-31 12:58:39 -07:00
Benoit Moriceau
e39f882b66 Bmoric/extract healt api (#18523)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* fix test

* format
2022-10-28 13:44:40 -07:00
Benoit Moriceau
abb7efd4c9 extract destination definition specification api (#18521)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* explicit path
2022-10-27 07:51:10 -07:00
Benoit Moriceau
66dc673268 Bmoric/extract destination definition api (#18515)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* format

* format

* explicit path
2022-10-27 07:33:46 -07:00
Benoit Moriceau
b91fab13e3 Bmoric/extract db migration api (#18459)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* restore destination factory initialization

* format
2022-10-27 07:07:38 -07:00
Malik Diarra
05b73999a6 Improve performance of the operation that retrieve all connector for a given definition (#18499) 2022-10-26 15:48:00 -07:00
Parker Mossman
5d4b564389 Persist geography updates (#18501)
* persist geography in connection update

* use ApiPojoConverters

* add test coverage for persisting geography update

* persist geography column for workspace

* use US instead of AUTO in mock data so that tests don't pass due to the database-level default column value

* use ApiPojoConverters instead of Enum.convertTo in workspace handler

* format
2022-10-26 15:10:48 -07:00
Benoit Moriceau
d693b2c0c3 Bmoric/extract destination api (#18502)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* Fix PMD errors

* Add explicit path to the controller

* Extract destiantion API
2022-10-26 13:54:58 -07:00
Benoit Moriceau
1aa20790bb Bmoric/extract connection api (#18409)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* Fix PMD errors

* Add explicit path to the controller
2022-10-26 10:23:06 -07:00
Jonathan Pearlin
657f997aae Update to latest Temporal SDK release (#18492) 2022-10-26 12:54:00 -04:00
Malik Diarra
418e238a2e Optimize listDestinationsForWorkspace route (#18456)
Add a function to fetch all destinations in a workspace
2022-10-26 08:00:14 -07:00
Malik Diarra
c466878fb9 Refactor the buildDestinationRead function (#18446)
Create a new buildDestinationRead function, that function can be called by top-level functions in the handler that
already have a DestinationConnection object.
2022-10-25 15:16:48 -07:00
Xiaohan Song
21431f6297 use api to do jobpersistence query (#18308)
* use api to do jobpersistence query

* renaming some variables

* fix test
2022-10-25 14:27:42 -07:00
Malik Diarra
066f19b84c Refactor SourceHandler:buildSourceRead function (#18430)
This function takes as input a `sourceId` and performs a database call to retrieve the full source information. This refactoring ensures that call sites that already have the full object can bypass that database call. This is especially important for methods that fetch and return a list of sources.
2022-10-25 12:55:38 -07:00
Benoit Moriceau
62eb6b6623 Extract the AttemptApi out of the ConfigurationApi (#18406)
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* format

* Rename to Controller
2022-10-25 09:04:29 -07:00
Benoit Moriceau
7ec6779df3 Bmoric/update connection list with breaking (#18125)
* add schemaChange

* merge conflict

* frontend tests

* tests

* l

* fix source catalog id

* test

* formatting

* move schema change to build backend web connection

* check if actor catalog id is different

* fix

* tests and fixes

* remove extra var

* remove logging

* continue to pass back new catalog id

* api updates

* fix mockdata

* tests

* add schemaChange

* merge conflict

* frontend tests

* tests

* l

* fix source catalog id

* test

* formatting

* move schema change to build backend web connection

* check if actor catalog id is different

* fix

* tests and fixes

* remove extra var

* remove logging

* continue to pass back new catalog id

* api updates

* fix mockdata

* tests

* tests

* optional of nullable

* Tmp

* For diff

* Add test

* More test

* Fix test and add some

* Fix merge and test

* Fix PMD

* Fix test

* Rm dead code

* Fix pmd

* Address PR comments

* RM unused column

Co-authored-by: alovew <anne@airbyte.io>
2022-10-24 16:13:38 -07:00
Malik Diarra
1ee02d416c Improve listSourcesForWorkspace route (#18207) 2022-10-24 14:37:19 -07:00
Octavia Squidington III
73336ff4de Bump Airbyte version from 0.40.16 to 0.40.17 (#18316)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2022-10-21 14:00:39 -07:00
Anne
5082ebfe2a Add SchemaChange to WebBackendConnectionRead object (#17969)
* add schemaChange field to WebBackendConnectionRead and breakingChange to ConnectionRead
2022-10-21 13:08:03 -07:00
Jimmy Ma
cf28f7a01f Add message migration to discover schema (#18205) 2022-10-21 09:32:42 -07:00
Octavia Squidington III
ce0bbf3f75 Bump Airbyte version from 0.40.15 to 0.40.16 (#18268)
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
2022-10-20 13:29:54 -07:00
Xiaohan Song
6b1c5ee384 save queue name into Attempts table (#17921)
* save queue name

* make input nullable because we changed signature

* PR Comments fix
2022-10-19 17:10:37 -07:00