yarden/source-mongodb-support-dynamic-arrays
2421 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Nataly Merezhuk
|
fafffc4607 |
🎉 Destination S3: Add root level flattening option to JSON output (#21355)
* Adds flattening option in the UI and to S3JsonlFormatConfig. * Corrects JsonL Serialized Buffer. * Corrects null value in JsonlFormatConfig and adds tests. * Adds flattening as requirement in spec. * Fixes testDataTypeTestWithNormalization in S3JsonlDestinationAcceptanceTest * Fixes testDataTypeTestWithNormalization in S3JsonlGzipDestinationAcceptanceTest * Corrects connector behavior and makes testDataTypeTestWithNormalization pass. * Delete unnecessarily committed file. * Fixes formatting. * Fixes build. * Adds flattening constants and extracts flattening enum. * Adds flattening type helper to check the flattening value. * Corrects some of the errors caused by refactoring. * Completes refactoring. * Cleans up refactor. * Delets unnecessary file. * Removes additional unnecessary files. * Fix conflict. * Formatting fix. * Removes flattening as a required parameter for Jsonl. * Requested change: removes constant. * Requested change: move Flattening enum out of the CSVFormatConfig. * Requested change: removes override and runs format check. * Requested changes: add lombok and update spec text. * Fix format. * Fixes imports. * Bumps version and adds changelog. * Roll back Dockerfile version update. * Requested changes: explicit imports. * Requested changes: fix failing check_connector_https_url_only. * Corrects http url. * Bumps docker version and corrects changelog date. * auto-bump connector version --------- Co-authored-by: octavia-squidington-iii <octavia-bot@airbyte.io> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Marcos Marx
|
aa08378edd | remove heap analytics destination connector (#23325) | ||
|
Ryan Fu
|
f35a0eb8b5 |
Added documentation for suggesting Staging over STD Inserts (#23362)
* Added documentation for suggesting Staging over STD Inserts * Clearer messaging for when using inserts with large datasets Co-authored-by: Edward Gao <edward.gao@airbyte.io> --------- Co-authored-by: Edward Gao <edward.gao@airbyte.io> |
||
|
Evan Tahler
|
81540475a3 |
Fix destination-exasol build and CatalogDefinition generation (#23321)
* fix exasol build * bump docker image * add logo * Remove Catalog downloader * Re-remove config-models * auto-bump connector version --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Christoph Pirkl
|
f9bc114d0d |
🎉 New Destination: Exasol (#21200)
* initial Exasol connector stub * CUrrent version * Exasol connector current version * Exasol connector current version * Exasol connector current version * Exasol connector current version * Remove unwanted changes * Upgrade dependencies * Use Exasol container for integration tests * Purge database before test * Fix integration tests * Adapt to new API * Fix integration tests * Use separate fields for host, port and fingerprint * Deactivate action on forks * Activate action on forks * Update documentation * Adapt type of ID column * Fix Exasol integration tests * Pass certificate fingerprint when creating a DB connection * Use the build directory for temp Exasol credentials * Add reason for overriding testcontainers dependency * Replace deprecated method * Code cleanup * Enable SSH support * Use JdbcDestinationAcceptanceTest instead of generic base class * Remove SSH support (will be added later) * Apply suggestions from code review Co-authored-by: Christoph Kuhnke <github@kuhnke.net> * Improve debugging * Use default name transformer implementation where possible * Add unit tests for name transformer * Add unit tests for destination * Implement review findings by @ckunki * Implement review findings by @itaseskii * Apply suggestions from code review Co-authored-by: Christoph Kuhnke <github@kuhnke.net> * Describe data type limitation * Make schema name mandatory * Remove workaround for bug in ExasolContainer * Add test for executing statements in a transaction Replace Exasol specific code with default method * add destination-exasol to destination_definitions.yaml * add destination-exasol to destination_specs.yaml --------- Co-authored-by: ThomasBestfleisch <thomas.bestfleisch@exasol.com> Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com> Co-authored-by: Christoph Kuhnke <github@kuhnke.net> Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com> |
||
|
Ella Rohm-Ensing
|
aad9d400b1 |
resolve connector dependency workflow missing changelogs (#23314)
* Add changelog entry for destination-clickhouse 0.2.2 * Delete csv.md * local-csv.md -> csv.md |
||
|
Ivica Taseski
|
cc4bad6ad7 |
🎉 Destination Azure Blob Storage: Spill blob when exceeding object size threshold (#21467)
* spill blob when exceeding threshold * fix issues and add integration test * add doc * bump version in dockerfile * auto-bump connector version --------- Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-bot@airbyte.io> Co-authored-by: Sunny Hashmi <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Alex Birdsall
|
362b2ef508 | Trivial grammar fix: add missing "be" (#22615) | ||
|
Eugene
|
a70d6e865a |
Destination-BigQuery: fixed table already exists error (#22497)
* [22178] Destination-bigquery: fixed table already exists error --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Tobias Macey
|
b0a42bfa6c |
destination-s3-glue: Add TableType and fix race condition (#22220)
- The TableType attribute is not getting populated in the Glue catalog which has started to lead to errors when trying to query the table from e.g. Trino. The error message observed is `Cannot invoke "String.equals(Object)" because "tableType" is null` - There is a race condition in initializing the destination connector where a failure occurs if multiple connections attempt to initialize at the same time, because the test table that gets created is statically named. This adds a random suffix to the table to avoid that race condition. - The Airbyte sync ID and emitted_at fields are useful for deduplicating data. This adds those fields to the table definition since they are already included in the records as written. |
||
|
Ryan Fu
|
d21068c989 | Tentatively disallowed normalization for DuckDB (#22528) | ||
|
Simon Späti
|
2bbc4f6f83 |
🎉 New Destination: DuckDB (#17494)
This is the first version of the DuckDB destination. There are potential edge cases that still need to be taken care of. But looking forward to your feedback. |
||
|
Julian
|
dcde74d73f | Docs: Header for S3 Destination Source S3: Wrong header on Step 1 #22331 (#22333) | ||
|
Eugene
|
15fce4f54e |
Destination-s3: updated test and documentation - set minimum required policies (#22058)
* [21331] Destination-s3: updated documentation - set minimum required policies |
||
|
Eugene
|
9d6fe68d6f |
🐛Destination-Snowflake: Added "Table/Stage already exists, but no permissions" handler on Sync action (#21912)
* [21843] Destination-Snowflake: Added "Table/Stage already exists, but no permissions" handler on Sync action --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ryan Fu
|
c0d39006a5 |
Fixed mismatch spec variable and updates default value (#21890)
* Fixed mismatch spec variable and updates default value * Updates spec default values * Bumps connector version and documentation * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ryan Fu
|
aa6afd0aa8 |
🚨 Destination Snowflake: checkpointing flush/commit and emit STATE message 🚨 (#20631)
* Checkpointing flush/commit and emit STATE message * Fixed tests for SerializedBufferingStrategy * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Fixed BufferedStreamConsumerTest to support new logic for checkpointing * Removed cleanup logic that no longer applies with checkpointing changes * Checkpointing flush/commit and emit STATE message * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Resolved BigQuery partitioning tests and parameterized GCS Staging test * Fixed review comments and bumps version number * Definition generation |
||
|
Joe Bell
|
3d92cb6491 |
check for permissions error on snowflake destinations (#21764)
* check for permissions error on snowflake destinations * update tests to use asset throws, make config error exception message clearer * update docs * change error message test * the right write * catch known permissions errors and rethrow as config errors * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ben Church
|
74c8327a1e | Typo Snowlight -> Snowsight (#21834) | ||
|
Greg Solovyev
|
615bce0c05 |
Community PR: Support CMEK or the GCS destination connector (#21682)
Community PR #20351: Support CMEK or the GCS destination connector (#21682) |
||
|
Andy
|
ea5574ac43 | Update limitations (#21354) | ||
|
Joe Bell
|
db37aadf67 |
Improve Authentication Error handling (#21087)
* refactor checks * Add Auth Error Checks * rebase from master * some weird versioning issue * 🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * add substream slicer * add substream and cartesian slicer * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * do not allow path for substream slicer request option * do not show request option for substream slicer * rewrite stream slice field tooltip Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * 🪟🔧 Connector builder: Performance improvements (#20620) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * speed up some bits * more changes * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * review comments * fix more * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * review comment * make sure testing state and form state stay consistent * improve builder errors * remove test state from streamconfig view * remove console log * remove unnecessary positive index check Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Remove workspace helper from fetchConfigActivity (#21048) * Remove workspace helper and replace with workspaceApi * Publish new version of destination-redshift (#21083) * Update changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Progress Bar Read APIs (#20937) Follow up PR to #20787 . Make stats available to the read apis so these are available to the webapp. After this, all that is left is writing these stats as the job progresses. Add the required logic in JobHistoryHandler.java. Took the chance to also rename our internal Attempt models field from id to attemptNumber to better reflect that the field stores not the row's database id, but the job's attempt number. Most of the files changes here are due to that rename. * Rm temporal version (#21045) * Rm temporal version * Remove temporal version * Update the replayed workflow * Format * Fix pmd * 🪟🎉 Connector builder: Available inputs dropdown (#20983) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * available inputs dropdown * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * improve styling * make sure padding is set correctly * make sure focus is set right * comment * use correct state hook * add tooltip to user input button and fix wording for new user input option Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Fixed appstore docs link (#21098) * fill in all default values on switch (#21059) * 🐛Source Freshdesk: Fix schema types (#20416) * Fix schema types * Updated version * Updated acceptance tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * CDK: Add schema inferrer class (#20941) * fix stuff * Update schema_inferrer.py * Update schema_inferrer.py * bump version * review comments * code style * fix formatting * improve tests * Source Facebook Marketing: Docs update (#21105) * Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (partial revert of #20662) (#21082) * Revert "Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)" This reverts commit |
||
|
Eugene
|
2d65d62a65 |
🐛Destination-Snowflake: updated check method to handle more possible s3 and gcs stagings issues (#21450)
* [18312] Destination-Snowflake: updated check method to handle more possible s3 and gcs stagings issues |
||
|
Eugene
|
8ed229ffb6 |
🐛Destination-Bigquery: Added an explicit error message if sync fails due to a config issue (#21144)
* [19998] Destination-Bigquery: Added an explicit error message in sync fails due to a config issue |
||
|
Bob van Luijt
|
1c93be447d | Added some reference links to Weaviate docs (#21449) | ||
|
Jeremy
|
fd9fe9a795 |
Update databend.md (#21179)
* Update databend.md |
||
|
Edward Gao
|
1e44c343af |
JDBC Destinations: improve error message for conflicting streams (#21342)
* catch conflicting streams as configerror * add test * bump version + changelog * derp, fix test setup * derp * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Sam Stoelinga
|
477861552c |
🎉 New Destination: Weaviate (#20094)
* Add Weaviate Destination #20012 * Fix formatting and standards * Fix flake issue * Fix unused client variable * Add support for int based ID fields * Ensure stream name meets Weaviate class reqs * add integration test for using pokemon as source * handle nested objects by converting to json string * create schema for transforming data to weaviate * Add docs for weaviate destination * Remove pokemon-schema external dependency * Remove pikachu integration test external dep * Add large batch test case * add test for second sync * Fix issue with fields starting with uppercase * add more checks to line_break test * Update README for Weaviate * Make batch_size configurable with 100 as default * Add support for providing vectors * Update docs * Add test for existing Weaviate class * Add trying to create schema in check connection * Add support for mongodb _id fields * Add support for providing custom ID * remove unused file * fix flow of is_ready() check * Move standalone functions to utils.py * Support overwrite mode * Add regex based stream_name_class_name conversion * remove unneeded print statement * Add "airbyte_secret" : true to password config * add support for array of arrays * remove unneeded variable declaration * change to MutableMapping since we use del * change name from queued_write to buffered_write * add retry on partial batch error * Fix partial batch retry and add tests * fix ID generation * Clean up recursive retry logic * fix flake tests * ran flake reformat * add definitions Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com> Co-authored-by: itaseski <itaseski@debian-BULLSEYE-live-builder-AMD64> |
||
|
Greg Solovyev
|
8398e648c8 |
Enforce HTTPS (#21182)
* Enforce HTTPS |
||
|
Nataly Merezhuk
|
364973e732 |
🎉 Destination Local CSV: add custom delimiter (#17998)
* Adds delimeter to the spec file. Adds function to get delimeter from the config. * New delimiter works, but only checked raw airbyte tables. * Fixes testWriteFailure(), testWriteSuccess() still broken. * Corrects CSVFormat and now all tests pass. * Implements tab separator. * Corrects tooltip on destination settings page. * Creates CSV file with delimiters and prints it as stirng. * Adds try catch block for assertion. Deletes file after test run. * Removes separate format for tab dleimiter, it is not needed. * Cleans up code. * Adds missing bracket. * Adds files from incorrect rebase. * Corrects imports. * Fixes connectors base build. * Corrects Dockerfile version bump. Adds changelog. * Corrects getProtocolVersion method and makes CSVDataArgumentsProvider class static. * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Greg Solovyev
|
1ed04fd232 |
Destination connector for teradata vantage (rebased community PR) (#21160)
* Destination connector for teradata vantage is implemented * add teradata destination to seed yaml * Add teradata destination to specs seed file Co-authored-by: Satish Chinthanippu <satishchinthanippu@gmail.com> |
||
|
Greg Solovyev
|
56c686440e |
New destination: databend (community PR #19815) (#20909)
* feat: Add databend destination Co-authored-by: hantmac <hantmac@outlook.com> Co-authored-by: josephkmh <joseph@airbyte.io> Co-authored-by: Sajarin <sajarindider@gmail.com> |
||
|
Greg Solovyev
|
7e097b7bb6 |
Publish new version of destination-redshift (#21083)
* Update changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Greg Solovyev
|
606e59eb0d |
Common Jsons: add flag to apply flatten to arrays (#20993)
* add flag to apply flatten to arrays * add additional unit test cases for array flattening * add backward compatibility function * bump dest-redshift version and add changelog Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Adam Bloom <adam-bloom@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> |
||
|
Simon Späti
|
26053ef727 |
📝 Add warning for working with local files and Docker Desktop (/tmp and /private) (#21004)
|
||
|
JJ2788
|
50916e7c2b |
New Desination: Heap Analytics [python cdk] (#20785)
- implement a heap client to load data via the server-side API: https://developers.heap.io/reference/server-side-apis-overview - the connector supports a generic data source, and the api_type determines the output schema. The output schema is dynamic. - users pick the columns that will be loaded to the destination - Consequently, each configured catalog only includes one stream |
||
|
Rodi Reich Zilberman
|
6a68d1f4a7 |
Map number(integer) into an integer rather than a float (#20730)
* Fix failing test * Bigquery-denormalized update spec plus destination_definitions.yaml to fix integration tests * sanity * Update version number and release note * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Amruta Ranade
|
cae63965bd | Deployment docs and sidebar cleanup (#20965) | ||
|
Ryan Fu
|
64254a44f6 |
Destination Redshift: Introduces configurable value for file buffer count (#20879)
* Increased default buffer count and introduces configurable value for destination redshift * Updates logic to ensure filebuffers are at minimum the default number * Reverted changes to increase the file buffer default but allow for users to increase with adequate warnings * Bumps version number and updates changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Joe Reuter
|
2654e050dc |
Fix snowflake destination spec (#20566)
Fix spec to specify a const member for each possible object in a oneOf clause |
||
|
Evan Tahler
|
a55eb7df2d |
Fix CI Dependency Check Failures (#20666)
* pardot * plaid * quickbooks * appfollow * appstore * cloudtrail * clickup * clockify * coda * Coinmarketcap * cooper * dixa * dv-360 * exchange-rates * file * gridly * Hellobaton * kustomer * mailersend * microsoft dataverse * n8n * PersistIq * Survey Sparrow * Twilio Taskrouter * YouTube Analytics Business * Younium * Yahoo Finance Price * Yandex Metrica * Xero * WooCommerce * XKCD * Webflow * US Census API * Qonto * Pivotal Tracker * KVDB * Firestore * Ignore even more connectors * test run * SFTP JSON * cleanup * move pardot changelog * update links * remove testing HACK * Update docs/integrations/sources/dixa.md Co-authored-by: Augustin <augustin@airbyte.io> * Update docs/integrations/sources/kustomer-singer.md Co-authored-by: Augustin <augustin@airbyte.io> * Update docs/integrations/sources/pardot.md Co-authored-by: Augustin <augustin@airbyte.io> * Update docs/integrations/sources/kustomer-singer.md Co-authored-by: Augustin <augustin@airbyte.io> * Update docs/integrations/sources/pardot.md Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Augustin <augustin@airbyte.io> |
||
|
Greg Solovyev
|
a12ca28d72 |
Do not swallow GCS failure when checking BigQuery config (#20501)
* Do not swallow GCS failure when checking BigQuery config * bump version of BigQuery denormalized * add changelog * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Yevhen Sukhomud
|
1b8376eb79 |
🎉 Update destination-s3 to handle the new data types protocol (#20088)
* Update base-java-s3 to support V1 protocol * Fix formatting * Azur Blob test fix * Destination GCS test fix * java-base-s3 correct test dataset * Destination GCS test fix * Refactor tests comparators from epoch * Remove bignumber from V1 dataset * Clean up * Update json-avro-converter version * Bump version + changelog * Base64 type should be decoded for s3 tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ivica Taseski
|
b4179379f9 |
🐛 Destination S3-Glue: fix parsing empty object in schema (#19907)
* fix empty object in schema * treat blank object as string * refactor removed class * bump connector version * auto-bump connector version Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Evan Tahler
|
c9483316a8 |
More destination fixes for ci_check_dependency (#20274)
|
||
|
Edward Gao
|
0164003e1b |
Destination Clickhouse: Publish strict encrypt variant (#20171)
* derp change * bump version |
||
|
Evan Tahler
|
92ad0fdc07 |
Destination doc and warning updates (#20110)
* Doc updates * Bigquery Denormalized * bump faker for change * ignore missing strict-encrypt connectors from ci check * Apply suggestions from code review Co-authored-by: Augustin <augustin@airbyte.io> * Fix MD titles Co-authored-by: Augustin <augustin@airbyte.io> |
||
|
Geoff Genz
|
b7816f4f58 |
🐛 Destination ClickHouse: Update Normalization Docker File (#19573)
* Update ClickHouse normalization docker file * bump destination and norm version * auto-bump connector version * update doc Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Erica Struthers
|
b6a237c25a |
Update mssql.md (#13663)
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> |
||
|
Kris Sikora
|
b7d26812a8 |
Destination:PubSub Add Batching and Ordering configuration (#15705)
* Add Batching and Ordering configuration * Add changelog * fix config for tests * format files * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |