9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
36 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Edward Gao
|
934acaa137 |
Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""
This reverts commit
|
||
|
Edward Gao
|
52b8cbe39d |
Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)" (#27891)
* Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"
This reverts commit
|
||
|
Edward Gao
|
ba3e39bb0c |
Destination Bigquery: Scaffolding for destinations v2 (#27268)
* copy files from edgao branch * start writing create table statement * add basic unit test setup * create a table, probably * remove outdated todo * derp, one more column * ugh * add partitioning+clustering * use StringSubstitutor * substitutions in updateTable * wip generate update/insert statement * split up into smaller methods * handle json types correctly * rename stuff * more json_query vs _value stuff * minor tweak * super basic test setup * laying foundation for type parsing * more stuff * tweaks * more progress on type parsing * fix json_value stuff? * misc fixes in insert * fix dedupFinalTable * add testDedupRaw * full e2e test * type parsing: gave up and mirrored the dbt code structure to avoid bugs * type parsing - more cleanup * handle column name collisions * handle tablename collisions...? * comments * remove original ns/name from quotedstream * also javadoc * remove redundant method * fix table rename * add incremental append test * add full refresh append test * comment * call T+D sql in a reasonable location for standard inserts * add config option * use config option here * type parsing - fix fromJsonSchema * gate everything * log query + runtime * add spec option temporarily * Raw Table Updates * fix more stuff * first big pass at toDialectType * no quotes * wrap everything in quotes * resolve some TODOs * log sql statement in tests * overwriteFinalTable returns optional * minor clean up * add raw dataset override * try to preserve the original namespace for t+d? * write to the raw table correctly * update todos * write directly to raw table this is kind of dumb because we're still trying to do tmp table operations, and we still don't ack state until the end of the entire sync. * standard inserts write to raw table correctly * imports + log statements * move logs + add comment * explicitly create raw table * move comment to better place * Typing issues * bash attempt * formatting updates * formatting updates * write to the airbyte schema by default unless overriden by config options * standard inserts truncate raw table at start of sync * full refresh overwrite will overwrite correctly! * fix avro record schema parsing * better raw table recreate * rename raw table to match standard inserts * full refresh overwrite does tmp table things * small clean up * small clean up * remove errors entry if no errors * pull out destination config into singleton * clean up singleton stuff * make sure dest config exists when trying to do lookups * avoid stringifying null * quick thoughts on alter table * add basic cdc testcase * tweak cdc test setup * rename raw table to match standard inserts * minor tweak * delete exact sql string assertions * switch to JSON type * minor cleanup * sql whitespace changes * explain cdc deletions * GCS Staging Full Refresh create temp table * assert schema * first out of order cdc test * add another cdc test case (currently failing) * better test structure * make this work * oops, fix test * stop trying to delete deletion records * minor improvements to code+test * enable concurrent test runs on integration test * move stuff to static initializer * extract utility method * formatting * Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been. * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * switch back to empty list; write big assert * minor wording tweaks * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * DestinationConfigTest * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * formatting * remove ParsedType * 🤖 Auto format destination-gcs code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * tests verify every data type * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * full update with all data types * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * move stuff to new base lib * 🤖 Auto format destination-gcs code [skip ci] * Automated Commit - Formatting Changes * 🤖 Auto format destination-bigquery code [skip ci] * fix test * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * asserts in dedupFinalTable * better asserts in dedupRawTable * [wip] test case for all data types * 🤖 Auto format destination-gcs code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * AirbyteTypeTest * Automated Commit - Formatting Changes * remove comments * test chooseOneOf * slightly better test output * Automated Commit - Formatting Changes * add some awful pretty print code * more comment * minor tweaks * verify array/object type * fix test * handle deletions more correctly * test toDialectType * Destinations v2: better namespace handling (#27682) * [wip] better namespace handling * 🤖 Auto format destination-bigquery code [skip ci] * wip also implement in gcs * get gcs working (?) * 🤖 Auto format destination-bigquery code [skip ci] * remove duplicate method * 🤖 Auto format destination-bigquery code [skip ci] * fixed my code style settings * make ci happy? * 🤖 Auto format destination-bigquery code [skip ci] * make ci happy? * remove incorrect test * blank line change * initialize singleton --------- Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> * reset args correctly * Automated Commit - Formatting Changes * more bash stuff * parse implicit structs * initialize singleton in more tests * Automated Commit - Formatting Changes * I missed this namespace handling thing * test more schemas * fix singular types specified in arrays * Automated Commit - Formatting Changes * disable test for unimplemented feature * initialize singleton * remove spec options; changelogs+metadata * randomize namespace * also bump dockerfile * unremove namespace sanitizing in legacy mode * ... disable the correct test * even more unit test fixes! * move integration test to integration tests --------- Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> |
||
|
Ryan Fu
|
769ed3efa7 |
Adds PartialAirbyteMessage overhead & removes serialize/deserialize on CSV writer (#27222)
* No Op change to see if Connectors Base is unrelated * Isolated test to see the memory used after processing CSV writer * Changes record writer * Removed Json.serialize on string * Turn on Snowflake Async * Fixed emittedAt timestamp in milliseconds * Add comments. Undo Evan build changes. * Allow remote debugging. * Adds calc for PartialAirbyteRecordMessage memory overhead * Revert back airbyte build architecture * Passes in the parsed emittedAt to the CSVwriter * Cleans up lingering comments * Performance enhancement - increase the available JVM memory known by the container * Serializes the data earlier to avoid normalization issues and lowers optimal batch size * Passes messageString instead of serializing the data due to memory overhead causes OOM early * Removes memory overhead of record data * Cleans up PR, fixes memory issues & increases throughput by lowering optimal batch size * Add calculation comment & removes procps from docker install * Automated Commit - Format and Process Resources Changes * Turns on Async for testing purposes * Moves up implementation higher up the classes to remove unnecessary no-op * Automated Commit - Format and Process Resources Changes * Adds TODOs for migrating buffers to use only serialized records * Breaks down calc & throws UnsupportedOperationException * Adds TODO for migration all destinations to use the getDataRow(id, formattedString, emittedAt) to avoid unnecessary ser-de overhead --------- Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: ryankfu <ryankfu@users.noreply.github.com> |
||
|
Charles
|
480a43b9ab | grab bag of non-controversial clean up tasks (#26702) | ||
|
Edward Gao
|
67f3cdb125 |
🐛 Destination S3: Fix Parquet LZO compression (#26284)
* Revert "Move hadoop-lzo to test dependency (#21085)"
This reverts commit
|
||
|
Edward Gao
|
ccb6a071a9 |
Avro-based destinations: map airbyte_type: integer to long (#25570)
* map the other integer schema to long * fix test + add test * delete_public_access_block for bucket if public (#25663) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 🐛 Source Facebook Marketing: fix `expected records` for CAT (#25604) * publish normalization (#25591) * publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com> * Bump Airbyte version from 0.44.2 to 0.44.3 * Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (#25588) * add AIRBYTE_ENTRYPOINT env var for kube * amazing, absolute genius * version bump + changelog * derp, no need to publish denormalizeid * fix changelog entry * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * version bumps + changelog * whoops * bump metadata * bump metadatas * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version --------- Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Jeff Cowan (Airbyte)
|
79db9f8e68 |
Clean up destination bases (#25346)
Changes in this refactor PR * Use the proper interface name for the OnStartFunction * Use the proper interface name for the OnCloseFunction * Create and use a proper interface name for the FlushBufferFunction * Create and use a proper interface name for the BufferCreateFunction * Mostly naming consistency changes. These are things caught in static, compile time checks so should be low risk. --------- Co-authored-by: jcowanpdx <jcowanpdx@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Rodi Reich Zilberman
|
0bab1756b8 |
Rename airbyte-config module (#24885)
* rename airbyte-config module * Automated Commit - Formatting Changes * sanity * update import * update import * update script * update script * update script * update script * Automated Change * Automated Change * Automated Change * Automated Change * update awsdatalake icon * point slash commands to new path * sanity * Automated Commit - Formatting Changes * sanity * Automated Change * Automated Change * sanity --------- Co-authored-by: rodireich <rodireich@users.noreply.github.com> |
||
|
Edward Gao
|
0974a05c55 |
S3 destinations: use STANDARD retry mode (#24736)
* use standard retry mode * Automated Change * dest-s3 version bump + changelog * also in redhsift + snowflake * auto-bump connector version * version bumps --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Eugene
|
c4fbc8c9a5 |
S3-Parquet: added handler to process null values in arrays (#23788)
* [22807] Parquet - added handler to null values in arrays |
||
|
Eugene
|
6672b9cb3a |
Destination-S3: Correctly generate int64 values (#23466)
* [17564] Updated s3 avro to use long instead of int --------- Co-authored-by: etsybaev <etsybaev@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Sherif A. Nada
|
e85eda088e | Remove ExtendedNameTrasformer (#23655) | ||
|
Ryan Fu
|
32ae1b0c94 |
Logging recordWriter and onStreamFlush completion (#23360)
* Adds additional logging when flushing buffer and writing records * Removes logging for writeRecord since this will explode log lines * Added logging when uploading records to stage/bucket * Fixes log lines to properly capture when records have been uploaded * Bumps version and fixes logging message to more accurately reflect logic * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Nataly Merezhuk
|
fafffc4607 |
🎉 Destination S3: Add root level flattening option to JSON output (#21355)
* Adds flattening option in the UI and to S3JsonlFormatConfig. * Corrects JsonL Serialized Buffer. * Corrects null value in JsonlFormatConfig and adds tests. * Adds flattening as requirement in spec. * Fixes testDataTypeTestWithNormalization in S3JsonlDestinationAcceptanceTest * Fixes testDataTypeTestWithNormalization in S3JsonlGzipDestinationAcceptanceTest * Corrects connector behavior and makes testDataTypeTestWithNormalization pass. * Delete unnecessarily committed file. * Fixes formatting. * Fixes build. * Adds flattening constants and extracts flattening enum. * Adds flattening type helper to check the flattening value. * Corrects some of the errors caused by refactoring. * Completes refactoring. * Cleans up refactor. * Delets unnecessary file. * Removes additional unnecessary files. * Fix conflict. * Formatting fix. * Removes flattening as a required parameter for Jsonl. * Requested change: removes constant. * Requested change: move Flattening enum out of the CSVFormatConfig. * Requested change: removes override and runs format check. * Requested changes: add lombok and update spec text. * Fix format. * Fixes imports. * Bumps version and adds changelog. * Roll back Dockerfile version update. * Requested changes: explicit imports. * Requested changes: fix failing check_connector_https_url_only. * Corrects http url. * Bumps docker version and corrects changelog date. * auto-bump connector version --------- Co-authored-by: octavia-squidington-iii <octavia-bot@airbyte.io> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Peter Hu
|
e5eac0a4cc |
use published protocol models jar (#22498)
* add airbyte-protocol to deps.toml
* use published protocol jar for platform
* use published protocol jar for connectors
* point at published jar
* fix dep
* bump gcs storage
* fix build failures in standard-source-test
* fix deps
* downgrade alloy db because it is missing strictness tests
* Revert "downgrade alloy db because it is missing strictness tests"
This reverts commit
|
||
|
Cole Snodgrass
|
2e099acc52 |
update headers from 2022 -> 2023 (#22594)
* It's 2023! * 2022 -> 2023 --------- Co-authored-by: evantahler <evan@airbyte.io> |
||
|
Yevhen Sukhomud
|
2226a2e516 |
Base Java S3: Update Avro TimeWithTimezone schema mapping (#21909)
* 21908 Base Java S3: Update Avro TimeWithTimezone schema mapping * 21908 Base Java S3: Formatting * 21908 Base Java S3: fix integration test gcs + S3 * 21908 Base Java S3: fix unit test * 21908 Base Java S3: fix format |
||
|
Ryan Fu
|
aa6afd0aa8 |
🚨 Destination Snowflake: checkpointing flush/commit and emit STATE message 🚨 (#20631)
* Checkpointing flush/commit and emit STATE message * Fixed tests for SerializedBufferingStrategy * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Fixed BufferedStreamConsumerTest to support new logic for checkpointing * Removed cleanup logic that no longer applies with checkpointing changes * Checkpointing flush/commit and emit STATE message * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Resolved BigQuery partitioning tests and parameterized GCS Staging test * Fixed review comments and bumps version number * Definition generation |
||
|
Joe Bell
|
39ec49b9da |
remove failing line (#21740)
Co-authored-by: Topher Lubaway <asimplechris@gmail.com> |
||
|
Joe Bell
|
db37aadf67 |
Improve Authentication Error handling (#21087)
* refactor checks * Add Auth Error Checks * rebase from master * some weird versioning issue * 🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * add substream slicer * add substream and cartesian slicer * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * do not allow path for substream slicer request option * do not show request option for substream slicer * rewrite stream slice field tooltip Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * 🪟🔧 Connector builder: Performance improvements (#20620) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * speed up some bits * more changes * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * review comments * fix more * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * review comment * make sure testing state and form state stay consistent * improve builder errors * remove test state from streamconfig view * remove console log * remove unnecessary positive index check Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Remove workspace helper from fetchConfigActivity (#21048) * Remove workspace helper and replace with workspaceApi * Publish new version of destination-redshift (#21083) * Update changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Progress Bar Read APIs (#20937) Follow up PR to #20787 . Make stats available to the read apis so these are available to the webapp. After this, all that is left is writing these stats as the job progresses. Add the required logic in JobHistoryHandler.java. Took the chance to also rename our internal Attempt models field from id to attemptNumber to better reflect that the field stores not the row's database id, but the job's attempt number. Most of the files changes here are due to that rename. * Rm temporal version (#21045) * Rm temporal version * Remove temporal version * Update the replayed workflow * Format * Fix pmd * 🪟🎉 Connector builder: Available inputs dropdown (#20983) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * available inputs dropdown * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * improve styling * make sure padding is set correctly * make sure focus is set right * comment * use correct state hook * add tooltip to user input button and fix wording for new user input option Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Fixed appstore docs link (#21098) * fill in all default values on switch (#21059) * 🐛Source Freshdesk: Fix schema types (#20416) * Fix schema types * Updated version * Updated acceptance tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * CDK: Add schema inferrer class (#20941) * fix stuff * Update schema_inferrer.py * Update schema_inferrer.py * bump version * review comments * code style * fix formatting * improve tests * Source Facebook Marketing: Docs update (#21105) * Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (partial revert of #20662) (#21082) * Revert "Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)" This reverts commit |
||
|
Eugene
|
2d65d62a65 |
🐛Destination-Snowflake: updated check method to handle more possible s3 and gcs stagings issues (#21450)
* [18312] Destination-Snowflake: updated check method to handle more possible s3 and gcs stagings issues |
||
|
Greg Solovyev
|
1241569ba3 |
Move hadoop-lzo to test dependency (#21085)
* remove hadoop-lzo from gradle dependencies in destination-s3 project * move hadoop-lzo from build time to test time dependencies in base-s3 project |
||
|
Jimmy Ma
|
094aff1520 |
Java connectors use protocol v0 objects (#20404)
* Dupl v0 objects * v0 namespace swap * Introduce some object conversions in the tests * Update source-java-jdbc template * Use v0 objects in db-lib * Fix more protocol object references * Fix arg capture in tests |
||
|
Yevhen Sukhomud
|
1b8376eb79 |
🎉 Update destination-s3 to handle the new data types protocol (#20088)
* Update base-java-s3 to support V1 protocol * Fix formatting * Azur Blob test fix * Destination GCS test fix * java-base-s3 correct test dataset * Destination GCS test fix * Refactor tests comparators from epoch * Remove bignumber from V1 dataset * Clean up * Update json-avro-converter version * Bump version + changelog * Base64 type should be decoded for s3 tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ivica Taseski
|
34d4a71875 |
🎉 New Destination: S3-Glue (#18695)
* destination-s3-glue * reimplement schema generation to use recursion * configure serialization library * include data on update * flatten data field * improve location path * generate s3-glue destination * refactor s3-glue as separate connector * add acceptance tests and cleanup * check field presence * override test image name * add s3-glue readme * format files * add redpanda readme * add s3 glue to source def * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Davin Chia
|
0d4a2bb281 |
Introduce the AirbyteStreamNamespaceNamePair to the DefaultReplicationWorker. (#19360)
#19191 made me realise the DefaultReplicationWorker's metric tracking today has a bug where we aren't accounting for namespace when tracking metrics today. i.e. Streams with the same name and duplicate namespace will merge metrics. While reading the code to figure out a fix, I realised we don't have a good conceptual representation of stream namespace <> name pairs within the platform today. We use a concatenated string. Though this works, it will become harder and harder to read/track as we do more operations that involve namespace i.e. progress bars and column selection. This PR introduces the AirbyteStreamNameNamespacePair object into the platform code to make it more convenient to work with Streams in the future. (Especially if we proceed with the project to make streams a first-class citizen!) The AirbyteStreamNameNamespacePair object was written to deal with the same issue of namespace <> name pair manipulation within the Java destination code. It implements the Comparable interface, which makes it convenient to use for Collections operations. For an example of how this is consumed, see #19361. |
||
|
Liren Tu
|
61c67bada9 |
Databricks destination: introduce personal staging location (#18557)
* Update s3 parquet writer - Rename gcsFileLocation. - Merge local variables. * Implement staging location getter * Add integration test and fix post request * Add comment and format code |
||
|
Edward Gao
|
f90ffb0a12 |
Destination Redshift: handle empty s3 bucket_path (#18434)
* make bucket_path required * check for nonempty path * not required * handle null/empty path * version bump + changelog * fix comment * auto-bump connector version * format Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Edward Gao
|
ba21d9048c | fix formatting + imports (#18457) | ||
|
N-o-Z
|
9b57d972af |
🐛 Fix S3 destination integration tests to use bucket path (#18031)
* Fix S3 destination tests * Add Changelog * CR Fixes * add unit test * version bump * fix dockerfile * auto-bump connector version Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Augustin
|
710c432f00 | fix Connector Base build (#18351) | ||
|
Augustin
|
b564f3eb78 |
Protocol: make supported_sync_modes a required not empty list on AirbyteStream (#15591)
|
||
|
Ryan Fu
|
1d956df434 |
🚨 Add SSL documentation and check logic for S3 Destination 🚨 (#17340)
* Adds logic to fail upon non-deterministic custom S3 endpoint and documentation for insecure settings * Reused config factory settings to a single static variable * Updated error message and example in the spec.json to match expectation of secured endpoint * Added validation check within the base s3 * Integrated AdaptiveDestinationRunner with S3Destination * Reduced visibility for testing and fixed AdaptiveDestinationRunner issue * Adds speicifc secure protocol with S3 and empty endpoint check * Bumps docker version and adds comments and clearer string methods * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Liren Tu
|
d14b4423ea |
Publish R2 destination (#17122)
* Format code * Update doc * Add r2 to build doc * Update readme * Add r2 to destination definitions * Move test resource to base-java-s3 * Remove redundant dependency * Add missing dependency * Format json * Remove s3-destination-base-integration-test setting.gradle root * Add missing lzo dependency * Fix import in destination jdbc |
||
|
Nugusbayev Kanagat
|
3494405078 |
🎉 New Destination: R2 Cloudflare (#15296)
* added destination-r2 spec added base-java-s3 module updated common s3 lib local changed dependency of s3 in all realted modules minor style fixes added common module for s3 integration tests * fix configuration for r2 integration tests * fix configuration for r2 integration tests * added docs for destination-r2 * minor import fixes * mark test as disabled * fixed imports in destination-snowflake * restored styling task * added tests for S3DEstinationConfig * added upload threads count configuration for r2(due to limitations) * deleted parquet format type * Fix import in gcs * Fix import in redshift * Fix import in snowflake * Fix one more import in gcs * Fix one more import in redshift * Fix import in databricks Co-authored-by: Liren Tu <tuliren.git@outlook.com> |