9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
215 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Edward Gao
|
df274b7f40 |
Destination snowflake: destinations v2 scaffolding (#28584)
* deps * scaffolding * logistics * base-jdbc depends on base-td |
||
|
Davin Chia
|
496854caf4 |
Fix Async Framework Race Condition. (#28342)
While running the Snowflake certification test, we noticed NPE on this line, indicating the state id queue was empty. This should never happen as we always keep an 'open' state id with a running counter to associate the next state the source emits to. I eventually realised this happens because of a race condition in the FlushWorkers, where multiple threads flush the same queue not realising the state id the flushing logic was conducted on, was already flushed by another thread. |
||
|
Edward Gao
|
934acaa137 |
Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""
This reverts commit
|
||
|
Charles
|
6c2fb9ff81 |
fix async destination order (#28257)
Partly closes airbytehq/oncall#2437 - a rare condition that can occur when comparator values change mid-comparison. Closes #28154 - a NPE when retrieving from the queue. - For the comparator error, we are unfortunately unable to write a test case to prove the error. It's exceedingly difficult to force the java.lang.IllegalArgumentException: Comparison method violates its general contract! error. We do know the current implementation contains an error this PR fixes. - The NPE error is straight-forward. The peek returns a null and we were not accounting for that before. |
||
|
Joe Bell
|
b03be1b714 |
🐛 Destination Snowflake: Merge old snowflake work (#27935)
* Adds data as JsonNode to pass through, running into memory issues so add JVM args to attach VisualVM * JsonNode * Lowers the optimal batch size to see if this improvements movement * Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirbyteRecord * Fixes config switch when config is not explicitly set (config migration needed) * Adds logic to check if queue has elements before getting timeOfLastMessage * Add PartialSerialisedMessage test. (#27452) * Test deserialise. * Add tests. * Simplify and fix tests. * Format. * Adds tests for deserializeAirbyteMessage * Adds tests for deserializeAirbyteMessage with bad data * Cleans up deserializeAirbyteMessage and throws Exception when invalid message * More code cleanup --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io> * 🤖 Auto format destination-snowflake code [skip ci] * Cleans up code w/o JVM args & rebase * 🤖 Auto format destination-snowflake code [skip ci] * Adds breadcrumb on the STATE message deviation and where the deserialize/serialize is done to unpack * 🤖 Auto format destination-snowflake code [skip ci] * Adds back line formatter removed and comment describing rational for lower batchSize * Bumps Snowflake version and type checks * Added note to remove PartialAirbyteRecordMessage with low resource testing * Automated Commit - Format and Process Resources Changes * Fix issue with multiple namespaces in snowflake not writing to the correct staging schema * Fix issue with multiple namespaces in snowflake not writing to the correct staging schema * remove stage name maniuplating method * update readme * Source Stripe: update credit_notes expected records (#27941) * Source Zendesk Talk: update expected records (#27942) * Source Xero: update expected records (#27943) * Metadata: Persist Registry entries (#27766) * DNC * Update poetry * Update dagster * Apply partition * Get metadata entry * Use helpers * Write registry entry to appropriate location * Delete when registry removed * Update to use new file (broken) * Render registry from registry entries * Run format * Fix plural issue * Update to all metadata file blobs * Fix test * Update to all blobs * Add ignore validation error for version logic * Rename to max_run_request * Pedros review * Ella suggestions Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Update airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/assets/registry_entry.py Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Update naming * Add tests for connector type and deletion * Test safe parse * Format --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Fix Dagster Deploy Failure (#27955) * Add pydantic * Add pydantic to orchestration deploy pipeline * 🐛 Source Jira: update expected records (#27951) * Source Jira: update expected records * Update issues expected records * Source Zendesk Chat: update expected records (#27965) * 🐛 Source Pipedrive: update expected records (#27967) * 🐛 Source Pinterest: update expected records (#27964) * ✨ Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607) * Add stream for sponsored brands v3 * Add new stream Portfolios * Source Google Search Console: added discover and googleNews to searchType (#27952) * added discover and googleNews to searchType * updated changelog * fixed types for streams * 🎉 Source Instagram: Improve, refactor `STATE` management (#27908) * add test for enabling * update versions * fix test * update other snowflake loading method types * remove standard --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: ryankfu <ryankfu@users.noreply.github.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com> Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> |
||
|
Edward Gao
|
52b8cbe39d |
Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)" (#27891)
* Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"
This reverts commit
|
||
|
Cynthia Yin
|
98a0dad893 | Destinations V2: change singleton log message level from warn to debug (#27887) | ||
|
Edward Gao
|
ba3e39bb0c |
Destination Bigquery: Scaffolding for destinations v2 (#27268)
* copy files from edgao branch * start writing create table statement * add basic unit test setup * create a table, probably * remove outdated todo * derp, one more column * ugh * add partitioning+clustering * use StringSubstitutor * substitutions in updateTable * wip generate update/insert statement * split up into smaller methods * handle json types correctly * rename stuff * more json_query vs _value stuff * minor tweak * super basic test setup * laying foundation for type parsing * more stuff * tweaks * more progress on type parsing * fix json_value stuff? * misc fixes in insert * fix dedupFinalTable * add testDedupRaw * full e2e test * type parsing: gave up and mirrored the dbt code structure to avoid bugs * type parsing - more cleanup * handle column name collisions * handle tablename collisions...? * comments * remove original ns/name from quotedstream * also javadoc * remove redundant method * fix table rename * add incremental append test * add full refresh append test * comment * call T+D sql in a reasonable location for standard inserts * add config option * use config option here * type parsing - fix fromJsonSchema * gate everything * log query + runtime * add spec option temporarily * Raw Table Updates * fix more stuff * first big pass at toDialectType * no quotes * wrap everything in quotes * resolve some TODOs * log sql statement in tests * overwriteFinalTable returns optional * minor clean up * add raw dataset override * try to preserve the original namespace for t+d? * write to the raw table correctly * update todos * write directly to raw table this is kind of dumb because we're still trying to do tmp table operations, and we still don't ack state until the end of the entire sync. * standard inserts write to raw table correctly * imports + log statements * move logs + add comment * explicitly create raw table * move comment to better place * Typing issues * bash attempt * formatting updates * formatting updates * write to the airbyte schema by default unless overriden by config options * standard inserts truncate raw table at start of sync * full refresh overwrite will overwrite correctly! * fix avro record schema parsing * better raw table recreate * rename raw table to match standard inserts * full refresh overwrite does tmp table things * small clean up * small clean up * remove errors entry if no errors * pull out destination config into singleton * clean up singleton stuff * make sure dest config exists when trying to do lookups * avoid stringifying null * quick thoughts on alter table * add basic cdc testcase * tweak cdc test setup * rename raw table to match standard inserts * minor tweak * delete exact sql string assertions * switch to JSON type * minor cleanup * sql whitespace changes * explain cdc deletions * GCS Staging Full Refresh create temp table * assert schema * first out of order cdc test * add another cdc test case (currently failing) * better test structure * make this work * oops, fix test * stop trying to delete deletion records * minor improvements to code+test * enable concurrent test runs on integration test * move stuff to static initializer * extract utility method * formatting * Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been. * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * switch back to empty list; write big assert * minor wording tweaks * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * DestinationConfigTest * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * formatting * remove ParsedType * 🤖 Auto format destination-gcs code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * tests verify every data type * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * full update with all data types * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * move stuff to new base lib * 🤖 Auto format destination-gcs code [skip ci] * Automated Commit - Formatting Changes * 🤖 Auto format destination-bigquery code [skip ci] * fix test * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * 🤖 Auto format destination-gcs code [skip ci] * asserts in dedupFinalTable * better asserts in dedupRawTable * [wip] test case for all data types * 🤖 Auto format destination-gcs code [skip ci] * 🤖 Auto format destination-bigquery code [skip ci] * AirbyteTypeTest * Automated Commit - Formatting Changes * remove comments * test chooseOneOf * slightly better test output * Automated Commit - Formatting Changes * add some awful pretty print code * more comment * minor tweaks * verify array/object type * fix test * handle deletions more correctly * test toDialectType * Destinations v2: better namespace handling (#27682) * [wip] better namespace handling * 🤖 Auto format destination-bigquery code [skip ci] * wip also implement in gcs * get gcs working (?) * 🤖 Auto format destination-bigquery code [skip ci] * remove duplicate method * 🤖 Auto format destination-bigquery code [skip ci] * fixed my code style settings * make ci happy? * 🤖 Auto format destination-bigquery code [skip ci] * make ci happy? * remove incorrect test * blank line change * initialize singleton --------- Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> * reset args correctly * Automated Commit - Formatting Changes * more bash stuff * parse implicit structs * initialize singleton in more tests * Automated Commit - Formatting Changes * I missed this namespace handling thing * test more schemas * fix singular types specified in arrays * Automated Commit - Formatting Changes * disable test for unimplemented feature * initialize singleton * remove spec options; changelogs+metadata * randomize namespace * also bump dockerfile * unremove namespace sanitizing in legacy mode * ... disable the correct test * even more unit test fixes! * move integration test to integration tests --------- Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> |
||
|
Augustin
|
5b8200181c | connectors-ci: deprecate slash test (#27200) | ||
|
Ryan Fu
|
769ed3efa7 |
Adds PartialAirbyteMessage overhead & removes serialize/deserialize on CSV writer (#27222)
* No Op change to see if Connectors Base is unrelated * Isolated test to see the memory used after processing CSV writer * Changes record writer * Removed Json.serialize on string * Turn on Snowflake Async * Fixed emittedAt timestamp in milliseconds * Add comments. Undo Evan build changes. * Allow remote debugging. * Adds calc for PartialAirbyteRecordMessage memory overhead * Revert back airbyte build architecture * Passes in the parsed emittedAt to the CSVwriter * Cleans up lingering comments * Performance enhancement - increase the available JVM memory known by the container * Serializes the data earlier to avoid normalization issues and lowers optimal batch size * Passes messageString instead of serializing the data due to memory overhead causes OOM early * Removes memory overhead of record data * Cleans up PR, fixes memory issues & increases throughput by lowering optimal batch size * Add calculation comment & removes procps from docker install * Automated Commit - Format and Process Resources Changes * Turns on Async for testing purposes * Moves up implementation higher up the classes to remove unnecessary no-op * Automated Commit - Format and Process Resources Changes * Adds TODOs for migrating buffers to use only serialized records * Breaks down calc & throws UnsupportedOperationException * Adds TODO for migration all destinations to use the getDataRow(id, formattedString, emittedAt) to avoid unnecessary ser-de overhead --------- Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: ryankfu <ryankfu@users.noreply.github.com> |
||
|
Charles
|
4ac62f3b4f |
Async Snowflake Destination (#26703)
* snowflake at end of coding retreat week * turn off async snowflake * add comment * Fixed test to point to the ShimMessageConsumer instead of AsyncMessageConsumer * Automated Change --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io> Co-authored-by: ryankfu <ryankfu@users.noreply.github.com> |
||
|
Charles
|
223f698516 | encapsulate trigger logic for flush workers (#26706) | ||
|
Charles
|
480a43b9ab | grab bag of non-controversial clean up tasks (#26702) | ||
|
Charles
|
567f839834 | Serialized Message Consumer (#26700) | ||
|
Edward Gao
|
cf2ded2bbb |
Destination Bigquery: small tweak to clarify logs (#26585)
* make logs less misleading * version bumps + changelog * tweak wording |
||
|
Augustin
|
fd3655707e |
connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489)
|
||
|
Charles
|
5f3ed16408 |
hide queue inside MemoryBoundedLinkedBlockingQueue (#26375)
Hiding the actual java Queue has an inner class to avoid the chance that someone tries to use native queue methods that we haven't overridden. Good thing that I did this too, because one of the changes we made during hack dayz wasn't reflected in our current feature branch. We need to override poll(time, unit) not just poll. This PR makes sure we won't make that mistake again! |
||
|
Davin Chia
|
1ed32e55bc |
Async Code V0: Flush Worker Improvements. (#26384)
Incorporate the changes from #26178 . - Update queue method from setMaxMemory to addMaxMemory. - Update work retrieval logic to only assign work - if there are free worker threads - to account for in-progress worker threads |
||
|
Davin Chia
|
a7442e3c2d |
Async Destination V0: Async Stream Consumer (#26366)
Follow up after #26324 . Introduce the AsyncStreamConsumer. After this, one more PR to add the Staging Consumer changes in. |
||
|
Davin Chia
|
988ce24b3f |
Async Destination V0 - Split up BufferManager (#26331)
Follow up to #26324 - here we split up the BufferManager and add tests and comments. - Split up the buffer manager class into -> BufferManager, BufferEnqueue and BufferDequeue. - Move all buffer related code to the buffers package. - Rename test classes to match this split. - Add java docs and tests as part of this split. - Simplify the BufferDequeue interface to return a set streams representing the buffered streams instead of the underlying map of buffers. This lets us keep the memory queue package private. - all getYMethods now return Optionals for better error handling. This would have resulted in NPEs previously. |
||
|
Augustin
|
80032f73f9 | connectors-ci: deprecate slash publish (#25865) | ||
|
Davin Chia
|
8bfbef23cb |
Async Code V0 (#26324)
Split out the smallest set of reasonable changes from #26086 . My goal was to split out the interface, as well as show how the interface it's meant to be used. Follow up PRs: - Split out classes from BufferManager and add more tests there. - Add in the AsyncConsumer with tests. - Add in the StagingConsumer factory. |
||
|
Augustin
|
5c5eab0308 | connectors-ci: fix postgres integration testing (#25942) | ||
|
Jeff Cowan (Airbyte)
|
79db9f8e68 |
Clean up destination bases (#25346)
Changes in this refactor PR * Use the proper interface name for the OnStartFunction * Use the proper interface name for the OnCloseFunction * Create and use a proper interface name for the FlushBufferFunction * Create and use a proper interface name for the BufferCreateFunction * Mostly naming consistency changes. These are things caught in static, compile time checks so should be low risk. --------- Co-authored-by: jcowanpdx <jcowanpdx@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Jonathan Pearlin
|
a38af089e9 |
Track stream status in source (#24971)
* WIP Track stream status in source * Revert formatting * Revert formatting changes * Remove unnecessary file * Automated Change * Automated Change * Use new stream status trace message * Rename class * Remove unnecessary import * Formatting * Add tests * Fix compile issues * Automated Commit - Formatting Changes * Remove TODO * Fix compilation error * Split STOPPED into INCOMPLETE and COMPLETE * Remove unused import * Changelog updates for source-postgres * Remove unused import * auto-bump connector version --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Edward Gao
|
7abda87840 |
destination bigquery: run normalization inside container if env var is set (#25097)
* super hacky start * also check that we're writing * v0 convert normalization logs to airbytemessage * add start+end logs * aggregate errors into a single trace? * pipefail; quick tweaks to log parser * make spotbugs happy * more comments, uncomment env var check * copy in SentryExceptionHelper * final fixes * write tests + fix bugs * move to base-java * remove outdated comment * fix spotbugs * Automated Change * minor version bump * changelog * fix behavior when env var not set * run normalization even if destination fails * better logic * better logging * oops * move to base-java * rebump version * Automated Change * auto-bump connector version * wtf how did this work previously * auto-bump connector version --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Rodi Reich Zilberman
|
0bab1756b8 |
Rename airbyte-config module (#24885)
* rename airbyte-config module * Automated Commit - Formatting Changes * sanity * update import * update import * update script * update script * update script * update script * Automated Change * Automated Change * Automated Change * Automated Change * update awsdatalake icon * point slash commands to new path * sanity * Automated Commit - Formatting Changes * sanity * Automated Change * Automated Change * sanity --------- Co-authored-by: rodireich <rodireich@users.noreply.github.com> |
||
|
Rodi Reich Zilberman
|
cd928a7844 |
Fix all tests to pass on local and CI environments (#24683)
* test docker behavior on CI env * Automated Change * test docker behavior on CI env * Make all unit and integration tests in source-postgres pass locally * Fix mysql ssh integration test * Fix failing test * Fix source-mssql build * source-mssql runss tests locally. Fix compilation errors --------- Co-authored-by: rodireich <rodireich@users.noreply.github.com> |
||
|
Edward Gao
|
05860064f8 |
Staging destinations: Fail fast on error during periodic checkpoint (#24671)
* rethrow exception for fail fast * version bumps + changelog * auto-bump connector version * bump versions * regenerate --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Lake Mossman
|
b1fbd6f33f |
Simplify Github and Postgres forms #2 (#24255)
* add grouping and collapsing fields to postgres source * add auth group to github source connector * revert postgres field order changes and adjust group of schemas field * inject group into ssh tunnel spec for postgres only, through overloaded methods * Automated Change * bump Dockerfile versions and update changelogs * bump strict encrypt version as well * fix postgres acceptance test * fix acceptance test again * fix all postgres acceptance tests * add newline * undo other changes to postgres readme file * add security group to tunnel_method in expected_spec.json * bump version of strict encrypt * manually bump versions in seed files --------- Co-authored-by: lmossman <lmossman@users.noreply.github.com> |
||
|
oneshcheret
|
f6bcc4914f |
Postgres source: add integration with data dog (#21533)
* Source postgres: add dd for env running locally * Source postgres: add dd for running in cloud * auto-bump connector version * Source postgres: bump postgres strict-encrypt version * Source postgres: filter datadog agent env variables just for postgres source * Source postgres: format * Source postgres: clean code * Source postgres: pass java opts for all connectors * Source postgres: temp removing dd agent from image * Source postgres: add dd agent to image * Source postgres: temp revert adding dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp removing hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: rename to java_opts and pass data dog host * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp adding trace to integration runner * Source postgres: temp adding trace to integration runner * Source postgres: bump postgres source dd version * Source postgres: bump postgres source dd version * Source postgres: revert temp changes * Source postgres: merge with master * Automated Commit - Formatting Changes * Source postgres: move dd java agent to base java * Source postgres: move dd java agent to base java * Source postgres: clean up * Source postgres: clean up * Automated Change * Source postgres: clean up * Source postgres: bump version * Source postgres: bump version for test * Source postgres: temp bump version * Source postgres: bump version * Automated Change * Source postgres: bump version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: sashaNeshcheret <sashaNeshcheret@users.noreply.github.com> |
||
|
Lake Mossman
|
2d3c48da8d |
Revert "Simplify postgres and GitHub forms (#24127)" (#24219)
This reverts commit
|
||
|
Lake Mossman
|
fcc80cb5be |
Simplify postgres and GitHub forms (#24127)
* add grouping and collapsing fields to postgres source * add auth group to github source connector * revert postgres field order changes and adjust group of schemas field * inject group into ssh tunnel spec for postgres only, through overloaded methods * Automated Change * bump Dockerfile versions and update changelogs * bump strict encrypt version as well * fix postgres acceptance test * fix acceptance test again --------- Co-authored-by: lmossman <lmossman@users.noreply.github.com> |
||
|
Ryan Fu
|
85391864f7 |
Ryan/periodic buffer flush (#23931)
* Added support for periodic buffer flush with tests and uses env variable * Improves code readability and encapulates testing logic * Removed demo changes and created const for tests * Updated constructor to reuse method signature * Increases Snowflake parallel integration forks * Bumps version number, fixes linting issues and constant format * Generate seed spec |
||
|
Sherif A. Nada
|
e85eda088e | Remove ExtendedNameTrasformer (#23655) | ||
|
Ryan Fu
|
32ae1b0c94 |
Logging recordWriter and onStreamFlush completion (#23360)
* Adds additional logging when flushing buffer and writing records * Removes logging for writeRecord since this will explode log lines * Added logging when uploading records to stage/bucket * Fixes log lines to properly capture when records have been uploaded * Bumps version and fixes logging message to more accurately reflect logic * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Rodi Reich Zilberman
|
49f97f1142 |
Integration Branch for DB/DW Sources team for Feb '23 code freeze (#23185)
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358) * use a safer method for parsing a BigInteger cursor value * Add testing * fix format change * Fix failing integration tests * Try removing the failing incremental test * Try removing the failing incremental test * Fix failing test * Add metadata to connector logs (log level, class name, method name and line number) (#23105) * Issue #17861 Add labels, class, method name and line numbers to connector logs * Refactored unit test * fix for warning about UTF8 charset in test class --------- Co-authored-by: prateekmukhedkar <prateek@airbyte.io> * Update docker image and release notes * auto-bump connector version * manually bump version on spec --------- Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com> Co-authored-by: prateekmukhedkar <prateek@airbyte.io> Co-authored-by: Sergio Ropero <sergio@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Peter Hu
|
e5eac0a4cc |
use published protocol models jar (#22498)
* add airbyte-protocol to deps.toml
* use published protocol jar for platform
* use published protocol jar for connectors
* point at published jar
* fix dep
* bump gcs storage
* fix build failures in standard-source-test
* fix deps
* downgrade alloy db because it is missing strictness tests
* Revert "downgrade alloy db because it is missing strictness tests"
This reverts commit
|
||
|
Cole Snodgrass
|
2e099acc52 |
update headers from 2022 -> 2023 (#22594)
* It's 2023! * 2022 -> 2023 --------- Co-authored-by: evantahler <evan@airbyte.io> |
||
|
Cole Snodgrass
|
f45718d850 |
remove commons-cli dependency from integrations (#22525)
* remove commons-cli dependency from integrations * re-add commons-cli dependency, used in other places |
||
|
Ryan Fu
|
aa6afd0aa8 |
🚨 Destination Snowflake: checkpointing flush/commit and emit STATE message 🚨 (#20631)
* Checkpointing flush/commit and emit STATE message * Fixed tests for SerializedBufferingStrategy * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Fixed BufferedStreamConsumerTest to support new logic for checkpointing * Removed cleanup logic that no longer applies with checkpointing changes * Checkpointing flush/commit and emit STATE message * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging (#21028) * Updates BigQuery to support checkpointing and consolidates method naming for uploading from staging * Updated messages to reflect method changes * Updates createTable to include mimic replication by calling createPartitionTable and removes unused copyIntoTargetTable * Updates the COPY INTO methods to match writing to table * Fixed comments and non-executed path * Resolved BigQuery partitioning tests and parameterized GCS Staging test * Fixed review comments and bumps version number * Definition generation |
||
|
oneshcheret
|
438c1ebc19 |
Mysql source: handle ssh timeout exception (#20593)
* Ssh wrapped source: mark ssh timeout exception as config error. * Ssh wrapped source: move to single catch block * Ssh wrapped source: code format * Source postgres: update display message * Source mysql: format * Source mysql: bump versions * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Joe Bell
|
db37aadf67 |
Improve Authentication Error handling (#21087)
* refactor checks * Add Auth Error Checks * rebase from master * some weird versioning issue * 🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * add substream slicer * add substream and cartesian slicer * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * do not allow path for substream slicer request option * do not show request option for substream slicer * rewrite stream slice field tooltip Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * 🪟🔧 Connector builder: Performance improvements (#20620) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * speed up some bits * more changes * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * review comments * fix more * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * review comment * make sure testing state and form state stay consistent * improve builder errors * remove test state from streamconfig view * remove console log * remove unnecessary positive index check Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Remove workspace helper from fetchConfigActivity (#21048) * Remove workspace helper and replace with workspaceApi * Publish new version of destination-redshift (#21083) * Update changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Progress Bar Read APIs (#20937) Follow up PR to #20787 . Make stats available to the read apis so these are available to the webapp. After this, all that is left is writing these stats as the job progresses. Add the required logic in JobHistoryHandler.java. Took the chance to also rename our internal Attempt models field from id to attemptNumber to better reflect that the field stores not the row's database id, but the job's attempt number. Most of the files changes here are due to that rename. * Rm temporal version (#21045) * Rm temporal version * Remove temporal version * Update the replayed workflow * Format * Fix pmd * 🪟🎉 Connector builder: Available inputs dropdown (#20983) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * available inputs dropdown * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * improve styling * make sure padding is set correctly * make sure focus is set right * comment * use correct state hook * add tooltip to user input button and fix wording for new user input option Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * Fixed appstore docs link (#21098) * fill in all default values on switch (#21059) * 🐛Source Freshdesk: Fix schema types (#20416) * Fix schema types * Updated version * Updated acceptance tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * CDK: Add schema inferrer class (#20941) * fix stuff * Update schema_inferrer.py * Update schema_inferrer.py * bump version * review comments * code style * fix formatting * improve tests * Source Facebook Marketing: Docs update (#21105) * Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (partial revert of #20662) (#21082) * Revert "Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)" This reverts commit |
||
|
Akash Kulkarni
|
63e4482a5f |
Source Postgres : Emit estimate trace messages for non-CDC mode (#20783)
* Emit estimate trace messages * Update PostgresQueryUtils.java * Remaining merge conflicts * Code cleanip * Address comments * Formatting * Cleanup * Addressing comments * Bump version + documentation * Update strict-encrypt Dockerfile * Unpublish * Merge conflicts * Update Dockerfile * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Benoit Moriceau
|
954ca75517 |
Avoid parsing a message if this message is too big (#21090)
* Tmp * Format * TMP * TMP * Inject max memory * Clean up * Improve error message * PR comments * Unrelated changes * Fix pmd |
||
|
Ryan Fu
|
64254a44f6 |
Destination Redshift: Introduces configurable value for file buffer count (#20879)
* Increased default buffer count and introduces configurable value for destination redshift * Updates logic to ensure filebuffers are at minimum the default number * Reverted changes to increase the file buffer default but allow for users to increase with adequate warnings * Bumps version number and updates changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
Ryan Fu
|
d03757a3f8 |
Checkpointing: documentation of lifecycle of how data flows through destination connectors (#20118)
* Adds javadoc comments to files that touch writing data from destination connector into the destination * Updates comments for copyTableQuery * Fixed linter issues * Fixed remaining formatting issues with javadoc paragraphs |
||
|
Jimmy Ma
|
094aff1520 |
Java connectors use protocol v0 objects (#20404)
* Dupl v0 objects * v0 namespace swap * Introduce some object conversions in the tests * Update source-java-jdbc template * Use v0 objects in db-lib * Fix more protocol object references * Fix arg capture in tests |
||
|
oneshcheret
|
9366b12638 |
Mysql source: mark unknown column exception as config error (#20289)
* Source mysql: mark unknown column exception as config error * Source mysql: format code * Source mysql: bump version * Source mysql: bump version * Source mysql: bump spec and definition version * Source mysql: bump spec and definition version * Source mysql: revert spec and definition version with unfamiliar changes * Source mysql: bump spec and definition version * Source mysql: revert manually bumped spec and definition version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> |
||
|
oneshcheret
|
293075ef53 |
Postgres source: handle terminate connection exception with test. (#19887)
* Source postgres: catch termination connection PSQLException * Source postgres: move common code to util method * Source postgres: clean code * Source postgres: review fixes and added unit tests * Source postgres: clean code * Source postgres: bump version * Source postgres: format * Source postgres: format * Source postgres: bump version * Source postgres: fix failing test and bump version * Source mysql: make message visible for tests * Source postgres: reformat import Co-authored-by: VitaliiMaltsev <39538064+VitaliiMaltsev@users.noreply.github.com> Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> |