* save
* clean up more usages and remove withRefreshedCatalog
* make webapp use correct endpoint
* add back intercept
* fix acceptance test
* fix log
* remove 'new' from test name
It's difficult today to parse all the logs from tests. Engineers waste time scrolling through the log outputs and looking for the relevant stack trace.
This PR adds an action to generate a JUnit report so devs can understand test results at a glance. This generates 3 reports for each of the main build jobs when the build completes. We leave the frontend build out since this is aggregated by cypress.
See https://github.com/airbytehq/airbyte/pull/15271/checks?check_run_id=7683783016 for an example of how this works.
Use the https://github.com/dorny/test-reporter action and configure this to look at the Jacoco test report output for top level and second level builds. Note that most of the parameters into the action don't work.
Comment out the trap commands to output logs. Though this can be useful for debugging, there is little practical use in the day-to-day, and results in extremely noisy logs.
* sleep 1 second every connector lookup
to prevent running into docker hub API rate limits
* comment on why we sleep
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Google Sheets Destination flushes on every state message
* add PR number to readme
* suggestions
* reverted clean change
* added memory allocation check for records_buffer(stream)
* Update airbyte-integrations/connectors/destination-google-sheets/destination_google_sheets/writer.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* auto-bump connector version
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* set per stream feature flag to true for testing
* add a second table to cdc acceptance tests
* add partial reset test
* format
* add partial reset cdc tests
* test incremental after partial reset
* remove dev image from acceptance test
* fix flag and add comment
* Revert "set per stream feature flag to true for testing"
This reverts commit 164d7da05990268b09e315eb88ff297d3a9f52f4.
* set USE_STREAM_CAPABLE_STATE flag to true in acceptance test script
* call new update endpoint
* use methods in test harness instead
* remove comment
* format
* fix state check in basic acceptance test
* use test info for test name logging
* Add CONTAINER_ORCHESTRATOR=true flag to kube acceptance tests
* Add CONTAINER_ORCHESTRATOR=true flag to GKE acceptance tests too
* fix downtime test
* fix the other orchestrator tests
* move container orchestrator acceptance tests to their own class
* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
* explicitly use buildx for connector publish
* ... and use it
* use docker-container driver
* use `driver docker-container` only for normalization
* Update tools/integrations/manage.sh
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
* Mask certain keys from GitHub output
* add more mask keys
* Read the cwd
* also handle json strings as values
* Test value
* Log at beginning of line
* Testing logs
* Use real mask values
* Handle multiple lines
* Rename docker secrets and parameterize docker user
* Apply suggestions from code review
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
* Bump faker version to test M1 Connector builds
* add PR link to changelog
* Remove birhtday, it doesn't obey seed
* bump readme
* more birthday to remove
* remove image specificity in publish-command
* Set AMI for testing.
* test with including buildx emulators
* auto-bump connector version
* add TODO comment
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
