* Adding Google Cloud Storage as destination
* Removed few comments and amended the version
* Added documentation in docs/integrations/destinations/gcs.md
* Amended gcs.md with the right pull id
* Implemented all the fixes requested by tuliren as per https://github.com/airbytehq/airbyte/pull/4329
* Renaming all the files
* Branch alligned to S3 0.1.7 (with Avro and Jsonl). Removed redundant file by making S3 a dependency for GCS
* Removed some additional duplicates between GCS and S3
* Revert changes in the root files
* Revert jdbc files
* Fix package names
* Refactor gcs config
* Format code
* Fix gcs connection
* Format code
* Add acceptance tests
* Fix parquet acceptance test
* Add ci credentials
* Register the connector and update documentations
* Fix typo
* Format code
* Add unit test
* Add comments
* Update readme
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Marco Fontana <marco.fontana@sohohouse.com>
Co-authored-by: marcofontana.ing@gmail.com <marcofontana.ing@gmail.com>
Co-authored-by: Marco Fontana <MaxwellJK@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
* pre-PR
* add git config
* format
* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt
upd requirements.txt remove extra
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py
backoff time int to float (btw real return type in headers is integer)
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* requested changes
* fix newline absence && rm unnecessary temp file
* url_base to property
* rm extra var coming property
* rm extra var coming property
* save
* finishing updating the documentation
* forgotten definition
* add nullable to pass the test
* fix date in the log
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* incremental sync, acceptance test
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* incremental sync, acceptance test
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* updated slices and api limits, added validation for input dates
* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class
* added input param 'env' to support production and sandbox envs
* added support for sandbox option, updated pattern for optional end date option
* added github secrets
* added support for sandbox option, updated pattern for optional end date option
* fixed Copyright date, removed debug mesages
* added docs
* fix for test failure - The sync should produce at least one STATE message
* removed optional parameter 'end_date'
* removed detailed info about balances schema
* Delete employees.json
* Delete customers.json
* Added requests_per_minute rate limit
* added unit tests, added custom backoff
* added test for stream slices with stream state
* removed comments
* updated docs pages
* fixed format for json files
* fixed types in schemas and link to the schema. fixed primary key for Transactions stream
* updated stream slices
* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method
* added CHANGELOG.md
* Added build seeds
* fixed closing double quotation mark
* added paypal entry in builds.md
* add fixture helper
* added paypal transaction generator script
* fixed styling
* maximum allowed start_date is extracted from API response now.
* fixed schemas
* fixed schemas - removed datetime
* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.
* added possibility to specify additional properties
Co-authored-by: Sherif Nada <snadalive@gmail.com>
* add base files
* upd base
* save
* save sample files
* save & todo resolve state
* save the stage
* save the stage
* pre-gradle save
* fix catalogs
* merge && format
* normal stream slices
* apply requested changes
* requested_changes
* postfix
* update comment
* expand question stream add page_id
* upd typing once + rm missed inactual todo
* upd: caching with temp file
* upd requirements (requested)
* latest requested fixes
* posttext fix retn tempfile
* apply changes && comment
* newly requested changes
* return back spec -> changes to be set in new issue
* merge && usage comment
* add unit_test for get_updated_state function
* add simple date test
* parametrized (?) unittest
* upd comment on record_mode usage
* replace config with custom var
* pytest mark parametrized use
* rm unneeded var
* upd tests (requested)
* merge && upd texts
* add env airbyte_entrypoint
* Bump mysql destination version to pick up normalization
* Also change the publish command to run on ec2-runners to try and avoid build errors with gradle unable to find the right volume.
Co-authored-by: Davin Chia <davinchia@gmail.com>
* add comments and reframe bug-report issue
* add suggestion using details tag for log
* change abhi comments
* add more example os vrson/instance
* correct tag
* remove hidden comments and add submit pr
* correct some points
* correct details log block
* introduce automatic migration at the startup of server
* handle versions with non-zero patch
* it works!!!
* add dummy data
* cleanup orphan configs
* add more assertions
* format + add comments
* move migration acceptance test to acceptance test directory
* add automatic migration test to the build
* address review comments
* missed out on these
* format
* add more assertions
* format
* fix test
* format
* use default port for temporal
* move seed to server + introduce atomice replacement for config
* make tests better
* remove unwanted changes
* move atomic replacement logic behind persistence + pass path to latest seeds
* format
* update seeds
* review comments
* update seeds
* merge latest seeds with configs
* fix bug around latest seed
* update seed
* update seed
* seeds should be populated by separate container
* address review comment + change latest definition url
* update seeds
* format
* update seed references
* update seed
* update seed
* update seed
* update seed references
* update seed references + add Migration Acceptance Test
* update seed container in kube + disable automatic migration for kube + update docs
* update docs
* address review comments from Michel
* update doc
* temporary commmit to see if build becomes green
* delete seeds from airbyte config + undo temp commit
Fix a bug in ec2 runner that I committed to the underlying library in machulav/ec2-github-runner#51.
This points to master because the action's maintainer has yet to release a version. I'll move it to a proper version when one is released.
This is what the plugin's maintainer suggested: xvik/gradle-use-python-plugin#19.
I ran the build 10 times in a row without any checkPython error so this does seem to have fixed it.
Use Log4j2 appender to support routing logs to S3.
Create LogClient to support reading from S3.
Some clean up of the Log4j2 xml variables.
Several dependency changes to be more explicit when configuring jackson.
* use ec2 runner for kube acceptance tests
* add missing newline
* log outputs
* add user and home env vars
* log more
* use root user
* fix ec2 runner
* remaining debugging
* try overwrite forcing
* fail fast
* get kubectl location
* even more
* just look for it
* see if it's a symlink
* symlink
* make sure it's started
* try with overrides
* Revert "try with overrides"
This reverts commit 123e3c033e.
* clean up
* describe pods
* display exception when getting address in use error
* retry installing socat
* try inet4 address specifically
* switch order of install
* use unique ports for each
* try to detect locations with home and user set
* STOPTTTTTTTTTTT
* fix typo
* move socat back up one more
* add update
* Use CDK to generate source that can be configured to emit a certain number of records and always works.
* Checkpoint: socat works from inside the docker container.
* Override the entry point.
* Clean up and add ReadMe.
* Clean up socat.
* Checkpoint: connect to Kube cluster and list all the pods.
* Checkpoint: Sync worker pod is able to send output to the destination pod.
* Checkpoint: Sync worker creates Dest pod if none existed previously. It also waits for the pod to be ready before doing anything else. Sync worker will also remove the pod on termination.
* update readme
* Checkpoint: Dest pod does nott restart after finishing. Comment out delete command in Sync worker.
* working towards named pipes
* named pipes working
* update readme
* WIP named pipe / socat sidecar kube port forwarding (#3518)
* nearly working sources
* update
* stdin example
* move all kube testing yamls into the airbyte-workers directories. sort the airbyte-workers resource folder; place all the poc yamls together.
* Format.
* Put back the original KubeProcessBuilderFactory.
* Fix slight errors.
* Checkpoint: Worker pod knows its own IP. Successfully starts and writes to Dest pod after refactor.
* remove unused file and update readme
* Dest pod loops back into worker pod. However, the right messages do not seem to be passing in.
* Switch back to worker ip.
* SWEET VICTORY!.
* wrap kube pod in process (#3540)
also clean up kubernetes deploys.
* More clean up. (#3586)
The first 6 points of #3464.
The only interesting thing about this PR is the kube pod shutdown. For whatever reason, the OkHttpPool isn't respecting the evictAll call and 1 idle thread remains. So instead of shutting down immediately, the worker pod shuts down after 5 mins when the idle thread id reaped. There isn't an easy way to modify the pool's idle reap configuration now. I do not think this issue is blocking since it's relatively benign, so I vote we create a ticket and come back to this once we do an e2e test.
* Implements redirecting standard error as well. (#3623)
* Clean up before next implementation.
* kube process launching (#3790)
* processes must handle file mounting
* remove comment
* default to base entrypoint
* use process builder factory / select stdin / use a pool of ports
* fix up
* add super hacky copying example
* Checkpoint: Works end to end!
* Checkpoint: Use API to make sure init container is ready instead of blind sleep. Propagate exception in DefaultCheckConnectionWorker.
* Refactor KubePodProcess. Checked to make sure everything still works.
* Format.
* Clean up code. Begin putting this into variables and breaking up long constructor function.
* Add comments to explain what is happening.
* fix normalization test
* increase timeout for initcontainer
Co-authored-by: Davin Chia <davinchia@gmail.com>
* facepalm moment
* clean up kube poc pr (#3834)
* clean up
* remove source-always-works
* create separate commons-docker
* fix test
* enable kube e2e tests (#3866)
* enable kube e2e tests
* use more generally accepted env definition
* use new runners
* use its own runner and install minikube differently
* update name
* use kubectl alias
* use link instead of alias that doesn't propagate
* start minikube
* use driver=none
* go back to using action
* mess with versions
* revert runner
* install socat
* print logs after run
* also try re-runnining tasks
* always wait for file transfer
* use ports
* increase wait timeout for kube
* use different localhost ips and bump normalization to include an entrypoint
* proposed fix
* all working locally
* revert temporary changes
* revert normalization image change that's happening in a separate pr
* readability
* final comment
* Working Kube Cancel. (#3983)
* Port over the basic changes.
* Add logic to return proper exit code in the event of termination. Add comments to explain why.
* revert envs change and merge master to fix kube acceptance tests (#4012)
* use older env format
* fix build
Co-authored-by: jrhizor <me@jaredrhizor.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
* add AIRBYTE_ENTRYPOINT for kubernetes support
* bump versions
* bump version in seed
* Update generic template
* keep scaffold sources at 0.1.0
* add missing newline
* handle python base versions correctly
* re-bump mysql and postgres sources
* re-bump snowflake destination
* add skip tests option
* switch to running tests
* reverse conditional to make it safer
* fix publish to include the test running
* fix iterable version
* fix file generation
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
