* Allow SFTP to define port
* Define port as a string
* Provide a clearer error if port parsing fails
* bump source file version
* try to resolve 409 azure
Co-authored-by: Madison Swain-Bowden <bowdenm@spu.edu>
* unfinished jdbcsource separation
* creation AbstactRelation
* Migrate StateManager to new abstract level (JdbcSource -> RelationalSource)
* fix imports
* move configs to Database level + fix MySql source
* make in line jdbc source with a new impl
* Fix ScaffoldJavaJdbcSource template
* rename `AbstractField` to `CommonField`. Now it
s not an abstract class.
+ add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName`
* format
* rename generated files in line with their location
* bonus renaming
* move utility methods specific for jdbc source to a proper module
* internal review update
* BigQueryDatabase impl without row transformation
* add Static method for BigQueryDatabase instancing
* remove data type parameter limitation + rename class parameters
* Move DataTypeUtils from jdbs to common + impl basic types BigQueryUtils
* make DB2 in line with new relational abstract classes
* add missing import
* cover all biqquery classes + add type transformation method from StandardSQLTypeName to JsonSchemaPrimitive
* close unused connections
* add table list extract method
* bigquery source connector
* return all tables for a whole project instead of a dataset
* impl incremental fetch
* bigquery source connector
* bigquery source connector
* remove unnecessary databaseid
* add primitive type filtering
* add temporary workaround for test database.
* add dataset location
* fix table info retrieving
* handle dataset config
* Add working comprehensive test without data cases
* minor changes in the source processing
* acceptance tests; discover method fix
* discover method fix
* first comprehensinve test
* Comprehensive tests for the BigQuery source + database timeout config
* bigquery acceptance tests fix; formatting
* fix incremental sync using date, datetime, time and timestamp types
* Implement source checks: basic and dataset
* format
* revert: airbyte_protocol.by
* internal review update
* Add possibility to get list of comprehensive tests in a Markdown table format.
* Update airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* review update
* Implement processing for arrays and structures
* format
* added bigquery secrets
* added bigquery secrets
* spec fix
* test configs fix
* extend mapping for Arrays and Structs
* Process nested arrays
* handle arrays of records properly.
* format
* BigQuery source docs
* docs readme update
* hide evidences
* fix changlog order
* Add bigquery to source_defintions yaml
Co-authored-by: heade <danildubinin2@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* fix init state that cause loading all pages
* fix init state that caused read of all pages
* update changelog and apply suggestion from @sherifnada
* fix null state
* Fix SAT and clean up code a bit
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
* Upd money type: prices can be with floating point
* Upd version in source definitions
* Add a stub integration test to bypass CI/CD test stage
* Remove validate_output_from_all_streams param from acceptance-test-config
* Upd pipedrive changelog
* Fixed reading threads issue
* Added filtering for threads stream
* Added acceptance test for threads stream
* Added integration dummy test
* Added requirements file
* Deleted version of airbyte-cdk
* Deleted validate_output_from_all_streams flag
* Deleted condition status code in backoff
* Mixpanel initiation
* copied schemas and specs file from singer connector
* authentication and a few streams
* Added Funnels + FunnelsList
* Added example of funnel response
* added incremental Funnels stream with tests
* added Annotations, CohortMembers, Engage, Cohorts, Funnels
* added Revenue
* fixed formatting
* fixed variable names
* fixed cohort_members and updated export streams
* moved start_date and date checks into SourceMixpanel class
* added error handling
* added unit test, update docs and ci creds
* fix url base for export stream
* added full and incremental read for export stream
* updated acceptance tests, added limit correction based on number of streams, export cursor is stored in datatime string
* Funnel stream - added complex state which contains state for each funnel
* added attribution windows support and project timezone config
* fixed formatting
* added default timezone
* added dynamic schema generation for Engage and Export streams
* fixed formatting
* fixed ability to pass start_date in datetime format as well
* fixed ability to pass start_date in datetime format as well
* added additional_properties field for dynamic schemas. updates regex for start_date matching to support old config file
* fixed formatting
* export stream - convert all values to default type - string
* added schema ref
* added new properties for funnel stream
* fixed formatting in funnel schema
* added build related files
* update changelog
* fixed and added comments, renamed rate_limit variable
* fixed formatting
* changed normalization for reserved mixpanel attributes like $browser
* alphabetise spec fields
* added description about API limit handling
* updated comment
* Adding Google Cloud Storage as destination
* Removed few comments and amended the version
* Added documentation in docs/integrations/destinations/gcs.md
* Amended gcs.md with the right pull id
* Implemented all the fixes requested by tuliren as per https://github.com/airbytehq/airbyte/pull/4329
* Renaming all the files
* Branch alligned to S3 0.1.7 (with Avro and Jsonl). Removed redundant file by making S3 a dependency for GCS
* Removed some additional duplicates between GCS and S3
* Revert changes in the root files
* Revert jdbc files
* Fix package names
* Refactor gcs config
* Format code
* Fix gcs connection
* Format code
* Add acceptance tests
* Fix parquet acceptance test
* Add ci credentials
* Register the connector and update documentations
* Fix typo
* Format code
* Add unit test
* Add comments
* Update readme
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Marco Fontana <marco.fontana@sohohouse.com>
Co-authored-by: marcofontana.ing@gmail.com <marcofontana.ing@gmail.com>
Co-authored-by: Marco Fontana <MaxwellJK@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
* this should fix the check if no records in annotations stream
* update schemas for new SAT requirements && apply user hint upgrade on wrong api key
* save schema upd
* upd insights schema
* upd insights schema2
* upd insights schema3
* upd insights schema4
* upd insights schema5 (null is joking)
* upd insights schema6 (null is joking)
* upd insights schema7
* upd insights schema8
* upd insights schema8
* bump version && docs
* Few updates for GitHub source
Set correct `cursor_field` for `IssueEvents` stream.
Add rate limit handling.
Add handling for 403 error.
Add handling for 502 error.
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* first few classes for mssql cdc
* wip
* mssql cdc working against unit tests
* increment version
* add cdc acceptance test
* tweaks
* add file
* working on comprehensive tests
* change isolation from snapshot to read_committed_snapshot
* finalised type tests
* Revert "change isolation from snapshot to read_committed_snapshot"
This reverts commit 20c6768071.
* small docstring fix
* remove unused imports
* stress test fixes
* minor formatting improvements
* mssql cdc docs
* finish off cdc docs
* format fix
* update connector version
* add to changelog
* fix for sql server agent offline failing cdc enable on tables
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* use new cdc abstraction for mysql
* undo wanted change
* use cdc abstraction for postgres
* add files
* pull in latest changes
* ready
* rename class + add missing property
* use renamed class + move constants to MySqlSource
* use renamed class + move constants to PostgresSource
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* bring in latest changes from cdc abstraction
* reset to minutes
* bring in the latest changes
* format
* fix build
* address review comments
* bring in latest changes
* bring in latest changes
* use common abstraction for CDC via debezium for sql server
* remove debezium from build
* finalise PR
* should return Optional
* pull in latest changes
* pull in latest changes
* address review comments
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* lower version for tests to run on CI
* format
* Update docs/integrations/sources/mssql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* addressing review comments
* fix for testGetTargetPosition
* format changes
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
