While running the Snowflake certification test, we noticed NPE on this line, indicating the state id queue was empty.
This should never happen as we always keep an 'open' state id with a running counter to associate the next state the source emits to.
I eventually realised this happens because of a race condition in the FlushWorkers, where multiple threads flush the same queue not realising the state id the flushing logic was conducted on, was already flushed by another thread.
* rename for clarity
* fix cleanup method
* giant commit because I'm irresponsible
* rename constant
* better raw table creation
* fix build?
* move code around
* tweaks
* more code shuffling
* Automated Commit - Format and Process Resources Changes
* add tests
* minor tweak
* remove unimportant methods
* cleanup
* Automated Commit - Format and Process Resources Changes
* derp
* clean up tests
* some more fixes post-merge
* botched merge
* create NoopTyperDeduper
* try and update everything to work?
* tweak comment
* move suffix args to end of list
* fix exception message
* Automated Commit - Format and Process Resources Changes
* add sqlgenerator test for softReset
* only prepare once
* update log message
* do what intellij says
* implement one more test
* less indirection
* Automated Commit - Format and Process Resources Changes
* rename test
* use noop in test
* version bump + changelog
* use stringutils
* fix typo
* flip if-statement
* typo
* simplify logic
* fix schema change logic
* typo
* use spy for clarity
* Automated Commit - Format and Process Resources Changes
* better test teardown
* slightly better logs
* fix exception message
* softReset returns single string
* Automated Commit - Format and Process Resources Changes
* simplify if chain
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* First pass
* Get prerelease flag working
* Exclude none
* Move to json santized dict
* Add prepublish upload functionality
* Remove prerelease validation exception
* Format
* Fix optional arg
* Add tests
* Apply suggestions from code review
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* disable cache for all
* test main upload is called
* Enforce requirements
* Remove redundant docker_image_name
* Format and fix test
---------
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Update dependencies
* Use RawBsonDocument
* Add tests
* Formatting
* Incremental refresh test
* Formatting
* Fix NPE
* Test replica set/oplog via testcontainers
* Formatting
* PR feedback
* Automated Commit - Formatting Changes
* Add logging to debug integration test failures
* Fix failing test
* Remove credentials conversion logic
* Update documentation to include correct config format
* Use database from credentials
* Make expected catalog match test data
* Fix formatting
* Cleanup
* Improve logging
* Revert test catalog change
* Use test database
---------
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
* Add ability to detect differences in expected Schemas and perform soft resets
* Remove alter table for overwrite syncs since its unneccessary
* Updates after testing
* pr reorganize
* comments
* add collection util test
* Add Tests
* bump version
* Automated Commit - Format and Process Resources Changes
* Destination BigQuery - Reduce amount of typing and deduping for GCS staging (#28489)
* undo comment out
* centralize t&d logic for staging and standard, add valve to staging
* Share more logic for typing and deduping
* Remove record checking logic and use only time for staging inserts
* Add Javadoc
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* Change TableNotMigratedException to extend runtime exception, remove SqlGenerator interface method
* Make Lambda slightly more readable
* add test for validating v2 schemas
* change soft reset to single string
* convert back to list, update dockerfile
* remove needless default
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* added unexpected fields
* Update schema in forms stream
* Update schema for email_events, email_subscriptions, engagements, campaigns streams
* Update changelog
* Update schema for contact_lists and forms streams, update expected records
* Update version, remove fix step from dockerfile
* Add fields to goals and forms schema, add some fields to ignored_fields
* Add missing type declarations
---------
Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com>
* Remove outdated note on part of the EU workflow running in the US
* Add note about storage of Airbyte users information
* Remove us ip from the list of ip to whitelist of eu data residency
