- implement a heap client to load data via the server-side API:
https://developers.heap.io/reference/server-side-apis-overview
- the connector supports a generic data source, and the api_type
determines the output schema. The output schema is dynamic.
- users pick the columns that will be loaded to the destination
- Consequently, each configured catalog only includes one stream
* Fix failing test
* Bigquery-denormalized update spec plus destination_definitions.yaml to fix integration tests
* sanity
* Update version number and release note
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* WIP
* fix issues
* fix test
* format
* more tests
* one more test
* make lsn_commit_behaviour not required
* update expected spec
* update expected spec
* update expected spec
* PR comments
* PR comments
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Expected Records to `.jsonl` format
* fix formatting template
* remove endline
* update templates
* Update docs/connector-development/testing-connectors/source-acceptance-tests-reference.md
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
* Increased default buffer count and introduces configurable value for destination redshift
* Updates logic to ensure filebuffers are at minimum the default number
* Reverted changes to increase the file buffer default but allow for users to increase with adequate warnings
* Bumps version number and updates changelog
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* [faker] decouple stream state
* add PR #
* commit Stream instantiate changes
* fixup expected record
* skip backward test for this version too
* Apply suggestions from code review
Co-authored-by: Augustin <augustin@airbyte.io>
* lint
* Create realistic datasets of 10GB, 100GB, and 1TB in size (#20558)
* Faker CSV Streaming utilities
* readme
* don't do a final pipe to jq or you will run out or ram
* doc
* Faker gets 250% faster (#20741)
* Faker is 250% faster
* threads in spec + lint
* pass tests
* revert changes to record helper
* cleanup
* update expected_records
* bump default records-per-slice to 1k
* enforce unique email addresses
* cleanup
* more comments
* `parallelism` and pass tests
* update expected records
* cleanup notes
* update readme
* update expected records
* auto-bump connector version
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Implement the persistence layer changes following #19191.
This PR handles writing and reading stats to the new stream stat_table and columns in the existing sync_stats table.
At the same time we introduce upserts of stats records - i.e. merge updates into a single record - in preparation for real time stats updates vs the current approach where a new stat record is always written.
There will be two remaining PRs after this:
- First PR will be to fully wire up and test the API.
- Second PR will be to actually save stats while jobs are running.
* handwritten low code manifest example components
* add MinMaxDatetime to jsonschema
* add a basic gradle command to generate manifest components
* Add auth components to handwritten component schema
- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth
* Respect optional properties in DeclarativeOauth2Authenticator
* Fix `Dict[str, Any]` mapping in auth components
* add default error handler composite error handler and http response filter components
* [low code component schema] adding backoff strategies to schema
* [low code component schema] fix float types
* [low code component schema] add RecordFilter
* Remove `config` from auth components
* [low code component schema] add Interpolation (with pending question on 'type' not being defined)
* Add CartesianProductStreamSlicer & DatetimeStreamSlicer
* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer
* [low code component schema] add InterpolatedRequestOptionsProvider
* Add slicer components, and fix a couple of components after reviewing output
* [low code component schema] adding transformations and adding type to interpolators
* adding spec and a few small tweaks
* Add DefaultSchemaLoader
* [low code component schema] attempt on custom class
* Add descriptions for auth components
* add RequestOption
* remove interpolated objects from the schema in favor of strings only
* a few schema fixes and adding some custom pagination and stream slicer
* [low code component schema] fix CustomBackoffStrategy
* Add CustomRecordExtractor
* add some description and add additional properties
* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema
* [low code component schema] validating existing schemas
* [low code component schema] clean validation script
* add manifest transformer tests and a few tweaks to the schema
* Revert "[low code component schema] clean validation script"
This reverts commit 2408f41cf6.
* Revert "[low code component schema] validating existing schemas"
This reverts commit 9d39977815.
* [low code component schema] integrate validation script to gradle
* [low code component schema] updating validation script permissions
* remove a few model gen spike files and clean up comments
* default types should take parent type into account and a few schema changes
* add inline schema components and fix validation bugs
* add types to every component's literal enum and more little schema fixes
* missing component prefix
* pr feedback, renaming the schema and updating various docs
* fixing schema bugs from testing and new components
* last few small tweaks and fixes and fix the script
* bump the version before publsih
* include the declarative_component_schema.yaml as package data so its included when package is published
* roll the version to publish
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
* get rid of audience report streams using hourly dimension
* bump version and docs changelog
* fix tests
* update backwards compatibility tests since some streams are no longer available
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
