* Try running only on modified files
* make a change
* return something with the wrong type
* Revert "return something with the wrong type"
This reverts commit 23b828371e.
* fix typing in file-based
* format
* Mypy
* fix
* leave as Mapping
* Revert "leave as Mapping"
This reverts commit 908f063f70.
* Use Dict
* update
* move dict()
* Revert "move dict()"
This reverts commit fa347a8236.
* Revert "Revert "move dict()""
This reverts commit c9237df2e4.
* Revert "Revert "Revert "move dict()"""
This reverts commit 5ac1616414.
* use Mapping
* point to config file
* comment
* strict = False
* remove --
* Revert "comment"
This reverts commit 6000814a82.
* install types
* install types in same command as mypy runs
* non-interactive
* freeze version
* pydantic plugin
* plugins
* update
* ignore missing import
* Revert "ignore missing import"
This reverts commit 1da7930fb7.
* Install pydantic instead
* fix
* this passes locally
* strict = true
* format
* explicitly import models
* Update
* remove old mypy.ini config
* temporarily disable mypy
* format
* any
* format
* fix tests
* format
* Automated Commit - Formatting Changes
* Revert "temporarily disable mypy"
This reverts commit eb8470fa3f.
* implicit reexport
* update test
* fix mypy
* Automated Commit - Formatting Changes
* fix some errors in tests
* more type fixes
* more fixes
* more
* .
* done with tests
* fix last files
* format
* Update gradle
* change source-stripe
* only run mypy on cdk
* remove strict
* Add more rules
* update
* ignore missing imports
* cast to string
* Allow untyped decorator
* reset to master
* move to the cdk
* derp
* move explicit imports around
* Automated Commit - Formatting Changes
* Revert "move explicit imports around"
This reverts commit 56e306b72f.
* move explicit imports around
* Upgrade mypy version
* point to config file
* Update readme
* Ignore errors in the models module
* Automated Commit - Formatting Changes
* move check to gradle build
* Any
* try checking out master too
* Revert "try checking out master too"
This reverts commit 8a8f3e373c.
* fetch master
* install mypy
* try without origin
* fetch from the script
* checkout master
* ls the branches
* remotes/origin/master
* remove some cruft
* comment
* remove pydantic types
* unpin mypy
* fetch from the script
* Update connectors base too
* modify a non-cdk file to confirm it doesn't get checked by mypy
* run mypy after generateComponentManifestClassFiles
* run from the venv
* pass files as arguments
* update
* fix when running without args
* with subdir
* path
* try without /
* ./
* remove filter
* try resetting
* Revert "try resetting"
This reverts commit 3a54c424de.
* exclude autogen file
* do not use the github action
* works locally
* remove extra fetch
* run on connectors base
* try bad typing
* Revert "try bad typing"
This reverts commit 33b512a3e4.
* reset stripe
* Revert "reset stripe"
This reverts commit 28f23fc6dd.
* Revert "Revert "reset stripe""
This reverts commit 5bf5dee371.
* missing return type
* do not ignore the autogen file
* remove extra installs
* run from venv
* Only check files modified on current branch
* Revert "Only check files modified on current branch"
This reverts commit b4b728e654.
* use merge-base
* Revert "use merge-base"
This reverts commit 3136670cbf.
* try with updated mypy
* bump
* run other steps after mypy
* reset task ordering
* run mypy though
* looser config
* tests pass
* fix mypy issues
* type: ignore
* optional
* this is always a bool
* ignore
* fix typing issues
* remove ignore
* remove mapping
* Automated Commit - Formatting Changes
* Revert "remove ignore"
This reverts commit 9ffeeb6cb1.
* update config
---------
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
* fix a few connectors with failing custom component unit tests
* fix outdated expected records
* make square orders stream incremental
* Updated square SAT and delete custom extractor for monday.
* Added allowedHosts
* Delete DpathStringExtractor from init
* Updated abnormal records
* bump versions of greenhouse, tempo, woocommerce, and zenloop to use beta low code version
* remove woocommerce because it's already published with beta
* add allowedHosts for tempo and zenloop
* fix abnormal state file to use created_at for some streams
* auto-bump connector version
* bump versions for square and monday
* auto-bump connector version
* auto-bump connector version
* manually update square definition generation due to failure
* auto-bump connector version
---------
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Scripts to
* Run CATs against the local CDK for one connector
* Run CATs against the local CDK for multiple connectors
* Create a connecter image with the local CDK
---------
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* [ISSUE #19410] remove request_options_provider from the … (#21403)
* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest
* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well
* [ISSUE #19410] update source-monday
* [ISSUE #19410] code review
* [ISSUE #19410] formatting files
* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)
* refactor flows and tests to use parameters instead of options
* update documentation to reflect the change from options to parameters
* create migration script to replace options with parameters in existing manifests
* update template to use parameters instead of options
* fix tests after rebasing from the branch
* address pr feedback and extra uses of options that I missed
* additional changes needed after rebasing from master
* migrate low-code connectors to use parameters instead of options
* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)
* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)
* fully deprecate JsonSchema in favor of JsonFileSchemaLoader
* remove usage in the legacy registry
* Update migration scripts according to manifest file rename (#21920)
* Issue 21866 remove legacy factory and validation flow (#21878)
* [ISSUE #21866] clean ManifestDeclarativeSource validation
* [ISSUE #21866] remove dataclasses-jsonschema
* [ISSUE #21866] code review
* [ISSUE-21866] flake8
* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)
* [ISSUE #21559] remove DefaultPaginator.url_base
* [ISSUE #21559] code review
* [ISSUE #21559] update migration script
* [ISSUE #21559] code review
* [ISSUE #21559] update documentation
* [ISSUE #21559] run migration (#21824)
* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)
* [ISSUE #21559] remove DefaultPaginator.url_base
* [ISSUE #21559] code review
* [ISSUE #21559] update migration script
* [ISSUE #21559] code review
* [ISSUE #21559] update documentation
* [ISSUE #21559] run migration (#21824)
* [ISSUE #21559] fix manifests
* [ISSUE #21926] setup server to allow for local tests (#21974)
* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)
* Issue #21576 rename dpathextractor fieldpointer (#21990)
* [ISSUE #21926] setup server to allow for local tests
* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path
* [ISSUE #21576] migration script
* [ISSUE #21576] update source-monday and source-pocket as well
* [ISSUE #21576] migration (#21997)
* [ISSUE #21576] code review
* Remove checkpoint_interval from source-prestashop manifest (#22141)
* replacing options with parameters for a few connectors I missed or were newly added
* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)
* update schema to derive cursor_field from a stream slicer if it exists
* remove usage of stream_cursor_field on simple connector use cases
* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers
* fix documentation to replace references to stream_cursor_field
* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)
* fix eslint issues for webapp (#22462)
* 🪟🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)
* bump connector builder server to latest CDK version
* fix breaking CDK changes in connector builder FE
* [Low-Code CDK] Separate request path from RequestOption component (#22398)
* split apart path from RequestOption and fix usages and cleanup the code
* replace usage of path with RequestPath and get rid of default to RequestOption
* fix bug where stream_slice_field was used in outbound request instead of request_option field_name
* organize yaml schema names and update documentation for RequestOption and RequestPath
* clean up tests
* regenerate models
* [ISSUE #19961] refactor stream slices (#22225)
* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet
* [ISSUE #19961] fixing Google PageSpeed Insights
* move incremental_sync field to the stream level and perform merging into one stream slicer at that level
* add tests to merging incremental and iterable into cartesian
* rewrite documentation to separate incremental sync and iterator concepts
* update documentation to use partition router and revise the tutorial to reflect the new changes to the components
* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)
* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)
* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)
* [ISSUE 19961] rename CustomStreamSlicer (#22598)
* [ISSUE 19961] rename CustomStreamSlicer
* [ISSUE #19961] code review CustomStreamSlicer
* [ISSUE #19961] fix source_square incremental sync
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter
* remove SinglePartitionRouter from the schema
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596)
* [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter
* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end
* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] clean faulty merge
* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)
* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)
* [ISSUE 19961] rename CustomStreamSlicer (#22598)
* [ISSUE 19961] rename CustomStreamSlicer
* [ISSUE #19961] code review CustomStreamSlicer
* [ISSUE #19961] fix source_square incremental sync
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter
* remove SinglePartitionRouter from the schema
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] rename DatetimeStreamSlicer
* [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596)
* [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter
* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end
* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* update docs
* [ISSUE #19961] clean unit tests files
* [ISSUE #19961] code review
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)
* Fix a bug where child components of a custom component cannot receive fields from other components
* add tests, documentation and commenting
* fix test from merge
* add better error message for nested initialization failures
* 🪟🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)
* restrict name to stream level
* remove checkpoint interval
* adjust logic for new request options
* refactor slicers
* wording
* review comments
* make oldest supported version explicit
* separate the frontend and connector builder changes from the low-code to beta release
* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)
* consolidate all the changes into a new PR after I messed up the merge on the side branch
* add set to allow this to be called externally if necessary later
* remove last few extra fields i found and fix docs links
* fix docs one more time
---------
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
* airbyte_protocol as dependency for cdk
download from pypi instead of using local directory in monorepo
* remove generate-protocol-files.sh
* protocol_models => models
airbyte_protocol.models more closely reflects java package name
* use published pypi package
* fix imports
* context because someone will wonder what happened here
* run formatter
---------
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
Co-authored-by: Sherif Nada <snadalive@gmail.com>
Co-authored-by: cgardens <charles@airbyte.io>
* initial work to parse manifest objects into pydantic models
* pr feedback and some other validations
* rerun manifest schema generation
* remove field constraint due to bug
* initial work to construct most components from greenhouse
* custom components parse subcomponent fields correctly and adding a few more component constructors
* construct components from gnews
* first pass at posthog.yaml
* Handle nested custom components with list values.
Also includes updates to posthog.yaml, including autoformatting changes.
* adding constructors for slicers, filters, and transformations and a few bug fixes
* make sed work across multiple OS
* add NoAuth component
* fix handling of custom components with nested list
* Autogenerate `TYPE_NAME_TO_MODEL` mapping
* Handle default kwargs not defined on model for custom components
* Re-add `options` for CartesianProductStreamSlicer for backwards compat
with custom stream slicers
* add basic unit tests for the model component factory
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* Remove `_get_defaults`; using actual default values on classes instead
* Add backoff strategy component creation functions
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* add lots of tests to construct components from the pydantic models and a few bug fixes
* add a few tests for the model to component factory
* add catch
* fix a bug where propagated schema doesn't work with old factory
* clean up a few files
* add type inference for custom components, more tests and some refactoring of the model factory
* template, docs, manifest updates, pr feedback and some cleanup
* pr feedback and polish schema a bit
* fix tests from the latest rebase of master
* fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow
* fix CheckStream bug cleanup and a few small tweaks and polish
* add additional test to cover bug case
* fix formatting
* 🤖 Bump minor version of Airbyte CDK
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
* handwritten low code manifest example components
* add MinMaxDatetime to jsonschema
* add a basic gradle command to generate manifest components
* Add auth components to handwritten component schema
- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth
* Respect optional properties in DeclarativeOauth2Authenticator
* Fix `Dict[str, Any]` mapping in auth components
* add default error handler composite error handler and http response filter components
* [low code component schema] adding backoff strategies to schema
* [low code component schema] fix float types
* [low code component schema] add RecordFilter
* Remove `config` from auth components
* [low code component schema] add Interpolation (with pending question on 'type' not being defined)
* Add CartesianProductStreamSlicer & DatetimeStreamSlicer
* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer
* [low code component schema] add InterpolatedRequestOptionsProvider
* Add slicer components, and fix a couple of components after reviewing output
* [low code component schema] adding transformations and adding type to interpolators
* adding spec and a few small tweaks
* Add DefaultSchemaLoader
* [low code component schema] attempt on custom class
* Add descriptions for auth components
* add RequestOption
* remove interpolated objects from the schema in favor of strings only
* a few schema fixes and adding some custom pagination and stream slicer
* [low code component schema] fix CustomBackoffStrategy
* Add CustomRecordExtractor
* add some description and add additional properties
* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema
* [low code component schema] validating existing schemas
* [low code component schema] clean validation script
* add manifest transformer tests and a few tweaks to the schema
* Revert "[low code component schema] clean validation script"
This reverts commit 2408f41cf6.
* Revert "[low code component schema] validating existing schemas"
This reverts commit 9d39977815.
* [low code component schema] integrate validation script to gradle
* [low code component schema] updating validation script permissions
* remove a few model gen spike files and clean up comments
* default types should take parent type into account and a few schema changes
* add inline schema components and fix validation bugs
* add types to every component's literal enum and more little schema fixes
* missing component prefix
* pr feedback, renaming the schema and updating various docs
* fixing schema bugs from testing and new components
* last few small tweaks and fixes and fix the script
* bump the version before publsih
* include the declarative_component_schema.yaml as package data so its included when package is published
* roll the version to publish
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
* generate AirbyeTraceMessage `type` enum with descriptive class name
* add comment on `title` usage
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* apply changes to bases/airbyte-protocol
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
