* Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future.
* Added additional comments
* Moved usage of the hash_obj inside the conditional that confirms it exists
* Moved the hash function call inside a condition to ensure that it exists
* Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in.
* Added unit tests to validate new jinja hash functionality
* Updated unit test to pass numeric value as a float instead of string
* Removed unreferenced import to pytest
* Updated version
* format
* format
* format
* format
* format
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
* Improve airbyte cdk invalid message data type error message
* Test cdk invalid message data type custom error is raised
* Fix test to pass stream as a string
* Add valid record message data input type test
* Add object type and value to AirbyteRecordMessage validator message
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
* input can only be a string
* remove debug print
* cleanup tests
* fix in the factory
* Add a unit test
* fix for lists
* Update changelog
* Update changelog
* update abstract source and connector state manager to update shared and stream state
* clean up some extra comments and extra lines
* a few changes based on PR feedback
* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob
* rename a few paramets and add testing for state update via stream.state override
* replace shared_state processing with an explicit error and fix a few comments and pr feedback
* add some polish and additional test cases
* Add create_state_message to manager and emit per-stream state for connectors
* pr feedback and restructuring parts of the connector state manager initialization
* fix unfinished comment
* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* use pytest params to annotate tests better
* change to fix changed class name
* format test params a little better
* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* a few bits of pr feedback
* do not emit namespace in the outbound message if it is None
* fix some issues w/ prior merge from master and pr feedback
* add new tests to ensure we never send None namespace to the platform
* bump CDK version number
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* update abstract source and connector state manager to update shared and stream state
* clean up some extra comments and extra lines
* a few changes based on PR feedback
* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob
* rename a few paramets and add testing for state update via stream.state override
* replace shared_state processing with an explicit error and fix a few comments and pr feedback
* add some polish and additional test cases
* pr feedback and restructuring parts of the connector state manager initialization
* fix unfinished comment
* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* use pytest params to annotate tests better
* change to fix changed class name
* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* a few bits of pr feedback
* pr feedback and cleaning up some comments and variable renames
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* interpret legacy and new per-stream format into AirbyteStateMessages
* add ConnectorStateManager stubs for future work
* remove frozen for the time being until we need to hash descriptors
* add validation that AirbyteStateMessage has at least one of stream, global, or data fields
* pr feedback and clean up of the code
* remove changes to airbyte_protocol and perform validation in read_state()
* fix import formatting
* replace file retrieval with pkgutil to fix getting schema files
* slightly better error handling on missing files
* filter our schema gen warnings for some classes that cannot generate schemas
* add comment for todo
* add changelog and setup before publish
* draft: first pass at complete schema language generation and factory validator
* actually a working validator and fixes to the schema that went uncaught
* remove extra spike file
* fix formatting file
* Add method to generate the complete JSON schema of the low code declarative language
* add testing of a few components during schema gen
* pr feedback and a little bit of refactoring
* test for schema version
* fix some types that were erroneously marked as invalid schema
* some comments
* add jsonschemamixin to interfaces
* update tests now that interfaces are jsonschemamixin
* accidentally removed a mixin
* remove unneeded test
* make comment a little more clear
* update changelog
* bump version
* generic enum not enum class
* Add method to generate the complete JSON schema of the low code declarative language
* add testing of a few components during schema gen
* test for schema version
* update tests now that interfaces are jsonschemamixin
* accidentally removed a mixin
* remove unneeded test
* make comment a little more clear
* generic enum not enum class
* add generated json file and update docs to reference it
* verbage
* draft: first pass at complete schema language generation and factory validator
* actually a working validator and fixes to the schema that went uncaught
* remove extra spike file
* fix formatting file
* pr feedback and a little bit of refactoring
* fix some types that were erroneously marked as invalid schema
* some comments
* add jsonschemamixin to interfaces
* update changelog
* bump version
* 5-step tutorial
* move
* tiny bit of editing
* Update tutorial
* update docs
* reset
* move files
* record selector, request options, and more links
* update
* update
* connector definition
* link
* links
* update example
* footnote
* typo
* document string interpolation
* note on string interpolation
* update
* fix code sample
* fix
* update sample
* fix
* use the actual config
* Update as per comments
* write as yaml
* typo
* Clarify options overloading
* clarify that docker must be running
* remove extra footnote
* use venv directly
* Apply suggestions from code review
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* signup instructions
* update
* clarify that both dot and bracket notations are interchangeable
* Clarify how check works
* create spec and config before updating connector definition
* clarify what now_local() is
* rename to yaml structure
* Go through tutorial and update end of section code samples
* fix link
* update
* update code samples
* Update code samples
* Update to bracket notation
* remove superfluous comments
* Update docs/connector-development/config-based/tutorial/2-install-dependencies.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* Update docs/connector-development/config-based/tutorial/4-reading-data.md
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
* fix path
* update
* motivation blurp
* warning
* warning
* fix code block
* update code samples
* update code sample
* update code samples
* small updates
* update yaml structure
* custom class example
* language annotations
* update warning
* Update tutorial to use dpath extractor
* Update record selector docs
* unit test
* link to contributing
* tiny update
* $ in front of commands
* $ in front of commands
* More readings
* link to existing config-based connectors
* index
* update
* delete broken link
* supported features
* update
* Add some links
* Update docs/connector-development/config-based/overview.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* Update docs/connector-development/config-based/record-selector.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* Update docs/connector-development/config-based/overview.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* Update docs/connector-development/config-based/overview.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* Update docs/connector-development/config-based/overview.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* mention the unit
* headers
* remove mentions of interpolating on stream slice, etc.
* update
* exclude config-based docs
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* move components to definitions field
* Also update the references
* validate the top level fields and add version
* raise exception on unknown fields
* newline
* unit tests
* set version to 0.1.0
* newline
* fix parse and format methods
* define constant
* remove timestamp magic keyword
* comment
* test for ci
* uncomment test
* use timestamp()
* Bump cdk version
* bump to 0.1.72
* initial first pass converting every component to dataclasses
* replace the hackier options pass through
* get rid of the hackier way
* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses
* add dataclasses-jsonschema to setup
* fix oauth authenticator to avoid dataclass name collisions
* fix spacing for CI tests
* remove property from oauth and fix a interpolation bug
* pr feedback and cleaning up the code a bit, attempt at avoiding renaming
* fix templates and bugs surfaced during greenhouse spec testing
* fix tests
* fix missing options in some declarative components
* fix tests related to pulling latest master
* fix issue w/ passing state, slice, and token to subcomponents
* switch name back to get_access_token() since no name collision anymore
* requester is a request options provider
* get request options from slicer
* remove prints
* share interface
* actual fix with test
* small fix
* missing tests
* missing *
* simplify intersection logic
* bump cdk version
* Emit a state message even if no records were read
* newline
* merge
* comment
* implement logic in the abstract source
* remove logic from declarative source
* comment
* bump cdk version
* comment
* comment
* comments
* fix
* test for instantiating chain retrier
* fix parsing
* cleanup
* fix
* reset
* never raise on http error
* remove print
* comment
* comment
* comment
* comment
* remove prints
* add declarative stream to registry
* start working on limit paginator
* support for offset pagination
* tests
* move limit value
* extract request option
* boilerplate
* page increment
* delete offset paginator
* update conditional paginator
* refactor and fix test
* fix test
* small fix
* Delete dead code
* Add docstrings
* quick fix
* exponential backoff
* fix test
* fix
* delete unused properties
* fix
* missing unit tests
* uppercase
* docstrings
* rename to success
* compare full request instead of just url
* renmae module
* rename test file
* rename interface
* rename default retrier
* rename to compositeerrorhandler
* fix missing renames
* move action to filter
* str -> minmaxdatetime
* small fixes
* plural
* add example
* handle header variations
* also fix wait time from
* allow using a regex to extract the value
* group()
* docstring
* add docs
* update comment
* docstrings
* fix tests
* rename param
* cleanup stop_condition
* cleanup
* Add examples
* interpolated pagination strategy
* dont need duplicate class
* docstrings
* more docstrings
* docstrings
* fix tests
* first pass at substream
* seems to work for a single stream
* can also be defined in requester with stream_state
* tmp update
* update comment
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* version: Update Parquet library to latest release (#14502)
The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit c72862b613
* merge
* 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* Docs: Fixed broken links (#14622)
* fixing broken links
* more broken links
* source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620)
* Helm Chart: Add external temporal option (#14597)
* conflict env configmap and chart lock
* reverting lock
* add eof lines and documentation on values yaml
* conflict json file
* rollback json
* solve conflict
* correct minio with new version
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
* 🎉 Add YAML format to source-file reader (#14588)
* Add yaml reader
* Update docs
* Bumpversion of connector
* bump docs
* Update pyarrow dependency
* Upgrade pandas dependency
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 🎉 Source Okta: add GroupMembers stream (#14380)
* add Group_Members stream to okta source
- Group_Members return a list of users, the same schema of Users stream.
- Create a shared schema users, and both group_members and users sechema use it as a reference.
- Add Group_Members stream to source connector
* add tests and fix logs schema
- fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator
ddb87afad8/jsonschema/_validators.py (L279-L285)
- change grouop_members to use id as the cursor field since `filter` is not supported in the query string
- fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value
- remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api.
* last polish before submit the PR
- bump docker version
- update changelog
- add the right abnormal value for logs stream
- correct the sample catalog
* address comments::
- improve comments for until parameter under the logs stream
- add use_cache on groupMembers
* add use_cache to Group_Members
* change configured_catalog to test
* auto-bump connector version
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* split test files
* renames
* missing unit test
* add missing unit tests
* rename
* assert isinstance
* start extracting to their own files
* use final instead of classmethod
* assert we retry 429 errors
* Add log
* replace asserts with valueexceptions
* delete superfluous print statement
* only accept minmaxdatetime
* fix factory so we don't need to union everything with strings
* get class_name from type
* remove from class types registry
* process error handlers one at a time
* sort
* delete print statement
* comment
* comment
* format
* delete unused file
* comment
* interpolatedboolean
* comment
* not optional
* not optional
* unit tests
* fix request body data
* add test
* move file to right module
* update
* reset to master
* format
* rename to pass_by
* rename to page size
* fix
* fix some tests
* reset
* fix
* fix some of the tests
* fix test
* fix more tests
* all tests pass
* path is not optional
* reset
* reset
* reset
* delete print
* remove prints
* delete duplicate method
* add test
* fix body data
* delete extra newlines
* move to subpackage
* fix imports
* handle str body data
* simplify
* Update tests
* filter dates before stream state
* Revert "Update tests"
This reverts commit c0808c8009.
* update
* fix test
* state management
* add test
* delete dead code
* update cursor
* update cursor cartesian
* delete unused state class
* fix
* missing test
* update cursor substreams
* missing test
* fix typing
* fix typing
* delete unused field
* delete unused method
* update datetime stream slice
* cleanup
* assert
* request options
* request option cartesian
* assert when passing by path
* request options for substreams
* always return a map
* pass stream_state
* refactor and almost done fixing tests
* fix tests
* rename to inject_into
* only accept enum
* delete conditional paginator
* only return body data
* missing test
* update docstrings
* update docstrings
* update comment
* rename
* tests
* class_name -> type
* improve interface
* fix some of the tests
* fix more of the tests
* fix tests
* reset
* reset
* Revert "reset"
This reverts commit eb9a918a09.
* remove extra argument
* docstring
* update
* delete unused file
* reset
* reset
* rename
* fix timewindow
* create InterpolatedString
* helper method
* assert on request option
* better asserts
* format
* docstrings
* docstrings
* remove optional from type hint
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/stream_slicers/cartesian_product_stream_slicer.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* inherit from request options provider
* inherit from request options provider
* remove optional from type hint
* remove extra parameter
* none check
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
Co-authored-by: Bas Beelen <bjgbeelen@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
* interpolatedauth
* fix tests
* fix import
* no need for default
* Bump version
* Missing docstrings
* example
* missing example
* more docstrings
* interpolated types
