Ella Rohm-Ensing
221c9b1839
Reintroduce AvailabilityStrategy into the CDK (HttpAvailabilityStrategy default not turned on yet) ( #21484 )
...
* Restore AvailabilityStrategy
* Add test for http availability strategy on empty stream from original bug fix 7c17351631#21488 )
* turn off HttpAvailabilityStrategy as default (for now)
* Update imports accordingly
2023-01-18 13:32:48 -05:00
Ella Rohm-Ensing
d378294c2d
Improvements to edge cases of CheckStream ( #21404 )
...
* Add test for failure case
* Except StopIteration - make test pass
* Don't attempt to connect to a stream if we get no stream slices
* Make helper method for getting first record for a slice
* Add comments and exit early if stream to check isn't in list of source streams
* move helpers to helper module
* Clarify what it means when StopIteration is returned by helper methods
2023-01-13 16:26:28 -05:00
Ella Rohm-Ensing
8bb41282c5
CDK 0.15.0 and source-github 0.3.10 -- revert AvailabillityStrategy changes ( #20523 )
...
* Revert "source-github: move known error handling to GithubAvailabilityStrategy (#19978 )"
This reverts commit f97db17ccc🐛 Python CDK: fix `StopIteration` error for `check_availability` (#20429 )"
This reverts commit 4e9b014277#19977 )"
This reverts commit 55a32886a3octavia-squidington-iii@users.noreply.github.com >
2022-12-15 14:24:24 -05:00
Ella Rohm-Ensing
55a32886a3
CDK: AbstractSource.read() skips syncing stream if its unavailable (add AvailabilityStrategy concept) ( #19977 )
...
* Rough first implememtation of AvailabilityStrategy s
* Basic unit tests for AvailabilityStrategy and ScopedAvailabilityStrategy
* Make availability_strategy a property, separate out tests
* Remove from DeclarativeSource, remove Source parameter from methods, make default no AvailabilityStrategy
* Add skip stream if not available to read()
* Changes to CDK to get source-github working using AvailabilityStrategy, flakecheck
* reorganize cdk class, add HTTPAvailabilityStrategy test
* cleanup, docstrings
* pull out error handling into separate method
* Pass source and logger to check_connection method
* Add documentation links, handle 403 specifically
* Fix circular import
* Add AvailabilityStrategy to Stream and HTTPStream classes
* Remove AS from abstract_source, add to Stream, HTTPStream, AvailabilityStrategy unit tests passing for per-stream strategies
* Modify MockHttpStream to set no AvailabilityStrategy since source test mocking doesn't support this
* Move AvailabilityStrategy class to sources.streams
* Move HTTPAvailabilityStrategy to http module
* Use pascal case for HttpAvailabilityStrategy
* Remove docs message method :( and default to True availability on unhandled HTTPErrors
* add check_availability method to stream class
* Add optional source parameter
* Add test for connector-specific documentation, small tests refactor
* Add test that performs the read() function for stream with default availability strategy
* Add test for read function behavior when stream is unavailable
* Add 403 info in logger message
* Don't return error for other HTTPErrors
* Split up error handling into methods 'unavailable_error_codes' and 'get_reason_for_error'
* rework overrideable list of status codes to be a dict with reasons, to enforce that users provide reasons for all listed errors
* Fix incorrect typing
* Move HttpAvailability to its own module, fix flake errors
* Fix ScopedAvailabilityStrategy, docstrings and types for streams/availability_strategy.py
* Docstrings and types for core.py and http/availability_strategy.py
* Move _get_stream_slices to a StreamHelper class
* Docstrings + types for stream_helpers.py, cleanup test_availability.py
* Clean up test_source.py
* Move logic of getting the initial record from a stream to StreamHelper class
* Add changelog and bump minor version
* change 'is True' and 'is False' behavior
* use mocker.MagicMock
* Remove ScopedAvailabilityStrategy
* Don't except non-403 errors, check_stream uses availability_strategy if possible
* CDK: pass error to reasons_for_error_codes
* make get_stream_slice public
* Add tests for raising unhandled errors and retries are handled
* Add tests for CheckStream via AvailabilityStrategy
* Add documentation for stream availability of http streams
* Move availability unit tests to correct modules, report error message if possible
* Add test for reporting specific error if available
2022-12-12 14:32:34 -05:00
Brian Lai
de24ddced5
make sure stream_slices is an iterator ( #18092 )
...
* ensure output from stream_slices() is always an iterator
* bump cdk version
* bump version after merging latest cdk
* bump cdk version post rebase
2022-10-19 15:37:00 -04:00
Alexandre Girard
df72bbdf2c
Low-code: Pass stream_slice to read_records when reading from CheckStream ( #17804 )
...
* Implement a test
* Implement fix
* rename
* extract method
* bump
2022-10-17 05:47:56 -07:00
Brian Lai
bd31100774
initial first pass converting every component to dataclasses ( #15189 )
...
* initial first pass converting every component to dataclasses
* replace the hackier options pass through
* get rid of the hackier way
* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses
* add dataclasses-jsonschema to setup
* fix oauth authenticator to avoid dataclass name collisions
* fix spacing for CI tests
* remove property from oauth and fix a interpolation bug
* pr feedback and cleaning up the code a bit, attempt at avoiding renaming
* fix templates and bugs surfaced during greenhouse spec testing
* fix tests
* fix missing options in some declarative components
* fix tests related to pulling latest master
* fix issue w/ passing state, slice, and token to subcomponents
* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
a3ff80c179
[low-code-connectors] Disable parse-time interpolation in favor of runtime-only ( #14923 )
...
* abstract auth token
* basichttp
* remove prints
* docstrings
* get rid of parse-time interpolation
* always pass options through
* delete print
* delete misleading comment
* delete note
* reset
* pass down options
* delete duplicate file
* missing test
* refactor test
* rename to '$options'
* rename to ''
* interpolatedauth
* fix tests
* fix
* docstrings
* update docstring
* docstring
* update docstring
* remove extra field
* undo
* rename to runtime_parameters
* docstring
* update
* / -> *
* update template
* rename to options
* Add examples
* update docstring
* Update test
* newlines
* rename kwargs to options
* options init param
* delete duplicate line
* type hints
* update docstring
* Revert "delete duplicate line"
This reverts commit 4255d5b346
2022-07-28 08:57:17 -07:00
Alexandre Girard
01fc7a48c7
move jinja macros to their own file ( #14515 )
...
* move jinja macros to their own file
* format
* export as dict
* format
2022-07-08 07:56:06 -07:00
Alexandre Girard
e23789bbd6
Alex/declarative stream incremental fix ( #14268 )
...
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
2022-07-01 08:39:19 -07:00
Alexandre Girard
7fbe6dab5c
[Low code CDK] add support for iterating over parent stream slices ( #13287 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-02 08:53:04 -07:00
Alexandre Girard
aa92518721
Low-code connectors: configurable source from yaml ( #13038 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* sengrid low code connector
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* update import
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944snadalive@gmail.com >
* more precise exception
* rename class
* add comment
* Try to install packages to build jq
* isort
* only automake
* Revert "only automake"
This reverts commit c8fe154ffcab5a7afd087c9fb8eb33snadalive@gmail.com >
* fix tests and format
* format
* update
* better error message
* Add jq dependency
* Use request header provider
* rename
* rename field
* remove get_context method
* rename
* add a comment
* format
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-06-01 07:31:52 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
