Alexander Marquardt
26b6317198
Added new "filters" python file, along with a "hash" filter. This can… ( #18000 )
...
* Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future.
* Added additional comments
* Moved usage of the hash_obj inside the conditional that confirms it exists
* Moved the hash function call inside a condition to ensure that it exists
* Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in.
* Added unit tests to validate new jinja hash functionality
* Updated unit test to pass numeric value as a float instead of string
* Removed unreferenced import to pytest
* Updated version
* format
* format
* format
* format
* format
Co-authored-by: Alexandre Girard <alexandre@airbyte.io >
2022-10-17 23:19:55 -07:00
Brian Lai
4714a1c004
fix check for streams that do not use a stream slicer ( #18080 )
...
* fix check for streams that do not use a stream slicer
* increment version and changelog before publish
2022-10-17 17:11:11 -04:00
Alexandre Girard
0ffd503b21
🐛 [low-code] $options shouldn't overwrite values that are already defined ( #18060 )
...
* fix
* Add missing test
* remove prints
* extract to method
* rename
* Add missing test
* rename
* bump
2022-10-17 11:07:04 -07:00
Alexandre Girard
df72bbdf2c
Low-code: Pass stream_slice to read_records when reading from CheckStream ( #17804 )
...
* Implement a test
* Implement fix
* rename
* extract method
* bump
2022-10-17 05:47:56 -07:00
Alexandre Girard
62500af93b
get module name from sys.modules ( #17779 )
...
* get module name from sys.modules
* bump
* fix comment
* throw exception
* fix unittests
* Add missing files
* remove debug prints
* indent
2022-10-10 13:54:09 -07:00
Alexandre Girard
89a9d6497a
Add missing types to the registry ( #17763 )
...
* Add missing types to the registry
* bump
2022-10-09 01:25:38 -07:00
Alexandre Girard
18da341b91
Revert: #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response ( #17707 )
...
* Bump cdk version
* Revert "#17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626 )"
This reverts commit d9ad2725b1
2022-10-06 18:43:01 -07:00
Alexandre Girard
20ae75e4ee
🐛 : Add missing component types to the types registry ( #17700 )
...
* Add page increment to type registry
* Add page increment to type registry
2022-10-06 15:01:58 -07:00
Alexandre Girard
65be9b74f0
Simplify low-code connectors template ( #17476 )
...
* default schema loader
* Update generator
* update example
* remove more cruft
* Update code sampels in section 3 (but not the text)
* Update code sampels in 5
* Update instructions
* Update code samples
* Update samples
* Use now_utc instead of local
* isolate complexity to JsonSchema
* Bump cdk version
* Update changelog
* fix unit test
* slightly more helpful default
* Don't freeze cdk version in template
* remove newline
* newline
* bump
* Update changelog
* Bump
2022-10-06 08:12:21 -07:00
Alexandre Girard
3bdd7f4f43
Standardize HttpRequester's url_base and path format ( #17524 )
...
* add trailing slash to base url if missing
* remove leading / from path
* Bump
2022-10-05 15:30:37 -07:00
Alexandre Girard
61a5a1e967
[ #17107 ] Fix propagation of $options field in arrays ( #17196 )
...
* input can only be a string
* remove debug print
* cleanup tests
* fix in the factory
* Add a unit test
* fix for lists
* Update changelog
* Update changelog
2022-10-03 17:02:41 -07:00
Alexandre Girard
d6b79b8e52
Alex/limit paginator ( #17197 )
...
* remove page size from limit paginator
* Rename LimitPaginator to DefaultPaginator
* rename limit_options
* rename method
* cleanup
* comment
* update schema file
* Update changelog
* pin greenhouse connector
* fix doc in comment
* fix changelog
* missing -
* bump
* bump
2022-09-29 19:24:53 -07:00
Pedro S. Lopez
58f5128adb
🐞 CDK: fix error when logging warning for types that could not be transformed ( #17367 )
...
* fix error when logging warning for types that could not be transformed
* changelog and bump
2022-09-29 18:48:53 -04:00
Pedro S. Lopez
fe2238f3f7
🐞 CDK: properly emit state on empty slices when using iterators ( #17296 )
...
* fix: emit state on empty slices iterator
* bump version / update changelog
* format
2022-09-28 02:12:04 -04:00
Serhii Chvaliuk
a82f59ec1e
CDK: Evaluate response.text only in debug mode ( #16809 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-09-17 12:12:03 +03:00
Brian Lai
f283652313
[per-stream cdk] Emit per-stream state messages ( #16628 )
...
* update abstract source and connector state manager to update shared and stream state
* clean up some extra comments and extra lines
* a few changes based on PR feedback
* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob
* rename a few paramets and add testing for state update via stream.state override
* replace shared_state processing with an explicit error and fix a few comments and pr feedback
* add some polish and additional test cases
* Add create_state_message to manager and emit per-stream state for connectors
* pr feedback and restructuring parts of the connector state manager initialization
* fix unfinished comment
* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py
Co-authored-by: Augustin <augustin.lafanechere@gmail.com >
* use pytest params to annotate tests better
* change to fix changed class name
* format test params a little better
* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* a few bits of pr feedback
* do not emit namespace in the outbound message if it is None
* fix some issues w/ prior merge from master and pr feedback
* add new tests to ensure we never send None namespace to the platform
* bump CDK version number
Co-authored-by: Augustin <augustin.lafanechere@gmail.com >
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-09-15 23:42:49 -04:00
Serhii Chvaliuk
0d55a895cb
CDK: TypeTransformer - warning message more informative ( #16695 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-09-15 10:24:37 +03:00
Serhii Chvaliuk
5a3b6d82b6
CDK: make TypeTransformer more robust to incorrect incoming records ( #16544 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-09-13 10:30:16 +03:00
Brian Lai
6253406de9
forgot to also emit dictionary instead of empty lists when state not specified ( #16569 )
2022-09-12 17:45:48 -04:00
Brian Lai
037e8ed1a9
fix cdk bug to send legacy format if connector overrides read() ( #16566 )
...
* fix cdk bug to send legacy format if connector overrides read()
* fix comment
* update changelog and setup.py
2022-09-09 21:09:50 -04:00
Alexandre Girard
0cb44ca071
release cdk with frozen dataclasses-jsonschema lib ( #16525 )
2022-09-09 07:03:18 -07:00
Brian Lai
dceeef4683
[cdk] pin dataclasses-jsonschema to 2.15.1 ( #16253 )
2022-09-01 21:08:32 -04:00
Serhii Chvaliuk
3cfa489234
CDK: Fix regression in _checkpoint_state arg ( #16141 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-08-31 22:39:30 +03:00
Brian Lai
10a2bd1d3c
[low code connectors] add NoAuth to class types registry ( #15939 )
...
* add no auth to class type registry
* NoAuth should receive options and fail normally
* forgot to pass options in
* update changelog
2022-08-25 18:06:02 -04:00
Brian Lai
09cddffd36
[low code connectors] replace file retrieval with pkgutil to fix getting schema files ( #15814 )
...
* replace file retrieval with pkgutil to fix getting schema files
* slightly better error handling on missing files
* filter our schema gen warnings for some classes that cannot generate schemas
* add comment for todo
* add changelog and setup before publish
2022-08-23 03:06:20 -04:00
Brian Lai
ca6513625d
[low code connectors] read configs from package_data ( #15810 )
...
* read configs from package_data
* update changelog and setup
* commenting out failing tests in the short term
2022-08-19 21:16:20 -04:00
Brian Lai
ca80d3782a
[low code connectors] perform schema validation of the input config against the declarative language schema ( #15543 )
...
* draft: first pass at complete schema language generation and factory validator
* actually a working validator and fixes to the schema that went uncaught
* remove extra spike file
* fix formatting file
* pr feedback and a little bit of refactoring
* fix some types that were erroneously marked as invalid schema
* some comments
* add jsonschemamixin to interfaces
* update changelog
* bump version
2022-08-18 15:29:26 -04:00
Alexandre Girard
313ac11e6d
[low-code connectors] Get parent stream's full slice ( #15631 )
...
* always access parent stream using full_refresh mode
* Update test
* fix substream slicer
* bump
2022-08-18 10:24:10 -07:00
Serhii Chvaliuk
4e6cb05759
CDK: Improve filter_secrets skip empty string ( #15684 )
...
* Improve `filter_secrets` skip empty string
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-08-16 19:28:57 +03:00
Alexandre Girard
6332fd6527
[low-code-connectors] Replace JelloExtractor with DpathExtractor ( #15514 )
...
* Handle extracting no records from root
* handle missing keys
* record extractor interface
* dpath extractor
* docstring
* handle extract root array
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/jello.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/record_selector.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* update docstring
* respect extractor interface
* edge case handling
* document
* use dpath by default
* delete jello extractor
* bump cdk version
* delete jello dependency
* Update reference docs templates
* update template
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-08-11 12:34:54 -07:00
Alexandre Girard
29fafe26eb
[low-code connectors] Extract datetime parser and handle %s format directive ( #15429 )
...
* fix parse
* Revert "fix parse"
This reverts commit 3c76c5a782016cb69193
2022-08-10 16:35:29 -07:00
Alexandre Girard
9507d56be9
low-code connectors: fix parse and format methods ( #15326 )
...
* fix parse and format methods
* define constant
* remove timestamp magic keyword
* comment
* test for ci
* uncomment test
* use timestamp()
* Bump cdk version
* bump to 0.1.72
2022-08-08 19:02:02 -07:00
Alexandre Girard
6f1715eabe
low-code connectors: convert request headers to string before submitting them ( #15336 )
...
* convert values to strings
* introduce variable
* kwargs
* bump version
2022-08-06 10:10:23 -07:00
Brian Lai
bd31100774
initial first pass converting every component to dataclasses ( #15189 )
...
* initial first pass converting every component to dataclasses
* replace the hackier options pass through
* get rid of the hackier way
* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses
* add dataclasses-jsonschema to setup
* fix oauth authenticator to avoid dataclass name collisions
* fix spacing for CI tests
* remove property from oauth and fix a interpolation bug
* pr feedback and cleaning up the code a bit, attempt at avoiding renaming
* fix templates and bugs surfaced during greenhouse spec testing
* fix tests
* fix missing options in some declarative components
* fix tests related to pulling latest master
* fix issue w/ passing state, slice, and token to subcomponents
* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
6e59cfd7be
low-code connectors: Set slicer's request options ( #15283 )
...
* requester is a request options provider
* get request options from slicer
* remove prints
* share interface
* actual fix with test
* small fix
* missing tests
* missing *
* simplify intersection logic
* bump cdk version
2022-08-04 16:18:28 -07:00
Alexandre Girard
79a54a81fd
Emit a state message even if no records were read ( #15067 )
...
* Emit a state message even if no records were read
* newline
* merge
* comment
* implement logic in the abstract source
* remove logic from declarative source
* comment
* bump cdk version
2022-08-04 12:48:21 -07:00
Alexandre Girard
a3ff80c179
[low-code-connectors] Disable parse-time interpolation in favor of runtime-only ( #14923 )
...
* abstract auth token
* basichttp
* remove prints
* docstrings
* get rid of parse-time interpolation
* always pass options through
* delete print
* delete misleading comment
* delete note
* reset
* pass down options
* delete duplicate file
* missing test
* refactor test
* rename to '$options'
* rename to ''
* interpolatedauth
* fix tests
* fix
* docstrings
* update docstring
* docstring
* update docstring
* remove extra field
* undo
* rename to runtime_parameters
* docstring
* update
* / -> *
* update template
* rename to options
* Add examples
* update docstring
* Update test
* newlines
* rename kwargs to options
* options init param
* delete duplicate line
* type hints
* update docstring
* Revert "delete duplicate line"
This reverts commit 4255d5b346
2022-07-28 08:57:17 -07:00
Alexandre Girard
783923db76
[low-code CDK] Enable runtime string interpolation in authenticators ( #14914 )
...
* interpolatedauth
* fix tests
* fix import
* no need for default
* Bump version
* Missing docstrings
* example
* missing example
* more docstrings
* interpolated types
2022-07-25 19:04:05 -07:00
Augustin
b76b73bbfb
cdk: do not call init_uncaught_exception_handler from modules' root ( #14892 )
2022-07-21 16:34:20 +02:00
Brian Lai
ff74998057
forgot to publish a new version of airbyte-cdk to PyPi ( #14696 )
2022-07-14 13:22:12 -04:00
Brian Lai
374e265fcb
[Low Code CDK] configurable oauth request payload ( #13993 )
...
* configurable oauth request payload
* support interpolation for dictionaries that are not new subcomponents
* rewrite a declarative oauth authenticator that performs interpolation at runtime
* formatting
* whatever i don't know why factory gets flagged w/ the newline change
* we java now
* remove duplicate oauth
* add some comments
* parse time properly from string interpolation
* move declarative oauth to its own package in declarative module
* add changelog info
2022-07-08 16:49:16 -04:00
jordan-glitch
4395fbb4d2
cdk: Attach namespace to stream in catalog ( #13923 )
2022-07-04 18:04:31 +02:00
Sherif A. Nada
7acbcfbfb7
Fix CDK obfuscation of nested secrets ( #14035 )
2022-06-23 09:23:27 -07:00
George Claireaux
4d279f8238
Remove legacy sentry code from cdk ( #14016 )
...
* rip sentry out of cdk
* remove sentry dsn from gsc
2022-06-23 12:14:09 +01:00
Alexandre Girard
57dd7a9d4b
[low-code connectors] use jello instead of jq ( #13570 )
...
* use jello instead of jq
* reset
* rename
* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
