1
0
mirror of synced 2026-01-05 12:05:28 -05:00
Files
airbyte/airbyte-cdk/python/unit_tests/sources/declarative/auth/test_oauth.py
Brian Lai 1c482ee339 Low code cdk to beta (#22853)
* [ISSUE #19410] remove request_options_provider from the … (#21403)

* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest

* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well

* [ISSUE #19410] update source-monday

* [ISSUE #19410] code review

* [ISSUE #19410] formatting files

* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)

* refactor flows and tests to use parameters instead of options

* update documentation to reflect the change from options to parameters

* create migration script to replace options with parameters in existing manifests

* update template to use parameters instead of options

* fix tests after rebasing from the branch

* address pr feedback and extra uses of options that I missed

* additional changes needed after rebasing from master

* migrate low-code connectors to use parameters instead of options

* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)

* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)

* fully deprecate JsonSchema in favor of JsonFileSchemaLoader

* remove usage in the legacy registry

* Update migration scripts according to manifest file rename (#21920)

* Issue 21866 remove legacy factory and validation flow (#21878)

* [ISSUE #21866] clean ManifestDeclarativeSource validation

* [ISSUE #21866] remove dataclasses-jsonschema

* [ISSUE #21866] code review

* [ISSUE-21866] flake8

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] fix manifests

* [ISSUE #21926] setup server to allow for local tests (#21974)

* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)

* Issue #21576 rename dpathextractor fieldpointer (#21990)

* [ISSUE #21926] setup server to allow for local tests

* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path

* [ISSUE #21576] migration script

* [ISSUE #21576] update source-monday and source-pocket as well

* [ISSUE #21576] migration (#21997)

* [ISSUE #21576] code review

* Remove checkpoint_interval from source-prestashop manifest (#22141)

* replacing options with parameters for a few connectors I missed or were newly added

* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)

* update schema to derive cursor_field from a stream slicer if it exists

* remove usage of stream_cursor_field on simple connector use cases

* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers

* fix documentation to replace references to stream_cursor_field

* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)

* fix eslint issues for webapp (#22462)

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)

* bump connector builder server to latest CDK version

* fix breaking CDK changes in connector builder FE

* [Low-Code CDK] Separate request path from RequestOption component (#22398)

* split apart path from RequestOption and fix usages and cleanup the code

* replace usage of path with RequestPath and get rid of default to RequestOption

* fix bug where stream_slice_field was used in outbound request instead of request_option field_name

* organize yaml schema names and update documentation for RequestOption and RequestPath

* clean up tests

* regenerate models

* [ISSUE #19961] refactor stream slices (#22225)

* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet

* [ISSUE #19961] fixing Google PageSpeed Insights

* move incremental_sync field to the stream level and perform merging into one stream slicer at that level

* add tests to merging incremental and iterable into cartesian

* rewrite documentation to separate incremental sync and iterator concepts

* update documentation to use partition router and revise the tutorial to reflect the new changes to the components

* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)

* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] clean faulty merge

* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename DatetimeStreamSlicer

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* update docs

* [ISSUE #19961] clean unit tests files

* [ISSUE #19961] code review

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)

* Fix a bug where child components of a custom component cannot receive fields from other components

* add tests, documentation and commenting

* fix test from merge

* add better error message for nested initialization failures

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)

* restrict name to stream level

* remove checkpoint interval

* adjust logic for new request options

* refactor slicers

* wording

* review comments

* make oldest supported version explicit

* separate the frontend and connector builder changes from the low-code to beta release

* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)

* consolidate all the changes into a new PR after I messed up the merge on the side branch

* add set to allow this to be called externally if necessary later

* remove last few extra fields i found and fix docs links

* fix docs one more time

---------

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-02-15 21:46:49 -05:00

135 lines
5.1 KiB
Python

#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import logging
import freezegun
import pendulum
import pytest
import requests
from airbyte_cdk.sources.declarative.auth import DeclarativeOauth2Authenticator
from requests import Response
LOGGER = logging.getLogger(__name__)
resp = Response()
config = {
"refresh_endpoint": "refresh_end",
"client_id": "some_client_id",
"client_secret": "some_client_secret",
"token_expiry_date": pendulum.now().subtract(days=2).to_rfc3339_string(),
"custom_field": "in_outbound_request",
"another_field": "exists_in_body",
"grant_type": "some_grant_type",
}
parameters = {"refresh_token": "some_refresh_token"}
class TestOauth2Authenticator:
"""
Test class for OAuth2Authenticator.
"""
def test_refresh_request_body(self):
"""
Request body should match given configuration.
"""
scopes = ["scope1", "scope2"]
oauth = DeclarativeOauth2Authenticator(
token_refresh_endpoint="{{ config['refresh_endpoint'] }}",
client_id="{{ config['client_id'] }}",
client_secret="{{ config['client_secret'] }}",
refresh_token="{{ parameters['refresh_token'] }}",
config=config,
scopes=["scope1", "scope2"],
token_expiry_date="{{ config['token_expiry_date'] }}",
refresh_request_body={
"custom_field": "{{ config['custom_field'] }}",
"another_field": "{{ config['another_field'] }}",
"scopes": ["no_override"],
},
parameters=parameters,
grant_type="{{ config['grant_type'] }}",
)
body = oauth.build_refresh_request_body()
expected = {
"grant_type": "some_grant_type",
"client_id": "some_client_id",
"client_secret": "some_client_secret",
"refresh_token": "some_refresh_token",
"scopes": scopes,
"custom_field": "in_outbound_request",
"another_field": "exists_in_body",
}
assert body == expected
def test_refresh_access_token(self, mocker):
oauth = DeclarativeOauth2Authenticator(
token_refresh_endpoint="{{ config['refresh_endpoint'] }}",
client_id="{{ config['client_id'] }}",
client_secret="{{ config['client_secret'] }}",
refresh_token="{{ config['refresh_token'] }}",
config=config,
scopes=["scope1", "scope2"],
token_expiry_date="{{ config['token_expiry_date'] }}",
refresh_request_body={
"custom_field": "{{ config['custom_field'] }}",
"another_field": "{{ config['another_field'] }}",
"scopes": ["no_override"],
},
parameters={},
)
resp.status_code = 200
mocker.patch.object(resp, "json", return_value={"access_token": "access_token", "expires_in": 1000})
mocker.patch.object(requests, "request", side_effect=mock_request, autospec=True)
token = oauth.refresh_access_token()
assert ("access_token", 1000) == token
@pytest.mark.parametrize(
"expires_in_response, token_expiry_date_format",
[
(86400, None),
("2020-01-02T00:00:00Z", "YYYY-MM-DDTHH:mm:ss[Z]"),
("2020-01-02T00:00:00.000000+00:00", "YYYY-MM-DDTHH:mm:ss.SSSSSSZ"),
("2020-01-02", "YYYY-MM-DD"),
],
ids=["time_in_seconds", "rfc3339", "iso8601", "simple_date"],
)
@freezegun.freeze_time("2020-01-01")
def test_refresh_access_token_expire_format(self, mocker, expires_in_response, token_expiry_date_format):
next_day = "2020-01-02T00:00:00Z"
config.update({"token_expiry_date": pendulum.parse(next_day).subtract(days=2).to_rfc3339_string()})
oauth = DeclarativeOauth2Authenticator(
token_refresh_endpoint="{{ config['refresh_endpoint'] }}",
client_id="{{ config['client_id'] }}",
client_secret="{{ config['client_secret'] }}",
refresh_token="{{ config['refresh_token'] }}",
config=config,
scopes=["scope1", "scope2"],
token_expiry_date="{{ config['token_expiry_date'] }}",
token_expiry_date_format=token_expiry_date_format,
refresh_request_body={
"custom_field": "{{ config['custom_field'] }}",
"another_field": "{{ config['another_field'] }}",
"scopes": ["no_override"],
},
parameters={},
)
resp.status_code = 200
mocker.patch.object(resp, "json", return_value={"access_token": "access_token", "expires_in": expires_in_response})
mocker.patch.object(requests, "request", side_effect=mock_request, autospec=True)
token = oauth.get_access_token()
assert "access_token" == token
assert oauth.get_token_expiry_date() == pendulum.parse(next_day)
def mock_request(method, url, data):
if url == "refresh_end":
return resp
raise Exception(f"Error while refreshing access token with request: {method}, {url}, {data}")