* remove duplicate param
* remove duplicate params
* fix some of the typing issues
* fix typing issues
* newline
* format
* Enable by default
* Add missing file
* refactor and remove flag
* none check
* move line of code
* fix typing in rate_limiting
* comment
* use typedef
* else branch
* format
* gate the feature
* rename test
* fix the test
* only dedupe if the values are the same
* Add some tests
* convert values to strings
* Document the change
* implement in requester too
* add avro parser for inferring schema and reading records
* fix mypy check not caught locally
* pr feedback and some additional types
* add decimal_as_float for avro
* formatting + mypy
This PR adds a new authenticator: The SessionTokenAuthenticator. The existing authenticator under the same name is renamed to LegacySessionTokenAuthenticator.
* ingore unit tests in mypy check
* Update airbyte-cdk/python/bin/run-mypy-on-modified-files.sh
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
* work through mypy errors
* fix a bunch of stuff
* fix more type hints
* fix model_to_component_factory types
* format
* ignore list instead of allow list
---------
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
* tests pass
* everything except parquet config seems to work
* the file fortmat needs a literal
* Add a comment
* Update
* comment
* Ensure only one file type is specified
* Add a test
* add test
* update
* Automated Commit - Formatting Changes
* extract formats
* Automated Commit - Formatting Changes
* fix typo
* Update tests
* Also test jsonl
* Update airbyte-cdk/python/airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
* Update the spec
* update to new config format
* set decimal_as_float to True on legacy configs for backward compatibility
* comments
* Update airbyte-cdk/python/airbyte_cdk/sources/file_based/config/file_based_stream_config.py
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
* format
---------
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
* set test_read_limit_reached to true if we hit the max records limit
* rename slice to _slice to avoid shadowing a builtin keyword
* newline
* fix some of the typing issues
* fix some more typing issues
* another fix
* fix last typing issue
* format
* Automated Commit - Formatting Changes
* reset type
* fix the type
* Update for clarity
* Update types
---------
Co-authored-by: girarda <girarda@users.noreply.github.com>
* Try running only on modified files
* make a change
* return something with the wrong type
* Revert "return something with the wrong type"
This reverts commit 23b828371e.
* fix typing in file-based
* format
* Mypy
* fix
* leave as Mapping
* Revert "leave as Mapping"
This reverts commit 908f063f70.
* Use Dict
* update
* move dict()
* Revert "move dict()"
This reverts commit fa347a8236.
* Revert "Revert "move dict()""
This reverts commit c9237df2e4.
* Revert "Revert "Revert "move dict()"""
This reverts commit 5ac1616414.
* use Mapping
* point to config file
* comment
* strict = False
* remove --
* Revert "comment"
This reverts commit 6000814a82.
* install types
* install types in same command as mypy runs
* non-interactive
* freeze version
* pydantic plugin
* plugins
* update
* ignore missing import
* Revert "ignore missing import"
This reverts commit 1da7930fb7.
* Install pydantic instead
* fix
* this passes locally
* strict = true
* format
* explicitly import models
* Update
* remove old mypy.ini config
* temporarily disable mypy
* format
* any
* format
* fix tests
* format
* Automated Commit - Formatting Changes
* Revert "temporarily disable mypy"
This reverts commit eb8470fa3f.
* implicit reexport
* update test
* fix mypy
* Automated Commit - Formatting Changes
* fix some errors in tests
* more type fixes
* more fixes
* more
* .
* done with tests
* fix last files
* format
* Update gradle
* change source-stripe
* only run mypy on cdk
* remove strict
* Add more rules
* update
* ignore missing imports
* cast to string
* Allow untyped decorator
* reset to master
* move to the cdk
* derp
* move explicit imports around
* Automated Commit - Formatting Changes
* Revert "move explicit imports around"
This reverts commit 56e306b72f.
* move explicit imports around
* Upgrade mypy version
* point to config file
* Update readme
* Ignore errors in the models module
* Automated Commit - Formatting Changes
* move check to gradle build
* Any
* try checking out master too
* Revert "try checking out master too"
This reverts commit 8a8f3e373c.
* fetch master
* install mypy
* try without origin
* fetch from the script
* checkout master
* ls the branches
* remotes/origin/master
* remove some cruft
* comment
* remove pydantic types
* unpin mypy
* fetch from the script
* Update connectors base too
* modify a non-cdk file to confirm it doesn't get checked by mypy
* run mypy after generateComponentManifestClassFiles
* run from the venv
* pass files as arguments
* update
* fix when running without args
* with subdir
* path
* try without /
* ./
* remove filter
* try resetting
* Revert "try resetting"
This reverts commit 3a54c424de.
* exclude autogen file
* do not use the github action
* works locally
* remove extra fetch
* run on connectors base
* try bad typing
* Revert "try bad typing"
This reverts commit 33b512a3e4.
* reset stripe
* Revert "reset stripe"
This reverts commit 28f23fc6dd.
* Revert "Revert "reset stripe""
This reverts commit 5bf5dee371.
* missing return type
* do not ignore the autogen file
* remove extra installs
* run from venv
* Only check files modified on current branch
* Revert "Only check files modified on current branch"
This reverts commit b4b728e654.
* use merge-base
* Revert "use merge-base"
This reverts commit 3136670cbf.
* try with updated mypy
* bump
* run other steps after mypy
* reset task ordering
* run mypy though
* looser config
* tests pass
* fix mypy issues
* type: ignore
* optional
* this is always a bool
* ignore
* fix typing issues
* remove ignore
* remove mapping
* Automated Commit - Formatting Changes
* Revert "remove ignore"
This reverts commit 9ffeeb6cb1.
* update config
---------
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
* fix spec schema incompatibility with ui and improve spec documentation and titles
* fix schema to account for latest changes pulled from main
* tests
* remove duplicate test
* [ISSUE #27494] fix type issue caused by connector builder logging
* [ISSUE #21014] log request/response for oauth as 'global_requests'
* formatcdk
* [ISSUE #21014] support DeclarativeOauth2Authenticator as well
* [ISSUE #21014] improving message grouper tests
* formatcdk
* Test solution with logic in MessageRepository (#27990)
* Test solution with logic in MessageRepository
* Solution without creating a new ModelToComponentFactory
* [ISSUE #21014] adding tests
* [ISSUE #21014] add title and description to global requests
* Revert "Solution without creating a new ModelToComponentFactory"
This reverts commit f17799ecff.
* Automated Commit - Formatting Changes
* [ISSUE #21014] code review
* [ISSUE #21014] do not break on log appender conflict
* Automated Commit - Formatting Changes
* [ISSUE #21014] code review
* formatcdk
* [ISSUE #21014] moving is_global to is_auxiliary
* [ISSUE #26581] per partition cursor
* [ISSUE #26581] format
* [ISSUE #26581] clean up state management
* [ISSUE #26581] improving Hashabledict
* [ISSUE #26581] format cdk
* [ISSUE #26581] fix tests
* [ISSUE #26581] code review from girarda
* Retrigger pipeline
* Decouple cursor and stream slicer and pushing state management as far up cursor as possible
* Format cdk
* Small fixes/comments
* DatetimeBasedCursor should not update state based on slice (for now at least since it wasn't doing this before)
* [ISSUE #26581] code review
* Automated Commit - Formatting Changes
* [ISSUE #26581] validation overlapping keys
* [ISSUE #26581] add typing
* [ISSUE #26581] code review
* Remove SyncMode from stream_slices
* Removing SyncMode from stream_slices up until SimpleRetriever and fixing typing
* [ISSUE-26434] replacing Record primitive by class
* [ISSUE-26434] update Cursor.update_state to use new record object
* Issue 26343/data feed incremental sync solution 2 (#27481)
* TMP [ISSUE-26434] first solution to enable stop condition on pagination
* TMP [ISSUE-26434] second solution to enable stop condition on pagination
* TMP [ISSUE-26434] second solution fix
* [ISSUE #26343] fixing behavior and adding tests
* [ISSUE #26343] only updating state once a slice to allow for data feed
* [ISSUE #26343] removing freezing of cursor
* format cdk
* [ISSUE #26343] ensure data_feed doesn't have end_datetime
* [ISSUE #26343] self review
* [ISSUE #26343] code review
* [ISSUE #26343] code review clean up
* [ISSUE #26343] code review clean up
* Code review
* [ISSUE #26343] add warn log message in DatetimeBasedCursor
* format
* Format
* New file-based CDK module scaffolding
* Address code review comments
* Formatting
* Automated Commit - Formatting Changes
* Apply suggestions from code review
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
* Automated Commit - Formatting Changes
* address CR comments
* Update tests to use builder pattern
* Move max files for schema inference onto the discovery policy
* Reorganize stream & its dependencies
* File CDK: error handling for CSV parser (#27176)
* file url and updated_at timestamp is added to state's history field
* Address CR comments
* Address CR comments
* Use stream_slice to determine which files to sync
* fix
* test with no input state
* test with multiple files
* filter out older files
* group by timestamp
* Add another test
* comment
* use min time
* skip files that are already in the history
* move the code around
* include files that are not in the history
* remove start_timestamp
* cleanup
* sync misisng recent files even if history is more recent
* remove old files if history is full
* resync files if history is incomplete
* sync recent files
* comment
* configurable history size
* configurable days to sync if history is full
* move to a stateful object
* Only update state once per file
* two unit tests
* Unit tests
* missing files
* remove inner state
* fix tests
* fix interface
* fix constructor
* Update interface
* cleanup
* format
* Update
* cleanup
* Add timestamp and source file to schema
* set file uri on record
* format
* comment
* reset
* notes
* delete dead code
* format
* remove dead code
* remove dead code
* warning if history is not complete
* always set is_history_partial in the state
* rename
* Add a readme
* format
* Update
* rename
* rename
* missing files
* get instead of compute
* sort alphabetically, and sync everthing if the history is not partial
* unit tests
* Update airbyte-cdk/python/airbyte_cdk/sources/file_based/README.md
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
* Update docs
* reset
* Test to verify we remove files sorted (datetime, alphabetically)
* comment
* Update scenario
* Rename method to get_state
* If the file's ts is equal to the earliest ts, only sync it if its alphabetically greater than the file
* add missing test
* rename
* rename and update comments
* Update comment for clarity
* inject the cursor
* add interface
* comment
* Handle the case where the file has been modified since it was synced
* Only inject from AbstractFileSource
* keep the remote files in the stream slices
* Use file_based typedefs
* format
* Update the comment
* simplify the logic, update comment, and add a test
* Add a comment
* slightly cleaner
* clean up
* typing
* comment
* I think this is simpler to reason about
* create the cursor in the source
* update
* Remove methods from FiledBasedStreamReader and AbstractFileBasedStream interface (#27736)
* update the interface
* Add a comment
* rename
---------
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: clnoll <clnoll@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* for availability check - handle HttError happens during slice extraction (reading of parent stream),
updated reason messages,
moved check availability call under common try/except which handles errors during usual stream read,
moved log messages which indicate start of the stream sync before availability check in to make to understand which stream is the source of errors
* why do we return here and not try next stream?
* fixed bug in CheckStream, now we try to check availability for all streams
