1
0
mirror of synced 2026-01-04 00:04:25 -05:00
Commit Graph

53 Commits

Author SHA1 Message Date
Daryna Ishchenko
67381f8678 :bug:Source Iterable: fix type error (#28457)
* added __iter__ to StreamSlice

* updated change log
2023-07-20 10:58:28 +03:00
Serhii Chvaliuk
003ef5fc1e Source: Iterable - read timeout added (#26459)
* read timeout added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* iterable.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

---------

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-06-13 12:08:12 +03:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Serhii Chvaliuk
eff127ee20 Source: Iterable - improve 500 handling for Events stream (#26014)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

---------

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-15 20:30:17 +03:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Baz
40e2ac574e 🐛 Source Iterable: fix empty streams for CAT (#25480) 2023-04-25 19:36:36 +03:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Arsen Losenko
ecd6916675 Source Iterable: update expected records (#25144) 2023-04-13 20:50:51 +03:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Roman Yermilov [GL]
56038d0368 Source-Iterable: ignore 500 per slice for ListUsers stream (#24962)
* Source-Iterable: ignore 500 per slice for ListUsers stream

* Source-Iterable: add unittest

* Source-Iterable: fix unittest

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-07 13:52:22 +04:00
Ella Rohm-Ensing
9cfc647fa5 Ella/fail on extra fields (#23985)
* Pass argument along, add test that should pass and test that should fail

* Add tests with additionalProperties

* Set additionalproperties=false when not set|

* Parametrize test cases

* Make the behavior 'optional'

* Fix parametrization for all combinations

* Improve CI credentials README and rename param

* Update naming to be clearer about columns only

* record_has_unexpected_field > record_has_unexpected_column

* Automated Change

* Add bypass for beta + ga connectors that failed

* Update docs and TODOs

* Update changelog and dockerfile

* Update TODO

* Update a few neglected connectors

* Remove uploaded file

* Update dockerfile after merge conflict

---------

Co-authored-by: marcosmarxm <marcosmarxm@users.noreply.github.com>
2023-03-23 22:22:44 +00:00
Baz
8f7642f432 🐛 Source Iterable: improve retry on 500 - Generic Error (#23938) 2023-03-12 00:20:33 +02:00
Baz
9927b0ccd1 🐛 Source Iterable: add retry for 500 - Generic Error, increase reduce slice max attempts (#23821) 2023-03-09 17:52:40 +02:00
Arsen Losenko
73ae4f22a2 Source Iterable: Mark datefields in spec as dates (#22979)
* Source Iterable: Mark datefields in spec as dates

* Update changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-02 10:11:16 +02:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Augustin
fbd6dbf091 python-connectors: remove explicit dependencies to connector-acceptance-test build (#23332) 2023-02-22 21:21:38 +01:00
Artem Inzhyyants
1bf127e485 Source Iterable: Acceptance test high strictness level (#22579)
* Source Iterable: Acceptance test high strictness level

* Added expected records

* Source Iterable: Add allowed hosts

---------

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
2023-02-15 13:06:01 -06:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Ella Rohm-Ensing
842f66f087 Turn off default httpavailabilitystrategy for source-iterable (GA) (#22011)
* Turn off default for source-iterable (GA)

* Update streams.py

* Update iterable.md

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-02-01 17:11:58 +00:00
darynaishchenko
e29046fec6 fixed test_email_stream_chunked_encoding: added records filtering (#22202) 2023-02-01 14:55:47 +02:00
Serhii Chvaliuk
3c6f9792e5 Source iterable: fix pendulum.parse memory leak (#19913)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-12-02 12:08:58 +02:00
Denys Davydov
d1fb5a289f Source Iterable: distinguish 401 from empty stream (#18537)
* #829 source iterable: distinguish 401 from empty stream

* #829 source iterable: upd changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-27 19:17:47 +03:00
Denys Davydov
848046a158 Source Iterable - better processing of 401 and 429 errors (#18292)
* #829 source iterable - better processing of 401 and 429 errors

* source iterable: upd changelog

* #829 source iterable: fix unit tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-25 10:53:32 +03:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
Artem Inzhyyants
00d1f0d9e1 Source Iterable: Add permission check for stream (#17602)
* Source Iterable: Add permission check for stream

* Source Iterable: docs update

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-06 12:52:57 +02:00
Denys Davydov
5ba79894f2 Source Iterable: use optional end_date for SATs (#17573)
* #17506 source iterable: use optional end_date for SATs

* source iterable: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-05 09:20:08 +03:00
midavadim
7ae67d1b0d 🎉 Source Iterable - added new events streams (#16067)
* increased unit test coverage

* added additional events streams

* updated tests

* bumped connector version, update changelog

* fixed indentations

* api.py and iterable_streams.py merged into one stream.py file

* Fix unit tests

* updated release stage

* fixed import

* Updated version in seed

* auto-bump connector version [ci skip]

* updated source_specs.yaml

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-05 23:00:13 +03:00
midavadim
1a7cae0501 🎉 Source Iterable - API key is passed in headers (#15670)
* API key is passed in headers

* updated docs

* removed log

* formatted

* fixed tests

* increased timeout for incremental stream

* bumper source version

* bumper source version in seed

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-25 18:11:33 +03:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233)
* Bump year in license short to 2022

* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Maksym Pavlenok
bbd13802d8 🐛 Fix Python checker configs and Connector Base workflow (#10505) 2022-02-22 19:58:55 +02:00
Iryna Grankova
fefd5759c8 Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-12-11 01:47:29 +02:00
Marcos Marx
161a1ee7b5 format files (#8601) 2021-12-07 16:42:04 -03:00
Vadym
17504e978e 🎉 Source Iterable: Add email validation to list_users stream method execution (#8380)
* Update iterable.md docs.

* Update `Events` stream to use `export/userEvents` endpoint.
Update `events` stream schema.
Add `test_events_parse_response` unit test.

* Update events parse_response data collection.
Update events stream unit tests.
Update events stream schema.

* Bump docker version
2021-12-06 21:07:51 +02:00
Dmytro
cf3be4a441 Iterable adjustable stream slices (#8091) 2021-11-22 15:48:17 +02:00
Marcos Marx
760f429959 Source OneSignal: correct get_update_state and unit test (#7617)
* correct get_update_state and unit test

* bunmp version
2021-11-10 21:28:57 -03:00
Dmytro
6f70b6b423 Iterable: split email send stream into slices. (#7780) 2021-11-10 11:43:45 +02:00
Sherif A. Nada
9ef8626639 🐛Source Iterable: Fix date parsing (#7619) 2021-11-03 22:22:49 -07:00
Dmytro
0c2fcb8749 Fix iterable memory consumption (#7591) 2021-11-03 14:50:30 +02:00
Marcos Marx
825e487bb1 enable campaign_metrics stream (#6856) 2021-10-14 21:45:56 -03:00
Davin Chia
5a3353dcf5 Bootstrap initial custom metrics library (#6529)
Wrapper around Prometheus lib to interface with Datadog.

We use prometheus because:
- Future-proofing as it's uses the general open metrics format.
- Prometheus makes it's metrics available to a scraper to it lends itself better to the OSS set up.
- Datadog automatically converts promethues metrics into dd metrics so we don't lose much.
2021-10-05 21:51:34 +08:00
Arthur Galuza
8dae288f6d 🎉 Source Iterable: add new streams (#5915)
* Add new streams

* Upd requirements versions

* Upd docs

* Remove tests for the templates stream

* Upd csv field parsing

* Fix file permissions

* Set dependency version

* Refactor

* Merge

* Upd licence

* Add bulk metrics retrieving

* Actualize schema
2021-10-04 17:57:29 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Yaroslav Dudar
ff479dedfb 🐛 Source Iterable: Fixed normalization issue (#6242)
fix schema to be able to use normalization
2021-09-24 18:39:07 +03:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973)
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-09 13:01:45 -07:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Vadym
588553e647 Source Iterable: update to use latest version of CDK (#3378)
* Requirements updated to CDK.
airbyte-protocol and base-python requirements removed.

* Bugfix: BufferedStreamConsumer. (#3387)

* Format.

* Bump versions.

* main_dev.py renamed to main.py
README.md updated

* Source Stripe: Add Acceptance Tests to Stripe Connector (#3367)

* Add Acceptance Tests to Stripe Connector

* move configured_catalog.json to sample_files

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>

* Legacy lib references removed

* FB Marketing source - lookback window logic not functioning correctly

* FB Marketing source #1390 - returning buffered record while incremental sync

* FB Marketing source #1390 - improving checking while syncing buffered record

* FB Marketing source #1390 - adding loop_back to IncrementalStreamAPI

* FB Marketing source #1390 - bump version

* FB Marketing source #1390 - add CHANGELOG.md

* Stop formatting python with spotless (#3388)

* add test that migration output schema same as source schema (#3356)

* Add updated architecture diagram to high level docs. (#3399)

* Add updated architecture doc to high level docs.

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Correct GA readme error. (#3407)

* make shopify more resilient to timeouts (#3409)

* Update migration schema to include recent changes to the StandardSync object. (#3414)

* Update all of Pydantic to 1.6.2 per Dependabot. (#3408)

* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.

* Docker ignore update. Fix setup.py

* fixing ONLY problematic fields in freshdesk JSON schemas (#3376)

* bump airbyte-webapp version (#2266)

* add configuration for bumping webapp versionn

* set to current version

* Bump version: 0.16.0-alpha → 0.16.1-alpha

* Revert "Bump version: 0.16.0-alpha → 0.16.1-alpha"

Thiss reverts commit fdbf6dcaa6.

* also update package lock so we don't run into files changed errors

* use 0.19.0-alpha

* add npm webapp version

* Add a CDK speedrun tutorial doc (#3403)

* Add CDK Speedrun document.

* Finish speedrun doc.

* Address review comments

* Add to SUMMARY.md

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Add Rust as a connector specific dependency to source-file (#3426)

* Add Rust as a connector specific dependency to source-file

* Add more details about installation.

* Markdown lines are weird.

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* API update to latest airbyte-cdk version

* Add section Deploy Local on Windows (#3425)

* add deploy on windows steps

* correct minor

* change suggestions by @avaidyanatha

* GitBook: [master] 161 pages and 75 assets modified

* Display icons (#3140)

* Display icons

* Improve icons views

* MS SQL Server Destination implementation

Fixes issue #613.

Normalization is not yet enabled.  This will have to be added at a later point.

* Workflow to handle operations (custom transformation) (#3379)

* Keep normalization backward compatible with old settings from destination

* Bumpversion normalization image

* add npm install before all npm run generates' (#3442)

* restart containers if they fail automatically (#3423)

* Update link for contribution scheduling (#3443)

* Address issue with icon in onboarding (#3437)

* rename toy connector tutorial to "Build a connector the hard way"  (#3421)

* Upload test reports (from integration test slash commands) as GitHub artifacts (#3416)

* Archive test reports in github workflow

* Archive Test reports only when failures

* Fixing SqlServerOperations.java (#3454)

Fixing some issues with `SqlServerOperations`, which was out of sync with recent changes to `SqlOperations`.

* Add redirect to cdk tutorial page (#3456)

* add redirect to cdk tutorial page

* change path to cdk README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: vitaliizazmic <75620293+vitaliizazmic@users.noreply.github.com>
Co-authored-by: Charles <giardina.charles@gmail.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: vovavovavovavova <39351371+vovavovavovavova@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcos@airbyte.io>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: masonwheeler <masonwheeler@yahoo.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Michel Tricot <michel@dataline.io>
2021-05-18 18:26:58 +03:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00