1
0
mirror of synced 2025-12-23 03:47:05 -05:00
Commit Graph

191 Commits

Author SHA1 Message Date
Marius Posta
f5c7c1c0b8 chore: get ./gradlew format to pass for the whole repo (same java style) (#29786) 2023-08-24 05:09:42 -05:00
Ben Church
40781313da Update Internal Fields: update ql levels to better resemble previous high strictness (#29450) 2023-08-15 18:03:10 -05:00
Augustin
00d9462216 cat/connectors-ci: replace docker runner with dagger runner in CAT (#28000) 2023-08-11 17:58:48 +02:00
Serhii Lazebnyi
21871d66a3 Source Amazon Ads: Add new tactic support for sponsored_display_report_stream (#29212)
* Add source amazon ads T00030 tactic for sponsored_display_report_stream

* Fix code style and unit test

* Rename report_infos variable to report_info_list
2023-08-10 13:33:34 +02:00
Ben Church
2f7deaee02 [skip ci] Metadata: Remove leading underscore (#29024)
* DNC

* Add test models

* Add model test

* Remove underscore from metadata files

* Regenerate models

* Add test to check for key transformation

* Allow additional fields on metadata

* Delete transform
2023-08-03 10:56:13 -07:00
Ben Church
e9490e3fb6 Connector Levels: Add new internal metadata fields (#28904)
* Add airbyte internal

* Add tests

* First pass

* Set destinations to same levels as sources

* Best guess at missing statuses

* Best guess at _ql

* Add separate enum class

* Fix support level name

* Update templates

* Add one more test
2023-08-01 18:08:33 -05:00
Denys Davydov
ccac29af1b 🚨 🚨 Source Amazon Ads: fix schemas (#27868)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #1955 source amazon ads: fix stream schemas

* source amazon ads: upd changelog

* source amazon ads: upd CAT config

* upd backwards incompatible version

* update metadata with breakingChanges entry

* fix CATs

* fix md
2023-07-24 16:29:42 +03:00
Roman Yermilov [GL]
aea5a9ec8e Source Amazon Ads: fix Nonetype error when recordId is missing (#28155)
* Source Amazon Ads: fix Nonetype error when recordId is missing

* Source Amazon Ads: bump version and update changelog

* Source Amazon Ads: unittest for recordId generation
2023-07-13 15:10:41 +04:00
midavadim
d1802154ae 🎉 Amazon Ads - new streams for bids and keyword recommendations (#28002)
* added new streams for amazon adgroup bids keyword recommendations

* updated docs and configured catalog

* updated version in metadata

* fix test

* updated docs

* fix doc after merge

* fix comments

* updated version

* fux typo
2023-07-06 13:18:33 -04:00
Anatolii Yatsuk
df135afa26 Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607)
* Add stream for sponsored brands v3
* Add new stream Portfolios
2023-07-05 15:56:04 +03:00
midavadim
c7b2f99af5 🎉 Source Amazon Ads add campaign negative keywords stream (#27482)
* Add sponsored_product_campaign_negative_keywords stream

* revert changes

* 🤖 Auto format source-amazon-ads code [skip ci]

* Add sponsored_display_budget_rules stream

* 🤖 Auto format source-amazon-ads code [skip ci]

* ignore dailyBudget field
it can be updated, also it is sometimes integer, sometimes float"

* Fix sponsored_product_campaigns

* Fix sponsored_product_campaigns

* merged with the latest version

* updated source test

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-23 17:31:20 +03:00
Roman Yermilov [GL]
80e6a2666b 🐛Source Amazon Ads: fix portfolioId type (#25874)
* Source Amazon Ads: fix portfolioId type

* Source Amazon Ads: update metadata

* Source Amazon Ads: disable backward compatibility tests for 1.1.0

* 🤖 Auto format source-amazon-ads code [skip ci]

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 11:22:43 -05:00
Augustin
d8820d6e80 Fix python connectors formatting errors with black / isort (#26951)
* format source-tiktok-marketing

* format source-twilio

* format source-instgram

* format source-amazon-ads

* format source-google-search-console

* format source-stripe

* format source-google-analytics-data-api

* format source-facebook-marketing

* format source-zendesk-support

* format source-file
2023-06-02 09:26:04 -05:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Marcos Marx
215abfe4a8 bump amazon ads medatafile file (#26657)
Co-authored-by: Augustin <augustin@airbyte.io>
2023-05-26 14:57:37 +02:00
Shishir Rajmohan Verma
26c5cc0d6d 🐛 Source Amazon Ads: Add missing reporting metrics to the source connector (#25412)
* Add missing reporting metrics to source connector for Amazon Ads

* update changelog

* correct version
2023-05-25 15:57:21 -03:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Roman Yermilov [GL]
94d0d61248 Source Amazon Ads: get rid of fail_on_extra_columns: false in SAT (#25913)
* Source Amazon Ads: small schema fixes

* Source Amazon Ads: update changelog

* Source Amazon Ads: update unittest

* Source Amazon Ads: unittest additional property is boolean

* Source Amazon Ads: bump version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-16 21:47:33 +04:00
Serhii Chvaliuk
1cbd9252ba Source: Amazon Ads - Improve error handling for attribution_report(s) streams (#25885)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-05-16 17:43:26 +03:00
Roman Yermilov [GL]
c577e44633 Source Amazon Ads: add availability strategy for basic streams (#25792)
* Source Amazon Ads: add availability strategy for basic streams

* auto-bump connector version

* Source Amazon Ads: update metadata.yaml

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-08 11:06:17 +04:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Serhii Lazebnyi
ab784c5d90 Source: Amazon Ads - Add pk validator for reports when expected pk is not returned (#25146)
* Add pk validator for reports when expected pk is not returned

* Update PR number

* Fix formating

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-13 19:06:29 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Serhii Lazebnyi
d9a6ac9db8 Source Amazon Ads: Migrate products_report stream to API v3 (#22355)
* Migrate products_report stream to API v3

* Updated PR number

* fix

Signed-off-by: Your Name <you@example.com>

* fix tests

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Fixed unttests

* Fix Metric response

* Fix typo

* Fix integrations test tdue migration

* auto-bump connector version

---------

Signed-off-by: Your Name <you@example.com>
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-31 02:01:38 +02:00
Ella Rohm-Ensing
9cfc647fa5 Ella/fail on extra fields (#23985)
* Pass argument along, add test that should pass and test that should fail

* Add tests with additionalProperties

* Set additionalproperties=false when not set|

* Parametrize test cases

* Make the behavior 'optional'

* Fix parametrization for all combinations

* Improve CI credentials README and rename param

* Update naming to be clearer about columns only

* record_has_unexpected_field > record_has_unexpected_column

* Automated Change

* Add bypass for beta + ga connectors that failed

* Update docs and TODOs

* Update changelog and dockerfile

* Update TODO

* Update a few neglected connectors

* Remove uploaded file

* Update dockerfile after merge conflict

---------

Co-authored-by: marcosmarxm <marcosmarxm@users.noreply.github.com>
2023-03-23 22:22:44 +00:00
Augustin
bad5bce8ce CDK: remove unexpected error swallowing on abstract source's check method (#24240) 2023-03-23 13:04:51 +00:00
Ganpat Agarwal
2b81405f2f 🎉 Source Amazon Ads : use optional config report_record_types (#18677)
* use optional config report_record_types

* update spec

* update image version

* update

* correct indentation

* remove lint changes

* add tests for sponsored products report

* update spec

* add test for video report

* increment amazon-ads connector version

* add test_strictness_level as high

* gradle format update

* update

* update tests for new METRIC_RESPONSE

* align validation and transformation

* bump connector version

* auto-bump connector version

---------

Co-authored-by: Juozas V <monai@cure.lt>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Nataly Merezhuk <65251165+natalyjazzviolin@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-14 15:43:04 -03:00
Serhii Lazebnyi
3d8f81a3e4 Connector Health: Fixed SAT for Amazon Ads, Faker, Hubspot, Instagram, Pinterest, Sentry, Zendesk Talk (#23600)
* Fixed SAT for some connecotrs

* Fix unittest pinterest

* add allowedHosts

* Updated unittest for instagram

* Fix expected records instagram
2023-03-01 10:09:19 +01:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Augustin
fbd6dbf091 python-connectors: remove explicit dependencies to connector-acceptance-test build (#23332) 2023-02-22 21:21:38 +01:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Roman Yermilov [GL]
7d13ad5d5d Source Amazon Ads: fix reports stream records primary keys (#21677)
* Source Amazon Ads: fix reports stream records primary keys

* Source Amazon Ads: update changelog

* #1332 source Amazon Ads: review fixes

* #1332 source amazon ads: remove unused imports

* #1332 source amazon ads: bump major version

* auto-bump connector version

---------

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-02-06 18:53:18 +02:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Ella Rohm-Ensing
fe2ae98767 Turn off default httpavailabilitystrategy for source-amazon-ads (GA) (#22038)
* Turn off default for source-amazon-ads (GA)

* update import

* Update Dockerfile

* Update amazon-ads.md

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-30 13:02:19 -05:00
Daniele Dapuzzo
13eac98309 🎉 Source Amazon Ads: add option to customize look back window value (#19491)
* add new configurable field

* update docker image version

* amazon-ads.md updated

* remove unnecessary constant

* rollback changes inside seed files

* fully revert changes

* pre load lbw default value

* fix value access

* update config fixture

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2023-01-27 06:03:00 -03:00
Arsen Losenko
d7cbc790b8 Source Amazon Ads: enable SAT high strictness (#21646)
* Source Amazon Ads: enable SAT high strictness

* Update SAT config, remove config_report.json for now
2023-01-24 11:38:48 +01:00
Serhii Chvaliuk
e51853e046 Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (partial revert of #20662) (#21082)
* Revert "Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)"

This reverts commit ec995959f7.

* fix

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* revert master

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* revert to master

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* bump 0.1.27

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* fix

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* amazon-ads.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* amazon-ads.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* auto-bump connector version

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
2023-01-06 09:18:50 -06:00
Serhii Chvaliuk
ec995959f7 Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-01-04 20:09:00 +02:00
Evan Tahler
e39e1898c5 Expected Records to .jsonl format (#20850)
* Expected Records to `.jsonl` format

* fix formatting template

* remove endline

* update templates

* Update docs/connector-development/testing-connectors/source-acceptance-tests-reference.md

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2023-01-03 15:55:36 -08:00
Serhii Chvaliuk
70635632ce Source Amazon Ads: remove "report_wait_timeout", "report_generation_max_retries" from config (#18985)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-09 00:10:59 +02:00
darynaishchenko
02095d20da fixed acceptance tests (#18699) 2022-11-02 14:03:36 +02:00
Brian Lai
3521802545 Update Python Source connectors to use the latest airbyte-cdk minor version (#18261)
* update all connectors currently only adhering to patch versions to pull latest minor version

* also add greenhouse

* fix scaffolds
2022-10-20 16:06:02 -04:00
Sage Watterworth
0d9b4e5dce 🎉 Source Amazon Ads: filters for state on brand, product and display campaigns (#17475)
* amazon ad status filter

* update test report stream with parametrized campaigns

* remove config files related to previous acceptance test. revert to master acceptance test set up

* amazon ad status filter

* update test report stream with parametrized campaigns

* remove config files related to previous acceptance test. revert to master acceptance test set up

* oct 17 edits

* oct 17 edits: 2

* bump dockerfile to 1.24

* fix: match the cdk version to new one

* auto-bump connector version

Co-authored-by: sajarin <sajarindider@gmail.com>
Co-authored-by: Harshith Mullapudi <harshithmullapudi@interviewkickstart.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-20 12:50:55 -04:00
Sajarin
3154cfa68e Source Amazon Ads - bump dockerfile to 1.23 (#17906)
* bump dockerfile to 1.23

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-19 12:47:04 -04:00
Serhii Chvaliuk
220513db96 Source Amazon Ads : SAT recover expected_records.txt (#17938)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-13 15:05:45 +03:00
Greg Solovyev
034b3fb100 Remove unpulbished version of source-amazon-ads from seed file (#17904) 2022-10-12 13:24:53 -07:00
Ganpat Agarwal
ca198ecf36 update amazon-ads version to 0.1.23 (#17870) 2022-10-12 14:27:44 -04:00
Evan Tahler
49cb3360de Remove redundant title labels from connector specs (#17544)
* Remove redundant title labels from connector specs

* Manually update specs

* add env variable

* Remove debugging log
2022-10-05 12:58:38 -07:00
Ganpat Agarwal
454f84d3ed 🎉 Source Amazon Ads : Add attribution reports (#16342)
* use data field for json response

* add attribution reports

* update changelog

* add atrribution report integration test

* clean up expected_records
add empty streams to acceptance test config

* handle exception for profile

* update tests

Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
2022-10-03 10:24:33 -04:00