1
0
mirror of synced 2025-12-21 19:11:14 -05:00
Commit Graph

157 Commits

Author SHA1 Message Date
Serhii Lazebnyi
ae41c30b1a 🚨🚨🐛 Source Google Analytics: revert : auto replacement to underscore in column names (#30608) 2023-09-21 22:48:38 +02:00
Ryan Br
317f7acc22 Bigquery uses async framework (#30069)
Co-authored-by: tryangul <tryangul@users.noreply.github.com>
Co-authored-by: benmoriceau <benoit@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
2023-09-20 16:34:07 -05:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Serhii Lazebnyi
fa6ae0f2c4 Source Google Analytics (GAv4): add yearWeek, yearMonth, year dimensions cursor (#30421) 2023-09-19 17:09:09 +02:00
Brian Lai
90375ee501 🐛 [source-google-analytics-data-api] fix issue if metrics or dimensions api response does not have name field (#30506) 2023-09-17 13:53:17 -04:00
Serhii Lazebnyi
a984b753c2 Source Google Analytics (GAv4): change start date to optional; add suggested streams and update error… (#30417) 2023-09-15 21:38:42 +02:00
Baz
3f1f13c4d7 🐛 Source Google Analytics Data API: fix duplicated stream instance (#30424) 2023-09-15 11:59:17 +03:00
Edward Gao
68c6b01937 Destinations v2: threadsafe setup (#30439)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2023-09-14 11:56:12 -05:00
Roman Yermilov [GL]
6442bd110c Source Google Analytics Data API: ability to add multiple property ids (#30152) 2023-09-14 15:12:14 +04:00
Brian Lai
9593a8d341 🐛 [source-google-analytics-data-api] revert back to 1.1.3 (#30419) 2023-09-13 21:49:17 -04:00
Serhii Lazebnyi
9f3d120b67 Source Google Analytics Data API: add new reports (#30290) 2023-09-13 15:09:43 +02:00
Ben Church
4d5e17bc90 [skip ci] Update test_incremental to be unaware when source defines the cursor (#27872)
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-09-05 15:57:18 -07:00
Marius Posta
f5c7c1c0b8 chore: get ./gradlew format to pass for the whole repo (same java style) (#29786) 2023-08-24 05:09:42 -05:00
Ben Church
40781313da Update Internal Fields: update ql levels to better resemble previous high strictness (#29450) 2023-08-15 18:03:10 -05:00
Augustin
00d9462216 cat/connectors-ci: replace docker runner with dagger runner in CAT (#28000) 2023-08-11 17:58:48 +02:00
Christo Grabowski
6cccce84f7 Source Google Analytics GA4: update spec file (#29103)
* update spec for google ga4

* bump changelog
2023-08-08 14:54:26 -04:00
Ben Church
2f7deaee02 [skip ci] Metadata: Remove leading underscore (#29024)
* DNC

* Add test models

* Add model test

* Remove underscore from metadata files

* Regenerate models

* Add test to check for key transformation

* Allow additional fields on metadata

* Delete transform
2023-08-03 10:56:13 -07:00
Ben Church
e9490e3fb6 Connector Levels: Add new internal metadata fields (#28904)
* Add airbyte internal

* Add tests

* First pass

* Set destinations to same levels as sources

* Best guess at missing statuses

* Best guess at _ql

* Add separate enum class

* Fix support level name

* Update templates

* Add one more test
2023-08-01 18:08:33 -05:00
Daryna Ishchenko
624bb815d0 increased timeout for incremental tests (#28559) 2023-07-24 14:05:03 +03:00
Denys Davydov
e9fba26a99 🐛 Source Google Analytics Data API: limit custom report page size (#27909)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #2277 source google data analytics: limit page size of custom reports streams

* upd changelog

* limit page size only for check command
2023-07-03 20:19:00 +03:00
Denys Davydov
3d14fc721f 🐛 Source GA Data API: change page size when calling check() (#27718)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #2277 source GA: limit page size when calling check

* source GA: upd changelog

* code formatting

* #2277 review fixes
2023-06-29 10:52:04 +03:00
Evan Tahler
037a522f8c Google Source connector licenses to Elv2 (#27738)
* Google Source connector licenses to Elv2

* bump PR Ids

* Update metadata.yaml
2023-06-27 03:38:13 -04:00
Serhii Chvaliuk
019153f178 Source Google Analytics v4: - add pk and lookback window (#26283)
* primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* bump 0.2.3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* test_parse_response fixed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* metadata = None added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* LOOKBACK_WINDOW added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* google-analytics-data-api.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* threshold_days: 2 added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* remove uuid from acceptance-test-config.yml

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* uuid removed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* oss_registry.json - removed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range added for pivot_report pk

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* improve LOOKBACK_WINDOW

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range -> (startDate, endDate)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* bump 0.3.0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* get_source_defined_primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* added comment about cohort get_json_schema

* Major Version Number Bump

---------

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: grubberr <grubberr@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <darina.ishchenko17@gmail.com>
2023-06-22 19:00:38 +03:00
Artem Inzhyyants
5f35afee11 Source Google Analytics Data API: Fix formatting (#27531)
* Source Google Analytics Data API: Fix formatting

* Source Stripe: Bump version
2023-06-21 12:54:53 +02:00
Parker Mossman
c3dd582cb6 added resource requirements to the wrong connector, try again (#27449) 2023-06-16 16:36:24 -04:00
Artem Inzhyyants
8fb73442b2 Source Google Analytics Data API: update api rate limit messages (#27207)
* Source Google Analytics Data API: update api rate limit messages

* Source Google Analytics Data API: update docs
2023-06-12 14:17:44 +02:00
Artem Inzhyyants
797eb4f427 Source google analytics data api: Improve error messages (#27175)
* Source Google Analytics v4: refactor error messages

* Source Google Analytics Data API: improve error messages

* Source Google Analytics Data API: Refactor

* Source Google Analytics Data API: bump version + update docs

* Source Google Analytics Data API: fix test config

* Source Google Analytics Data API: update message
2023-06-09 14:11:44 +02:00
Augustin
9099aa78cc source-google-analytics-data-api: fix formatting (#27011) 2023-06-05 09:21:06 -05:00
Augustin
d8820d6e80 Fix python connectors formatting errors with black / isort (#26951)
* format source-tiktok-marketing

* format source-twilio

* format source-instgram

* format source-amazon-ads

* format source-google-search-console

* format source-stripe

* format source-google-analytics-data-api

* format source-facebook-marketing

* format source-zendesk-support

* format source-file
2023-06-02 09:26:04 -05:00
Denys Davydov
7c6b7da791 🐛 Source Google Analytics Data API: deprecate authSpecification (#26887)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #26230 source GA data api: deprecate authSpecification

* source GA data API: update changelog
2023-06-01 13:44:49 +03:00
mpetrykin
746ccb4035 Source Google Analytics (GA4): Fix Pagination (#26126)
* Fix paggination and add offset and limit to acceptable parameters in request body

* Change next_page_token and add tests

* Update dockerImageTag

* Update PR version

* Remove minimum, maximum, pattern fields

* Remove pattern limit and offset from test_source.py

* Remove offset and limit string type

* Remove offset and limit string type

* Increase limit number to 100000 and remove limit and offset from parameters

* Change return type value of next_page_token from int to dict

* Change return type value of next_page_token from int to dict

* Change page_size to offset and add constant PAGE_SIZE equals 100000

* Add comment to PAGE_SIZE constant and add constant to unit tests

* Remove offset and limit from PivotReport

* Import PAGE_SIZE in unit_tests from source.py

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
2023-05-31 13:57:34 -04:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Serhii Lazebnyi
1060415870 🎉Categorized Config Errors Accurately for Google Analytics 4 (GA4) and Google Ads (#25987)
* Categorized Config Errors Accurately

* Update PR number

* Update error list for GAv4

* Updated version

* Updated formating

* auto-bump connector version

* Skip spec backward compatibility

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-12 21:39:10 +02:00
Baz
d9f653bc47 🐛 Source Google Analytics V4 Data API: handle 429 - potentiallyThresholdedRequestsPerHour (#26008) 2023-05-12 09:01:28 +03:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Denys Davydov
3f327f60f3 Source Google Analytics Data API: implement advanced reports (#25179)
* #11689 source Google Analytics Data API: implement advanced reports

* #11689 source GA data API: upd changelog

* #11689 source GA data API: fix CAT

* #11689 source GA data API: fix CAT

* Have empty dict as default option

* Remove dateRange and dimensions from required params

* Generate source defintions manually

---------

Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com>
2023-04-25 19:55:55 +03:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Serhii Lazebnyi
03e1ec4cb8 Connector Health: Fixed SAT for close-com, facebook-marketing, freshsales, greenhouse, hubspot, intercom, jira, mixpanel, monday, pipedrive, recharge, sentry, slack, strava, stripe, trello, twitter, youtube-analytics, twillio, faker (#24361)
* Fix expected records for hubspot and intercom

* Fix typo for intercom

* Fix close com, fecebook marketing, freshsales, greenhouse expected records

* Fix intercom CAT

* Fix unittest for new pydentic version

* Fix facebook marketing unittests and jira CAT

* Fix formating for facebook marketing, jira and CAT for monday

* Fix pipedrive and recharge CAT

* Fix CAT for Slack, Strava, Stripe, Trello and Twitter

* Fix expected records for slack

* Fix CAT for Youtube and Sentry

* Fix unittests for mixpanel

* Added allowedHosts and fix facebook marketing CAT

* Fix CAT for Faker and Twillio

* Fix expected records for hubspot

* Fix expected records for Sentry

* Fix check status for GAv4
2023-03-28 20:31:41 +02:00
Artem Inzhyyants
3e812513b0 Source Google Analytics v4: certify to GA (#23872)
* Source Google Analytics v4: check custom reports; remove date as required field

* Source Google Analytics: fix unit tests

* Source Google Analytics: fix unit tests

* Source Google Analytics: update to latest cdk

* Source Google Analytics: http -> https

* Source Google Analytics: add allowed hosts

* Source Google Analytics: docs update

* Apply suggestions from code review

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Source Google Analytics: docs update

* Source Google Analytics: add expected records

* auto-bump connector version

---------

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-16 15:41:29 +01:00
Evan Tahler
3b47470fe9 Fixup Google Analytics DocumentationURLs and Spec Names (#24078)
* Fixup Google Analytics DocumentationURLs

* Fixup URLs

* Automated Change

* fix urls

* Automated Change

---------

Co-authored-by: evantahler <evantahler@users.noreply.github.com>
2023-03-15 09:38:53 -07:00
Baz
00752aaa50 🐛 Source Google Analytics 4 (GA4): improve rate limits messages and retry logic (#23822) 2023-03-11 21:15:07 +02:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Serhii Chvaliuk
59ff2a2f59 Source Google Analytics Data API: slicer updated, unit tests added (#21169)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-01-11 18:26:29 +02:00
Serhii Chvaliuk
4923edf20a Source: Google Analytics 4 (GA4) - improve config validation and SAT (#20889)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-01-09 16:52:08 +02:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
Evan Tahler
49cb3360de Remove redundant title labels from connector specs (#17544)
* Remove redundant title labels from connector specs

* Manually update specs

* add env variable

* Remove debugging log
2022-10-05 12:58:38 -07:00