1
0
mirror of synced 2025-12-22 11:31:02 -05:00
Commit Graph

31012 Commits

Author SHA1 Message Date
Sherif Nada
5a59e27f94 add destination tutorial to SUMMARY.md 2021-06-29 14:45:05 -07:00
Sherif A. Nada
c2b3c23a92 respond to post-merge code review for destination docs (#4418) 2021-06-29 14:35:22 -07:00
Sherif A. Nada
921263845e 📚 Add a tutorial for how to build a java destination (#4384)
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>

Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-06-29 13:54:37 -07:00
Subodh Kant Chaturvedi
887752822c 🎉 introduce automatic migration at the startup of server for docker environment (#3980)
* introduce automatic migration at the startup of server

* handle versions with non-zero patch

* it works!!!

* add dummy data

* cleanup orphan configs

* add more assertions

* format + add comments

* move migration acceptance test to acceptance test directory

* add automatic migration test to the build

* address review comments

* missed out on these

* format

* add more assertions

* format

* fix test

* format

* use default port for temporal

* move seed to server + introduce atomice replacement for config

* make tests better

* remove unwanted changes

* move atomic replacement logic behind persistence + pass path to latest seeds

* format

* update seeds

* review comments

* update seeds

* merge latest seeds with configs

* fix bug around latest seed

* update seed

* update seed

* seeds should be populated by separate container

* address review comment + change latest definition url

* update seeds

* format

* update seed references

* update seed

* update seed

* update seed

* update seed references

* update seed references + add Migration Acceptance Test

* update seed container in kube + disable automatic migration for kube + update docs

* update docs

* address review comments from Michel

* update doc

* temporary commmit to see if build becomes green

* delete seeds from airbyte config + undo temp commit
2021-06-29 23:50:00 +05:30
Marcos Marx
a8513af3a2 Add section how run custom dbt run (#4347)
* add section how run custom dbt run

* correct comments from @andresbravog

* add issue to improve the custom parsing

* eof
2021-06-29 13:30:49 -03:00
Jenny Brown
ea7002ec33 Add paged results to Job History retrieval (#4323)
* Add paged results to job history retrieval.

* Make job histories come back in a sort order controlled by the sql query

* Increased default job history to show, to make better defaults for UI before paging

* Code review cleanup, constants.

* Update airbyte-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobPersistence.java

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobPersistence.java

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Use a Pagination object for pagesize and offset

* NPE fix when pagination is not sent with the request and it falls back to defaults

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-28 12:53:23 -05:00
Fred Reimer
008b79fbb9 FIx schema for required properties (#4379)
Co-authored-by: Fred Reimer <freimer@presidio.com>
2021-06-28 13:42:45 -03:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Kirill Chernov
40aed2249d 🎉 fix Google Adwords CI & publish new version (#4205)
* added SAT tests, deleted incremental test sfrom pipeline

* pr changes and tests uncommented

* deleted facebook file from pr

* deleted facebook file from pr

* removed tap adwords from install_requires

* Update airbyte-integrations/connectors/source-google-adwords-singer/acceptance-test-config.yml

* added changelog info

* version bump

* version bump

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-06-26 01:45:22 +03:00
Sherif Nada
ea975027bc Merge branch 'master' of github.com:airbytehq/airbyte 2021-06-25 12:45:04 -07:00
Sherif Nada
10d1168a03 update docs 2021-06-25 12:44:57 -07:00
Abhi Vaidyanatha
00437b7e23 Update roadmap with more relevant information. (#4355)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-25 12:27:29 -07:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Prasanna Ram Venkatachalam
0c25e1cca1 🎉 New Destination: Google PubSub (#4339)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-25 11:09:07 -07:00
Oleksandr
0e0d0a102e 🐛 Bugfix for pagination in Shopify Source Connector #4284 (#4290)
#4284 - Bugfix for pagination in Shopify Source Connector

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-25 19:45:45 +03:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Yaroslav Dudar
2ccb1c2de0 🎉 New source: AWS CloudTrail (#4122)
AWS CloudTrail connector initial release
2021-06-25 11:10:07 +03:00
Jared Rhizor
5fb98b3c87 Bump version: 0.26.3-alpha → 0.26.4-alpha (#4336) 2021-06-24 19:03:47 -07:00
Marcos Marx
d269f91871 correct url (#4293) 2021-06-24 21:56:08 -03:00
Marcos Marx
77f7096d14 🐛 Source Mailchimp: use datacenter parameter from apikey info (#4285)
* replace datacenter parameter using apikey info

* change to use dynamic property

* add dynamic property

* bump version

* change config versino
2021-06-24 21:55:13 -03:00
Sherif A. Nada
c130465048 Release Shopify source 0.1.6 (#4332) 2021-06-24 16:39:35 -07:00
jacqueskpoty
e01576d461 🎉 Source Shopify: add pages, price rules and discount codes streams (#4009)
Co-authored-by: jsewawilson <jwilson@pebblepost.com>
2021-06-24 16:18:15 -07:00
Harshith Mullapudi
f4ddf4042d 🐛 Bugfix incorrectly failing check connection operations in Google Ads and Amazon AP API connectors (#4288) 2021-06-24 15:58:37 -07:00
Abhi Vaidyanatha
539db2c8b8 6/24/2021 connector changelog (#4325)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-24 14:25:29 -07:00
Abhi Vaidyanatha
6e11950a66 Add Db2 to Summary.md (#4324)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-24 14:14:05 -07:00
Abhi Vaidyanatha
8522a1cd9f Change all instances of DBT to dbt in markdown files. (#4308)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-23 19:00:44 -07:00
LiRen Tu
0009c97324 Bump version: 0.26.2-alpha → 0.26.3-alpha (#4310) 2021-06-23 16:56:42 -07:00
LiRen Tu
e2c5b1d4c7 🎉 S3 destination: Avro & Jsonl output (#4227)
* Add jsonl format to spec.json

* Implement jsonl writer

* Add documentation

* Add acceptance test

* Update document

* Bump version

* Update document example

* Implement avro writer

* Implement compression codec

* Update documentation

* Revise documentation

* Add more tests

* Add acceptance test

* Format code

* Create helper method for name updater

* Update csv doc with normalization

* Update version date
2021-06-23 14:54:39 -07:00
Jared Rhizor
38ed70c816 remove s3 logging from docker-compose and update upgrade docs (#4278) 2021-06-23 08:12:55 -07:00
irynakruk
0a4712cbe7 🎉 New Source: IBM DB2 (#4197)
* Db2 source: implementation and acceptance tests

* Db2 source: added documentation, build info, Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: updated docs

* Db2 source: added test for the special values in Decfloat

* Formatter changes

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-06-23 18:06:31 +03:00
Christophe Duong
75a1dda07e 🎉 New BigQuery destination with Structured/Repeated Records (#4176) 2021-06-23 16:19:36 +02:00
Eugene Kulak
2f7c15a540 Source Facebook: migrate to CDK (#3743)
* move to CDK implementation
* make insights streams configurable
* improve Async implementation using slices
* minimize set of streams in configured_catalog.json
* apply suggestions from @sherifnada
* improve async jobs handling even more

Co-authored-by: Ka-Ping Yee <zestyping@gmail.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-22 18:48:17 -04:00
Jenny Brown
b3c65050a2 Standardize error structure in json responses. (#4170)
* Standardize error structure in json responses. Use named types for raising exceptions instead of http status codes peppered throughout the code.

* Whitespace fix

* Rename fix

* Consistent naming for things related to KnownException, for traceability.

* For input field validation errors, list the affected fields as separate json objects. Also uses cleaner helpers for exception mappers.

* Whitespace

* Rename to clarify exception class hierarchy

* Wrap not found responses in a schema and adjusted mapper to match

* Include issue id in comments.
2021-06-22 17:42:48 -05:00
jrhizor
400d730d27 hotfix: run generateSourceTestDocs 2021-06-22 10:26:13 -07:00
Vadym
4c2fc3ba00 Bump harvest version (#4277)
* Bump source Harvest docker version

* Update harvest.md docs
2021-06-22 20:22:02 +03:00
Jared Rhizor
0eb72867b6 test for AIRBYTE_ENTRYPOINT env variable in connectors (#4255)
* test for AIRBYTE_ENTRYPOINT env variable in connectors

* use limit=2

* fix entrypoint checker
2021-06-22 09:47:53 -07:00
Abhi Vaidyanatha
39fe5fed16 Add 0.26.2 changelog (#4259)
* Add 0.26.2 changelog

* Removing the core

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-21 17:10:54 -07:00
Marcos Marx
b57736b0ce Documentation correct markdown list in csv and json destination (#4165)
* correct list in markdown csv and json

* add copy cmd and change variable name to used in dest settings
2021-06-21 19:53:26 -03:00
Abhi Vaidyanatha
cd8b2a8299 Add architecture diagram to K8s contribution doc (#4257)
* Add architecture diagram to contribution doc

* Remove extra line

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-21 15:45:04 -07:00
Abhi Vaidyanatha
76f1b942f6 Update deployment title (#4256)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-21 15:23:27 -07:00
Jared Rhizor
51bb5ccf42 kubernetes v2 docs (#4234)
* some logs

* Add cloud logging section.

* Add instructions on upgrading Kube K8s.

* minor edit

* minor clarification

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-06-21 15:00:28 -07:00
Marcos Marx
156629ca95 Bump Harvest Source connector 0.1.2 (#4223)
* fixes #4221. Fix broken spec file property and point to correct docs page

* bump harvest connector

* add changelog for harvest

* bump dockerfile version

Co-authored-by: Tyler Schroeder <tyler@tylerschroeder.com>
2021-06-21 15:22:48 -03:00
Jared Rhizor
cb09c0403c Bump version: 0.26.1-alpha → 0.26.2-alpha (#4249) 2021-06-21 11:21:03 -07:00
Jared Rhizor
27ae6f182e fix build (#4250) 2021-06-21 11:17:05 -07:00
Christophe Duong
55d3b4dc0c Add details to docs (#4244) 2021-06-21 18:36:31 +02:00
Christophe Duong
6b112b0bd7 Write how to connect to external db (#4211) 2021-06-21 12:30:52 +02:00
Marcos Marx
88d7953a18 Add CHANGELOG for Sources (#4215)
* add changelog for postres source

* add changelog for mysql source

* add changelog for mssql source

* add changelog for file source

* add changelog for google sheets

* add full changelog for file source

* add full changelog for google sheet source

* add full changelog for mssql source

* add changelog for http requests

* add changelog for salesforce source

* add changelog for facebook marketing source

* add changelog for google analytics source

* add changelog for shopify native source connector
2021-06-18 17:45:09 -03:00
Abhi Vaidyanatha
135af259d1 Fix formatting on 6/18 changelog (#4217)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-18 13:29:43 -07:00
Abhi Vaidyanatha
023539ec6a Finishing touches on Changelog 6/17 (#4216)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-18 13:10:13 -07:00
Davin Chia
8d037ae3c0 Bump version: 0.26.0-alpha → 0.26.1-alpha (#4196) 2021-06-18 14:26:52 +08:00