jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 11:01:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

206 Commits

Author SHA1 Message Date
Denis Davydov
2570f9ce73 Source file: remove nested properties (#14535) 
* #13787 source file: remove nested properties

* #13787: upd changelog

* #13787 revert processing json array, add unit test

* #13787 source file: upgrade pyarrow to allow builds on aarch64

* #13787 source file: upd version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 11:29:51 +03:00
Christophe Duong
43f905b8f9 🎉 Add YAML format to source-file reader (#14588) 
* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-12 19:34:17 +02:00
Marcos Marx
d3b1d15ade Source File: correct spec json to match json format (#13738) 2022-06-28 13:11:35 -03:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233) 
* Bump year in license short to 2022

* remove protocol from cdk
 2022-05-26 15:00:42 -03:00
George Claireaux
26aa43b31e Source File: Update Dockerfile to latest Airbyte template (#11969) 
* update to our newer Dockerfile format

* fixes to source-file spec

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-13 16:39:59 +01:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763) 
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
 2022-04-11 20:51:37 -07:00
Serhii Chvaliuk
c021426752 🎉 Source File: Update airbyte-cdk>=0.1.47 to be compatible with latest SAT tests (#9974) 
* bump version 0.2.8 -> 0.2.9

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-02-01 19:50:47 +02:00
dependabot[bot]
757ba946bc Bump lxml in /airbyte-integrations/connectors/source-file (#8758) 
Bumps [lxml](https://github.com/lxml/lxml) from 4.6.3 to 4.6.5.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.6.3...lxml-4.6.5)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-14 18:06:06 -03:00
Iryna Grankova
fefd5759c8 Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-12-11 01:47:29 +02:00
Yevhenii
269298c854 🎉 Source File: Migrate File source to CDK structure (#7387) 
* Migrate File source to CDK structure

* fix .dockerignore file

* remove SAT requirements

* update Dockerfile

* change Dockerfile to base images python:3.7-slim

* add SAT tests

* update tests

* add secret/config.json for source-file

* update changelogs
 2021-10-29 13:15:13 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Marcos Marx
b3c8214b72 Bump version for File connector (#5689) 
* source-file: adding support to XLSB format

* contributor pr + bump connector version

Co-authored-by: Muutech <info@muutech.com>
 2021-08-26 21:25:32 -03:00
Muutech Monitoring Solutions
ad22ee601e source-file: adding support to XLSB format (#5613) 2021-08-26 21:09:23 -03:00
Marcos Marx
e269c5f976 Test/Bump Version Source File (#5006) 
* Allow SFTP to define port

* Define port as a string

* Provide a clearer error if port parsing fails

* bump source file version

* try to resolve 409 azure

Co-authored-by: Madison Swain-Bowden <bowdenm@spu.edu>
 2021-07-27 19:29:13 -03:00
Madison Swain-Bowden
2d5d7e8e98 🎉 Allow non-default source SFTP port (#4953) 
* Allow SFTP to define port

* Define port as a string

* Provide a clearer error if port parsing fails
 2021-07-27 18:52:55 -03:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973) 
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-09 13:01:45 -07:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Davin Chia
cd24ccd064 Release Source File (#3771) 2021-05-31 21:35:57 -07:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660) 
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-06-01 10:42:13 +08:00
Davin Chia
65c04d6b9e Fix Source-File build. (#3738) 
The main problem seems to be the newer pip resolver isn't happy with our current dependencies. Changing this was enough to fix it.

Got this version by removing the constraint, running the install step, then using the installed version from pip list.
 2021-05-29 22:39:06 +08:00
Abhi Vaidyanatha
6ffada861b Add Rust as a connector specific dependency to source-file (#3426) 
* Add Rust as a connector specific dependency to source-file

* Add more details about installation.

* Markdown lines are weird.

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-05-14 17:42:41 -07:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017) 
* as a java developer I want to be able to run spotlessApply without changing styles in python code
 2021-04-23 09:21:41 -07:00
oleh.zorenko
df80593108 csv discovery takes too much memory (#2089) (#2883) 
* csv discovery takes too much memory (#2089)

Use `chunksize` parameter when reading csv files.

* csv discovery takes too much memory (#2089)

Handle `None` value for `fields` variable in `client.read()` function.

* csv discovery takes too much memory (#2089)

Refactor code in `client.read()` function. Remove duplicate code.

* csv discovery takes too much memory (#2089)

Implement version with yield statements.

* csv discovery takes too much memory (#2089)

Update return annotation for load_dataframes().

* csv discovery takes too much memory (#2089)

Remove unused import.

* csv discovery takes too much memory (#2089)

Update source-file connector version from `0.2.1` to `0.2.2`.
 2021-04-16 19:51:53 +03:00
Sherif A. Nada
2d2d0173d6 Update codegen template READMEs (#2873) 2021-04-13 11:41:52 -07:00
Jared Rhizor
0609011e05 fix connector base versioning (#2726) 
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
 2021-04-02 22:46:06 -07:00
Christophe Duong
bc1547f185 Fix integration tests of connectors related to destination_sync_mode and sync_mode being required (#2688) 2021-04-01 09:58:38 +02:00
Sherif Nada
77b72dcf5b Revert "Revert "Merge branch 'master' of github.com:airbytehq/airbyte"" 
This reverts commit de66bf010d.
 2021-03-23 15:12:01 -07:00
Sherif Nada
de66bf010d Revert "Merge branch 'master' of github.com:airbytehq/airbyte" 
This reverts commit a3a27f2fa2, reversing
changes made to 5917feee28.
 2021-03-23 15:09:44 -07:00
Sherif A. Nada
41e14e24d3 fix source-file CI (#2563) 2021-03-22 16:00:18 -07:00
dependabot[bot]
d7c8476923 Bump lxml in /airbyte-integrations/connectors/source-file (#2550) 
Bumps [lxml](https://github.com/lxml/lxml) from 4.6.2 to 4.6.3.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.6.2...lxml-4.6.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-03-22 13:37:03 -07:00
Sherif A. Nada
02a5b9b8a8 depend on published base singer/python images in existing connectors (#2469) 2021-03-15 18:21:05 -07:00
Sherif A. Nada
fe22594bb9 update all python Readmes to describe dev process without gradle (#2442) 2021-03-15 18:05:38 -07:00
Sherif A. Nada
2139a9656b Cleanup Python packages (#2440) 2021-03-15 01:24:21 -07:00
Christophe Duong
070575ffdf Protocol allows future / unknown properties (#2238) 
* Allow new extra properties in validation
* Create migration script to upgrade all connectors versions
* Bumpversion of all connectors
 2021-03-09 13:36:36 +01:00
Sherif A. Nada
ac0bcf7d01 Add comment at the top of all requirements.txt files clarifying they should not be used for dependencies (#2366) 2021-03-08 19:01:47 -08:00
Sherif A. Nada
d4267818ae Source File: Support JsonL format (#2118) 2021-02-18 13:54:26 -08:00
vitaliizazmic
caf673dfab Test file formats #1392 (#1768) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-02-02 14:49:11 -08:00
Eugene K
e9769c8159 Source File: fix issue with boolean types in schema (#1873) 2021-01-28 23:02:53 -08:00
Eugene K
a54dd89418 Adopt connector best practices for File Source (providers) #1584 (#1738) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-01-27 16:49:33 -08:00
Charles
639cb2a7da kill source stock ticker (#1558) 
* remove symlinks from python sources and template

* kill stock ticker
 2021-01-06 17:05:20 -08:00
Christophe Duong
140f5a234d BumpVersions of some sources (#1363) 2020-12-17 18:58:33 +01:00
Christophe Duong
e92e9aa8c4 Refactor python sources (#1331) 
* Make ConfigContainer internal to SingerSource only
 2020-12-16 17:52:17 +01:00
Christophe Duong
104436c8f6 Handle NaN values from python to JSON (#1246) (#1249) 2020-12-08 18:05:58 +01:00
Sherif A. Nada
e8a332ae65 Standard source incremental tests (#1175) 2020-12-04 09:54:10 -08:00
Charles
02819a4b87 Incremental Docs and Data Model Update (#1021) 2020-11-19 22:07:32 -08:00
Sherif A. Nada
f4c3ac70f9 annotate secret fields (#1012) 2020-11-19 15:13:23 -08:00
Charles
e7edb2c858 Adding incremental to the catalog data model (#998) 
* Add ConfiguredAirbyteCatalog and ConfiguredAirbyteStream
 2020-11-18 14:15:59 -08:00
Christophe Duong
0d7ca6f424 Fix source file comments from #935 (#944) 
Adjust Source File from manual testings in #935
 2020-11-13 13:34:34 +01:00