jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-05 21:02:13 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,162 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

85 Commits

Author SHA1 Message Date
Evan Tahler
79dba56923 S3 and GCS connector license to Elv2 (#27725) 
* S3 and GCS connector license to Elv2

* docs update

* docs
 2023-06-26 18:27:18 -05:00
Artem Inzhyyants
c68afefdf0 Source S3: handle Bucket Access Errors (#27651) 
* Source S3: handle bucket access errors

* Source S3: update docs
 2023-06-23 13:22:57 +02:00
Artem Inzhyyants
0c3d4499d6 Source S3: fix start date (#27611) 
* Source S3: fix start date

* Source S3: update docs

* Source S3: bump version
 2023-06-22 17:17:52 +02:00
Artem Inzhyyants
eef872e9f3 Source S3: Add logging for file reading (#27604) 
* Source S3: Add logging for file reading

* Source S3: update docs
 2023-06-22 10:53:32 +02:00
Serhii Lazebnyi
da67a60b7c Source Confluence, Source Greenhouse, Source Hubspot, Source Stripe, Source Close com, Source Klaviyo, Source Notion, Source Pinterest, Source Snapchat Marketin, Source S3, Source Airtable and Source Posthog: fix builds (#27135) 
* Fix SAT tests for confluence, greenhouse, hubspot, stripe

* Fix CAT for close, klaviyo, notion, pinterest and snapchat marketing

* Fix CAT for source s3

* Fix CAT for airtable and posthog

* Bump posthog version
 2023-06-09 03:05:44 +02:00
Augustin
7ca0d2e476 source-s3: delete integration tests using minio (#26908) 2023-06-01 18:03:36 +02:00
Ben Church
1dabc6208e Metadata: add tags field (#26320) 
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-05-26 16:13:09 -07:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158) 
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
 2023-05-24 17:25:41 -07:00
Artem Inzhyyants
93f3286a0d 🚨🚨Source S3: use platform-handled schema evolution (#25127) 
* Source S3: Remove match_target_schema; use platform-handled schema evolution instead

* Source S3: Remove ab_additional_col

* Source S3: update docs; bump version

* Source S3: fix unit tests

* Source S3: fix expected_records

* Source S3: revert _match_target_schema

* Source S3: update expected records for parquet dataset

* Source S3: update metadata

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-15 17:14:26 +02:00
Augustin
8125866e6e source-s3: update connector version in metadata.yaml (#25957) 2023-05-10 08:35:00 -05:00
Artem Inzhyyants
f74d96f9e2 Source S3: support parquet dataset (#25937) 
* Source S3: support parquet dataset

* Source S3: update docs

* Source S3: Fix expected records

* Source S3: Fix expected records

* Source S3: update sem version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-10 11:25:00 +02:00
Baz
9fac7576f6 🐛 CI: fix check_metadata_version_matches_dockerfile_label (#25840) 2023-05-05 13:35:13 +00:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Artem Inzhyyants
64726c7413 Source S3: Parse nested avro schemas (#25361) 
* Source S3: Parse nested avro schemas

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-05-01 22:31:25 +03:00
Artem Inzhyyants
7ce322552e 🐛 Source S3: remove minimum block size (#25706) 
* Source S3: remove minimum block size

* Source S3: update docs

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 15:00:42 +02:00
Serhii Chvaliuk
67f047c703 delete_public_access_block for bucket if public (#25663) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-04-28 21:32:52 +03:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254) 
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-04-18 22:15:11 +02:00
Artem Inzhyyants
e22f9e4cc0 Source S3: handle block size related errors (#25067) 
* Source S3: handle pyarrow block size errors

* Source S3: bump version

* Automated Change

* Source S3: fix null field check

* Revert "Automated Change"

This reverts commit dc707f729d.

* Automated Change

* Source S3: bump version + update docs

* auto-bump connector version

---------

Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-18 16:08:23 +02:00
Artem Inzhyyants
3080f65429 Source S3: Add start date filter for files (#25010) 
* Source S3: Add start date filter for files

* Source S3: add docs

* Source S3: add unittest

* Source S3: add unittest

* Source S3: add unittest

* Source S3: Fix spec test

* Source S3: bump version

* Source S3: fix tests

* Source S3: fix description

* auto-bump connector version

* Source S3: refactor start_date filtering

* Source S3: update setup

* Source S3: serialize state for cache

* Source S3: refactor skip file filter

* Source S3: bump version + update docs

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-18 14:07:15 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Denys Davydov
13ac15130d Source S3: read a single record on check (#24429) 
* #1697 source S3: read a single record on check

* #1697 source s3: upd changelog

* #1697 source s3: fix unit_tests

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-27 12:56:48 +03:00
Denys Davydov
6a88625cca Source s3: fix datetime conversion (#24178) 
* #1669 source s3: fix datetime conversion

* #1669 source s3: review fixes

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-17 20:14:08 +02:00
Denys Davydov
db45f05814 Source S3: fix discovery issues (#24157) 
* #1652 #1664 Source S3: fix discovery issues

* #1652 #1664 source s3: upd changelog

* #1652 #1664 source s3: review comments

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-16 22:39:29 +02:00
Oliver Meyer
5975c323d8 🐛 Source S3: fix datetime format string in FileStream (#23195) 
* Fix datetime format string in FileStream

* Update changelog

* Fix integration tests

* Localize datetime objects

* Bump Dockerfile version

* auto-bump connector version

---------

Co-authored-by: Nataly Merezhuk <65251165+natalyjazzviolin@users.noreply.github.com>
Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-16 11:40:31 -07:00
Denys Davydov
3eecf5408c Source S3: infer schema of the first file only (#23189) 
* #1470 Source S3: infer schema of the first file

* #1470 source s3: upd changelog

* #1470 source s3: review fixes

* #1470 source s3: review fixes

* #1470 source s3: bump version

* #1470 source s3: review fixes

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-14 20:09:15 +02:00
Baz
6a6039bbc5 🐛 Source S3: Make Advanced Reader Options and Advanced Options truly Optional (#23669) 2023-03-03 15:12:49 +02:00
Artem Inzhyyants
f83621ae05 Source S3: fix error handling: raise error on guessing file schema (#23502) 
* Source S3: fix error handling: raise error on guessing file schema

* Source S3: update docs

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-02-27 19:19:52 +01:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084) 
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2023-02-24 16:13:42 -05:00
Denys Davydov
e17464703d Source s3: fix avro discovery (#23198) 
* #23197 source s3: fix avro discovery

* #23197 source s3: upd changelog

* #23197 source s3: add allowed hosts

* #23197 source s3: fix tests

* #23197 - fix build: formatting

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-02-24 12:37:00 +02:00
Denys Davydov
3dc79f5a99 Source S3: speed up discovery (#22500) 
* #1470 source S3: speed up discovery

* #1470 source s3: upd changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-02-09 21:44:48 +02:00
Denys Davydov
fcd3b0334e Source S3: validate CSV read options and convert options (#22550) 
* #1467 source S3: validate CSV read options and convert options

* #1467 source S3: upd changelog

* #1467 source s3: review fixes

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-02-09 18:27:25 +02:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594) 
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
 2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846) 
Rename source acceptance test to connector acceptance test
 2023-02-02 11:38:19 +01:00
Joe Reuter
6e373435f2 Small spec fixes to make sure they work with connector form UI (#21587) 2023-01-25 19:43:26 +01:00
Roman Yermilov [GL]
04a77ad3aa Source S3: keep processing but warn if OSError happen (#21604) 
* Source S3: keep processing but warn if OSError happen

* Source S3: bump version and update changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-01-24 20:00:51 +04:00
midavadim
1c4b6a1f95 added test_strictness_level: high (#21701) 2023-01-24 11:16:40 +02:00
Artem Inzhyyants
31edbd8bae Source S3: update block size for json (#21210) 
* Source S3: update block size for json

* Source S3: update docs

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-01-10 19:53:42 +00:00
Evan Tahler
e39e1898c5 Expected Records to .jsonl format (#20850) 
* Expected Records to `.jsonl` format

* fix formatting template

* remove endline

* update templates

* Update docs/connector-development/testing-connectors/source-acceptance-tests-reference.md

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
 2023-01-03 15:55:36 -08:00
Artem Inzhyyants
09cfcbf599 🐛 Source S3: Check config settings for CSV file format (#20262) 
* Source S3: get master schema on check connection

* Source S3: bump version

* Source S3: update docs

* Source S3: fix test

* Source S3: add fields validation for CSV source

* Source S3: add test

* Source S3: Refactor config validation

* Source S3: update docs

* Source S3: format

* Source S3: format

* Source S3: fix tests

* Source S3: fix tests

* Source S3: fix tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-12-14 21:53:06 +01:00
Xingyuan-Chen
425cc91c85 Source S3: Add virtual-hosted-style option (#19006) 
* add virtual-hosted-style option for S3 source

* update s3 version

* auto-bump connector version

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-08 10:48:16 -05:00
Denys Davydov
6a40ac52fe Source S3: use AirbyteTracedException (#18602) 
* #750 # 837 #904 Source S3: use AirbyteTracedException

* source s3: upd changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-29 07:23:39 +03:00
Denys Davydov
5aa25a1e1a Source S3 - fix schema inference (#17991) 
* #678 oncall. Source S3 - fix schema inference

* source s3: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-14 14:53:39 +03:00
Serhii Lazebnyi
5df66cd572 Source S3: Connector does not enforce SSL/TLS for non-S3 endpoints (#17800) 
* Deleted ssl/tsl flag from config

* Updated PR number

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-12 16:07:22 +02:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585) 
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
 2022-10-11 11:04:23 -04:00
Evan Tahler
49cb3360de Remove redundant title labels from connector specs (#17544) 
* Remove redundant title labels from connector specs

* Manually update specs

* add env variable

* Remove debugging log
 2022-10-05 12:58:38 -07:00
Augustin
ff4ea3961a Republish connectors using CDK 0.1.88 to 0.1.89 (#17304) 2022-09-28 18:18:59 +02:00
Denys Davydov
9054468c21 Source s3: upgrade pyarrow (#16921) 
* #423 oncall source s3: upgrade pyarrow

* source s3: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-20 19:24:07 +03:00
Denys Davydov
4dc394cb9a Source S3: fix reading jsonl files with nested data (#16607) 
* #531 source s3: fix reading nested jsonl files

* #531 source s3: upd changelog

* oncall #531 source s3: fix sample file

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-19 12:09:40 +03:00
Denys Davydov
73ba7b63d5 Source S3: choose between data types when merging master schema (#16631) 
* #422 source s3: choose broadest data type when there is a mismatch during merging json schemas

* #422 source s3: upd changelog
 2022-09-19 10:50:18 +03:00
sivankumar86
3d499557b7 source-S3: Support JSON format (#14213) 
* json format support added

* json format support added

* code formatted

* format convertion changed

* format naming convertion changed

* test cased issue fixed

* test case issued resolved

* sample file and config added for integration tests

* Json doc added

Json doc added

* update

* sample file and config added for integration tests

* sample file and config added for integration tests

* update jsonl files

* review 1

* review 1

* review 1

* pyarrow version upgrade

* clean integration test folder architecture

* add timestamp record to simple_test.jsonl

* fixed integration test and parser review change

* simplify table read

* doc update

* fix specs

* user sample files

* fix sample files

* add newlines at end of files

* rename json parser

* rename jsonfile to jsonlfile

* schema inference added

* patch review fix

* Update docs/integrations/sources/s3.md

doc update

Co-authored-by: George Claireaux <george@airbyte.io>

* changing the version

* changing the title to sync with other type

* fix expected csv records

* fix expected records for avro and parquet

* review fix

* fixed master schema handling

* remove sample configs

* fix expected records

* json doc update

added more details on json parser

* fixed api name

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: George Claireaux <george@airbyte.io>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-01 15:48:23 +01:00