## What
Fixes array column serialization errors in source-iterable streams by
properly defining array item types in JSONSchema definitions.
**Problem:** Array columns (e.g., `campaigns.labels`, `emailListIds`,
`channelIds`, `categories`) were being written as null values to the S3
Data Lake destination with `DESTINATION_SERIALIZATION_ERROR` in the
`_airbyte_meta` column.
**Root Cause:** Multiple stream schemas had array definitions with empty
`items: {}`, which is ambiguous and prevents proper type mapping to
destinations like Iceberg/Glue.
**Affected Streams:**
- `email_unsubscribe` (emailListIds, channelIds)
- `email_send` (categories)
- `email_send_skip` (categories)
- `email_subscribe` (emailListIds)
- `campaigns` (listIds, suppressionListIds, labels)
## How
Updated JSONSchema definitions across affected stream schema files to
specify explicit item types for all array fields:
**Before:**
```
"emailListIds": {
"type": ["null", "array"],
"items": {}
}
```
After:
```
"emailListIds": {
"type": ["null", "array"],
"items": {
"type": "integer"
}
}
```
## Review guide
source_iterable/schemas/campaigns.json - Check listIds,
suppressionListIds, labels
source_iterable/schemas/email_unsubscribe.json - Check emailListIds,
channelIds
source_iterable/schemas/email_send.json - Check categories (in
transactional data)
source_iterable/schemas/email_send_skip.json - Check categories (in
transactional data)
source_iterable/schemas/email_subscribe.json - Check emailListIds
Verify that all "items": {} instances have been replaced with proper
type definitions.
## User Impact
None expected - this is a schema clarification that aligns with actual
data types.
## Can this PR be safely reverted and rolled back?
- [X] YES 💚
- [ ] NO ❌
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Source Iterable: Mark datefields in spec as dates
* Update changelog
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* API key is passed in headers
* updated docs
* removed log
* formatted
* fixed tests
* increased timeout for incremental stream
* bumper source version
* bumper source version in seed
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Wrapper around Prometheus lib to interface with Datadog.
We use prometheus because:
- Future-proofing as it's uses the general open metrics format.
- Prometheus makes it's metrics available to a scraper to it lends itself better to the OSS set up.
- Datadog automatically converts promethues metrics into dd metrics so we don't lose much.
* Requirements updated to CDK.
airbyte-protocol and base-python requirements removed.
* Bugfix: BufferedStreamConsumer. (#3387)
* Format.
* Bump versions.
* main_dev.py renamed to main.py
README.md updated
* Source Stripe: Add Acceptance Tests to Stripe Connector (#3367)
* Add Acceptance Tests to Stripe Connector
* move configured_catalog.json to sample_files
* bump version
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
* Legacy lib references removed
* FB Marketing source - lookback window logic not functioning correctly
* FB Marketing source #1390 - returning buffered record while incremental sync
* FB Marketing source #1390 - improving checking while syncing buffered record
* FB Marketing source #1390 - adding loop_back to IncrementalStreamAPI
* FB Marketing source #1390 - bump version
* FB Marketing source #1390 - add CHANGELOG.md
* Stop formatting python with spotless (#3388)
* add test that migration output schema same as source schema (#3356)
* Add updated architecture diagram to high level docs. (#3399)
* Add updated architecture doc to high level docs.
* Address review comments
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* Correct GA readme error. (#3407)
* make shopify more resilient to timeouts (#3409)
* Update migration schema to include recent changes to the StandardSync object. (#3414)
* Update all of Pydantic to 1.6.2 per Dependabot. (#3408)
* Update all to 1.6.2.
* Publish new airbyte-cdk version.
* Use repr instead of str for exceptions.
* Use rc.
* Edit test.
* Bump for SAT.
* Format.
* Docker ignore update. Fix setup.py
* fixing ONLY problematic fields in freshdesk JSON schemas (#3376)
* bump airbyte-webapp version (#2266)
* add configuration for bumping webapp versionn
* set to current version
* Bump version: 0.16.0-alpha → 0.16.1-alpha
* Revert "Bump version: 0.16.0-alpha → 0.16.1-alpha"
Thiss reverts commit fdbf6dcaa6.
* also update package lock so we don't run into files changed errors
* use 0.19.0-alpha
* add npm webapp version
* Add a CDK speedrun tutorial doc (#3403)
* Add CDK Speedrun document.
* Finish speedrun doc.
* Address review comments
* Add to SUMMARY.md
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* Add Rust as a connector specific dependency to source-file (#3426)
* Add Rust as a connector specific dependency to source-file
* Add more details about installation.
* Markdown lines are weird.
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* API update to latest airbyte-cdk version
* Add section Deploy Local on Windows (#3425)
* add deploy on windows steps
* correct minor
* change suggestions by @avaidyanatha
* GitBook: [master] 161 pages and 75 assets modified
* Display icons (#3140)
* Display icons
* Improve icons views
* MS SQL Server Destination implementation
Fixes issue #613.
Normalization is not yet enabled. This will have to be added at a later point.
* Workflow to handle operations (custom transformation) (#3379)
* Keep normalization backward compatible with old settings from destination
* Bumpversion normalization image
* add npm install before all npm run generates' (#3442)
* restart containers if they fail automatically (#3423)
* Update link for contribution scheduling (#3443)
* Address issue with icon in onboarding (#3437)
* rename toy connector tutorial to "Build a connector the hard way" (#3421)
* Upload test reports (from integration test slash commands) as GitHub artifacts (#3416)
* Archive test reports in github workflow
* Archive Test reports only when failures
* Fixing SqlServerOperations.java (#3454)
Fixing some issues with `SqlServerOperations`, which was out of sync with recent changes to `SqlOperations`.
* Add redirect to cdk tutorial page (#3456)
* add redirect to cdk tutorial page
* change path to cdk README.md
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: vitaliizazmic <75620293+vitaliizazmic@users.noreply.github.com>
Co-authored-by: Charles <giardina.charles@gmail.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: vovavovavovavova <39351371+vovavovavovavova@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcos@airbyte.io>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: masonwheeler <masonwheeler@yahoo.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Michel Tricot <michel@dataline.io>
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.
General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.
Checked in and added tests for the generated modules.
