yarden/source-mongodb-support-dynamic-arrays
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Anatoli Makarevich
|
eba7941311 |
fix: fixed broken Outbrain connector pagination (#67079)
## What <!-- * Describe what the change is solving. Link all GitHub issues related to this change. --> I recently noticed that our Outbrain connector doesn't return all campaigns and promoted links for some marketers (Outbrain ad accounts). After quick debugging and building a script to download the data with a fresh API token I figured that the problem is in Airbyte connector. This PR fixes broken pagination for the Outbrain connector. Issue: https://github.com/airbytehq/airbyte/issues/67080 ## How <!-- * Describe how code changes achieve the solution. --> By adjusting the limit/offset logic in the connector, it was clearly broken. ## Review guide <!-- 1. `x.py` 2. `y.py` --> Ideally you want to download marketers/campaigns/promoted links and promoted links performance to make sure. I did this by running: ```bash python main.py read --config secrets/config.json --catalog secrets/catalog.json ``` with an actual access token and the following `secrets/catalog.json` file: ```json { "streams": [ { "stream": { "name": "marketers", "json_schema": {}, "supported_sync_modes": ["full_refresh"] }, "sync_mode": "full_refresh", "destination_sync_mode": "append_dedup" }, { "stream": { "name": "campaigns", "json_schema": { "$schema": "http://json-schema.org/schema#", "type": "object", "additionalProperties": true, "properties": { "id": { "type": ["string", "null"] }, "name": { "type": ["string", "null"] }, "enabled": { "type": ["boolean", "null"] }, "creationTime": { "type": ["string", "null"] }, "lastModified": { "type": ["string", "null"] }, "cpc": { "type": ["number", "null"] }, "autoArchived": { "type": ["boolean", "null"] }, "minimumCpc": { "type": ["number", "null"] }, "currency": { "type": ["string", "null"] }, "targeting": { "type": "object", "properties": { "platform": { "type": "array", "items": { "type": ["string", "null"] } }, "language": { "type": ["string", "null"] }, "excludeAdBlockUsers": { "type": ["boolean", "null"] }, "nativePlacements": { "type": "object", "properties": { "enabled": { "type": ["boolean", "null"] } } }, "includeCellularNetwork": { "type": ["boolean", "null"] }, "nativePlacementsEnabled": { "type": ["boolean", "null"] }, "locationsVersion": { "type": ["string", "null"] } } }, "marketerId": { "type": ["string", "null"] }, "autoExpirationOfAds": { "type": ["integer", "null"] }, "contentType": { "type": ["string", "null"] }, "budget": { "type": "object", "properties": { "id": { "type": ["string", "null"] }, "name": { "type": ["string", "null"] }, "shared": { "type": ["boolean", "null"] }, "amount": { "type": ["number", "null"] }, "currency": { "type": ["string", "null"] }, "creationTime": { "type": ["string", "null"] }, "lastModified": { "type": ["string", "null"] }, "startDate": { "type": ["string", "null"] }, "endDate": { "type": ["string", "null"] }, "runForever": { "type": ["boolean", "null"] }, "type": { "type": ["string", "null"] }, "pacing": { "type": ["string", "null"] } } }, "prefixTrackingCode": { "type": "object", "properties": { "prefix": { "type": ["string", "null"] }, "encode": { "type": ["boolean", "null"] } } }, "liveStatus": { "type": "object", "properties": { "onAirReason": { "type": ["string", "null"] }, "campaignOnAir": { "type": ["boolean", "null"] }, "amountSpent": { "type": ["number", "null"] }, "onAirModificationTime": { "type": ["string", "null"] } } }, "readonly": { "type": ["boolean", "null"] }, "startHour": { "type": ["string", "null"] }, "trackingPixels": { "type": "object", "properties": { "enabled": { "type": ["boolean", "null"] }, "urls": { "type": "array", "items": { "type": ["string", "null"] } } } }, "pixels": { "type": "object", "properties": { "impressionPixels": { "type": "array", "items": { "type": ["string", "null"] } }, "trackingPixels": { "type": "array", "items": { "type": ["string", "null"] } } } }, "onAirType": { "type": ["string", "null"] }, "objective": { "type": ["string", "null"] }, "creativeFormat": { "type": ["string", "null"] }, "dynamicRetargeting": { "type": ["boolean", "null"] } } }, "supported_sync_modes": ["full_refresh"] }, "sync_mode": "full_refresh", "destination_sync_mode": "append_dedup" }, { "stream": { "name": "promoted_links", "json_schema": { "$schema": "http://json-schema.org/schema#", "type": "object", "additionalProperties": true, "properties": { "id": { "type": ["string", "null"] }, "text": { "type": ["string", "null"] }, "creationTime": { "type": ["string", "null"] }, "lastModified": { "type": ["string", "null"] }, "url": { "type": ["string", "null"] }, "siteName": { "type": ["string", "null"] }, "sectionName": { "type": ["string", "null"] }, "status": { "type": ["string", "null"] }, "enabled": { "type": ["boolean", "null"] }, "campaignId": { "type": ["string", "null"] }, "archived": { "type": ["boolean", "null"] }, "documentLanguage": { "type": ["string", "null"] }, "onAirStatus": { "type": "object", "properties": { "onAir": { "type": ["boolean", "null"] }, "reason": { "type": ["string", "null"] } } }, "baseUrl": { "type": ["string", "null"] }, "documentId": { "type": ["string", "null"] }, "metaData": { "type": ["string", "null"] }, "approvalStatus": { "type": "object", "properties": { "status": { "type": ["string", "null"] }, "isEditable": { "type": ["boolean", "null"] } } }, "description": { "type": ["string", "null"] }, "callToAction": { "type": "object", "properties": { "type": { "type": ["string", "null"] }, "value": { "type": ["string", "null"] } } }, "imageType": { "type": ["string", "null"] }, "language": { "type": ["string", "null"] } } }, "supported_sync_modes": ["full_refresh"] }, "sync_mode": "full_refresh", "destination_sync_mode": "append_dedup" }, { "stream": { "name": "performance_promoted_links", "json_schema": { "$schema": "http://json-schema.org/schema#", "type": "object", "properties": { "metadata": { "type": "object", "properties": { "id": { "type": ["string", "null"] }, "fromDate": { "type": ["string", "null"] }, "toDate": { "type": ["string", "null"] } } }, "metrics": { "type": "object", "properties": { "impressions": { "type": ["number", "null"] }, "clicks": { "type": ["number", "null"] }, "totalConversions": { "type": ["number", "null"] }, "conversions": { "type": ["number", "null"] }, "viewConversions": { "type": ["number", "null"] }, "spend": { "type": ["number", "null"] }, "ecpc": { "type": ["number", "null"] }, "ctr": { "type": ["number", "null"] }, "conversionRate": { "type": ["number", "null"] }, "viewConversionRate": { "type": ["number", "null"] }, "cpa": { "type": ["number", "null"] }, "totalCpa": { "type": ["number", "null"] }, "totalSumValue": { "type": ["number", "null"] }, "sumValue": { "type": ["number", "null"] }, "viewSumValue": { "type": ["number", "null"] }, "totalAverageValue": { "type": ["number", "null"] }, "averageValue": { "type": ["number", "null"] }, "viewAverageValue": { "type": ["number", "null"] }, "totalRoas": { "type": ["number", "null"] }, "roas": { "type": ["number", "null"] }, "videoReachedFirstQ": { "type": ["integer", "null"] }, "videoReachedSecondQ": { "type": ["integer", "null"] }, "videoReachedThirdQ": { "type": ["integer", "null"] }, "videoReachedCompletion": { "type": ["integer", "null"] }, "videoViewDuration": { "type": ["integer", "null"] }, "videoAvgViewDuration": { "type": ["integer", "null"] }, "videoPlays": { "type": ["integer", "null"] } } }, "marketer_id": { "type": ["string", "null"] }, "campaign_id": { "type": ["string", "null"] }, "promoted_link_id": { "type": ["string", "null"] } } }, "supported_sync_modes": ["full_refresh"] }, "sync_mode": "full_refresh", "destination_sync_mode": "append_dedup" } ] } ``` I see that we got all campaigns, promoted links and daily performance records (we had 65 campaigns before, limited by the broken pagination): <img width="2314" height="320" alt="Screenshot 2025-10-06 at 12 43 32" src="https://github.com/user-attachments/assets/3b12234c-4fe4-40c0-99d9-bf71ad34be79" /> ## User Impact <!-- * What is the end result perceived by the user? * If there are negative side effects, please list them. --> 💀 💣 Every user with at least 50 campaigns won't get the full data from Airbyte connector. ## Can this PR be safely reverted and rolled back? <!-- * If unsure, leave it blank. --> - [x] YES 💚 - [ ] NO ❌ --------- Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com> |
||
|
Aaron ("AJ") Steers
|
83ecbe0fc3 | CI: apply pre-commit format fix from #49806 (#49852) | ||
|
Marius Posta
|
7ae97175a6 | gradle: fix repo wide behaviour (#30607) | ||
|
Zawar Khan
|
4abf6054c2 |
🎉 New Source: Outbrain-Amplify (#27008)
Co-authored-by: Sajarin <sajarindider@gmail.com> |