mirror of synced 2026-01-07 09:05:45 -05:00

Files

Yevhenii 81d66a6388 Source Salesforce: Handling 400 error when using BULK API (#8206 )

* Source Salesforce: Handling 400 error when using BULK API

2021-11-24 15:42:10 +02:00

16 KiB

Raw Blame History

Salesforce

Overview

The Salesforce source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run.

The Connector supports Custom Fields for each of their available streams

Output schema

Several output streams are available from this source. A list of these streams can be found below in the Streams section.

Features

Feature	Supported?
Full Refresh Sync	Yes
Incremental Sync	Yes
SSL connection	Yes
Namespaces	No

Performance considerations

The connector is restricted by normal Salesforce rate limiting. For large transfers we recommend using the BULK API.

Getting started

Requirements

Salesforce Account
Salesforce OAuth credentials

Setup guide

We recommend the following walkthrough while keeping in mind the edits we suggest below for setting up a Salesforce app that can pull data from Salesforce and locating the credentials you need to provide to Airbyte.

Suggested edits:

If your salesforce URL does not take the form X.salesforce.com, use your actual Salesforce domain name. For example, if your Salesforce URL is awesomecompany.force.com then use that instead of awesomecompany.salesforce.com.
When running a curl command, always run it with the -L option to follow any redirects.

is_sandbox

If you log in using at https://login.salesforce.com, then the value is false. If you log in at https://test.salesforce.com then the value should be true. If this is Greek to you, then this value should probably be false.

Streams

Note: The connector supports reading not only standard streams (listed below), but also reading Custom Objects.

List of available streams:

AIApplication
AIApplicationConfig
AIInsightAction
AIInsightFeedback
AIInsightReason
AIInsightValue
AIRecordInsight
AcceptedEventRelation
Account
AccountCleanInfo
AccountContactRole
AccountFeed
AccountHistory
AccountPartner
AccountShare
ActionLinkGroupTemplate
ActionLinkTemplate
ActiveFeatureLicenseMetric
ActivePermSetLicenseMetric
ActiveProfileMetric
ActiveScratchOrg
ActiveScratchOrgFeed
ActiveScratchOrgHistory
ActiveScratchOrgShare
AdditionalNumber
AlternativePaymentMethod
AlternativePaymentMethodShare
ApexClass
ApexComponent
ApexEmailNotification
ApexLog
ApexPage
ApexPageInfo
ApexTestQueueItem
ApexTestResult
ApexTestResultLimits
ApexTestRunResult
ApexTestSuite
ApexTrigger
ApiAnomalyEventStore
ApiAnomalyEventStoreFeed
ApiEvent
AppAnalyticsQueryRequest
AppDefinition
AppMenuItem
AppUsageAssignment
AppointmentSchedulingPolicy
AppointmentTopicTimeSlot
AppointmentTopicTimeSlotFeed
AppointmentTopicTimeSlotHistory
Asset
AssetAction
AssetActionSource
AssetFeed
AssetHistory
AssetRelationship
AssetRelationshipFeed
AssetRelationshipHistory
AssetShare
AssetStatePeriod
AssignedResource
AssignedResourceFeed
AssignmentRule
AssociatedLocation
AssociatedLocationHistory
AsyncApexJob
Attachment
AuraDefinition
AuraDefinitionBundle
AuraDefinitionBundleInfo
AuraDefinitionInfo
AuthConfig
AuthConfigProviders
AuthProvider
AuthSession
AuthorizationForm
AuthorizationFormConsent
AuthorizationFormConsentHistory
AuthorizationFormConsentShare
AuthorizationFormDataUse
AuthorizationFormDataUseHistory
AuthorizationFormDataUseShare
AuthorizationFormHistory
AuthorizationFormShare
AuthorizationFormText
AuthorizationFormTextFeed
AuthorizationFormTextHistory
BackgroundOperation
BrandTemplate
BrandingSet
BrandingSetProperty
BulkApiResultEventStore
BusinessHours
BusinessProcess
Calendar
CalendarView
CalendarViewShare
CallCenter
CallCoachingMediaProvider
Campaign
CampaignFeed
CampaignHistory
CampaignMember
CampaignMemberStatus
CampaignShare
CardPaymentMethod
Case
CaseComment
CaseContactRole
CaseFeed
CaseHistory
CaseShare
CaseSolution
CaseStatus
CaseTeamMember
CaseTeamRole
CaseTeamTemplate
CaseTeamTemplateMember
CaseTeamTemplateRecord
CategoryData
CategoryNode
ChatterActivity
ChatterExtension
ChatterExtensionConfig
ClientBrowser
CollaborationGroup
CollaborationGroupFeed
CollaborationGroupMember
CollaborationGroupMemberRequest
CollaborationInvitation
CommSubscription
CommSubscriptionChannelType
CommSubscriptionChannelTypeFeed
CommSubscriptionChannelTypeHistory
CommSubscriptionChannelTypeShare
CommSubscriptionConsent
CommSubscriptionConsentFeed
CommSubscriptionConsentHistory
CommSubscriptionConsentShare
CommSubscriptionFeed
CommSubscriptionHistory
CommSubscriptionShare
CommSubscriptionTiming
CommSubscriptionTimingFeed
CommSubscriptionTimingHistory
Community
ConferenceNumber
ConnectedApplication
ConsumptionRate
ConsumptionRateHistory
ConsumptionSchedule
ConsumptionScheduleFeed
ConsumptionScheduleHistory
ConsumptionScheduleShare
Contact
ContactCleanInfo
ContactFeed
ContactHistory
ContactPointAddress
ContactPointAddressHistory
ContactPointAddressShare
ContactPointConsent
ContactPointConsentHistory
ContactPointConsentShare
ContactPointEmail
ContactPointEmailHistory
ContactPointEmailShare
ContactPointPhone
ContactPointPhoneHistory
ContactPointPhoneShare
ContactPointTypeConsent
ContactPointTypeConsentHistory
ContactPointTypeConsentShare
ContactRequest
ContactRequestShare
ContactShare
ContentAsset
ContentDistribution
ContentDistributionView
ContentDocument
ContentDocumentFeed
ContentDocumentHistory
ContentDocumentSubscription
ContentFolder
ContentFolderLink
ContentNotification
ContentTagSubscription
ContentUserSubscription
ContentVersion
ContentVersionComment
ContentVersionHistory
ContentVersionRating
ContentWorkspace
ContentWorkspaceDoc
ContentWorkspaceMember
ContentWorkspacePermission
ContentWorkspaceSubscription
Contract
ContractContactRole
ContractFeed
ContractHistory
ContractStatus
ConversationEntry
CorsWhitelistEntry
CredentialStuffingEventStore
CredentialStuffingEventStoreFeed
CreditMemo
CreditMemoFeed
CreditMemoHistory
CreditMemoLine
CreditMemoLineFeed
CreditMemoLineHistory
CreditMemoShare
CronJobDetail
CronTrigger
CspTrustedSite
CustomBrand
CustomBrandAsset
CustomHelpMenuItem
CustomHelpMenuSection
CustomHttpHeader
CustomNotificationType
CustomObjectUserLicenseMetrics
CustomPermission
CustomPermissionDependency
DandBCompany
Dashboard
DashboardComponent
DashboardComponentFeed
DashboardFeed
DataAssessmentFieldMetric
DataAssessmentMetric
DataAssessmentValueMetric
DataAssetSemanticGraphEdge
DataAssetUsageTrackingInfo
DataUseLegalBasis
DataUseLegalBasisHistory
DataUseLegalBasisShare
DataUsePurpose
DataUsePurposeHistory
DataUsePurposeShare
DatacloudCompany
DatacloudContact
DatacloudOwnedEntity
DatacloudPurchaseUsage
DeclinedEventRelation
DeleteEvent
DigitalWallet
Document
DocumentAttachmentMap
Domain
DomainSite
DuplicateRecordItem
DuplicateRecordSet
DuplicateRule
EmailCapture
EmailDomainFilter
EmailDomainKey
EmailMessage
EmailMessageRelation
EmailRelay
EmailServicesAddress
EmailServicesFunction
EmailTemplate
EmbeddedServiceDetail
EmbeddedServiceLabel
EngagementChannelType
EngagementChannelTypeFeed
EngagementChannelTypeHistory
EngagementChannelTypeShare
EnhancedLetterhead
EnhancedLetterheadFeed
EntityDefinition
EntitySubscription
Event
EventBusSubscriber
EventFeed
EventLogFile
EventRelation
ExpressionFilter
ExpressionFilterCriteria
ExternalDataSource
ExternalDataUserAuth
ExternalEvent
ExternalEventMapping
ExternalEventMappingShare
FeedAttachment
FeedComment
FeedItem
FeedPollChoice
FeedPollVote
FeedRevision
FieldPermissions
FieldSecurityClassification
FileSearchActivity
FinanceBalanceSnapshot
FinanceBalanceSnapshotShare
FinanceTransaction
FinanceTransactionShare
FiscalYearSettings
FlowDefinitionView
FlowInterview
FlowInterviewLog
FlowInterviewLogEntry
FlowInterviewLogShare
FlowInterviewShare
FlowRecordRelation
FlowStageRelation
Folder
FormulaFunction
FormulaFunctionAllowedType
FormulaFunctionCategory
GrantedByLicense
Group
GroupMember
GtwyProvPaymentMethodType
Holiday
IPAddressRange
Idea
IdentityProviderEventStore
IdentityVerificationEvent
IdpEventLog
IframeWhiteListUrl
Image
ImageFeed
ImageHistory
ImageShare
Individual
IndividualHistory
IndividualShare
InstalledMobileApp
Invoice
InvoiceFeed
InvoiceHistory
InvoiceLine
InvoiceLineFeed
InvoiceLineHistory
InvoiceShare
KnowledgeableUser
Lead
LeadCleanInfo
LeadFeed
LeadHistory
LeadShare
LeadStatus
LegalEntity
LegalEntityFeed
LegalEntityHistory
LegalEntityShare
LightningExitByPageMetrics
LightningExperienceTheme
LightningOnboardingConfig
LightningToggleMetrics
LightningUriEvent
LightningUsageByAppTypeMetrics
LightningUsageByBrowserMetrics
LightningUsageByFlexiPageMetrics
LightningUsageByPageMetrics
ListEmail
ListEmailIndividualRecipient
ListEmailRecipientSource
ListEmailShare
ListView
ListViewChart
ListViewEvent
LiveChatSensitiveDataRule
Location
LocationFeed
LocationHistory
LocationShare
LoginAsEvent
LoginEvent
LoginGeo
LoginHistory
LoginIp
LogoutEvent
MLField
MLPredictionDefinition
Macro
MacroHistory
MacroInstruction
MacroShare
MacroUsage
MacroUsageShare
MailmergeTemplate
MatchingInformation
MatchingRule
MatchingRuleItem
MessagingChannel
MessagingChannelSkill
MessagingConfiguration
MessagingDeliveryError
MessagingEndUser
MessagingEndUserHistory
MessagingEndUserShare
MessagingLink
MessagingSession
MessagingSessionFeed
MessagingSessionHistory
MessagingSessionShare
MessagingTemplate
MetadataPackage
MetadataPackageVersion
MobileApplicationDetail
MsgChannelLanguageKeyword
MutingPermissionSet
MyDomainDiscoverableLogin
NamedCredential
NamespaceRegistry
NamespaceRegistryFeed
NamespaceRegistryHistory
Note
OauthCustomScope
OauthCustomScopeApp
OauthToken
ObjectPermissions
OnboardingMetrics
OperatingHours
OperatingHoursFeed
Opportunity
OpportunityCompetitor
OpportunityContactRole
OpportunityFeed
OpportunityFieldHistory
OpportunityHistory
OpportunityLineItem
OpportunityPartner
OpportunityShare
OpportunityStage
Order
OrderFeed
OrderHistory
OrderItem
OrderItemFeed
OrderItemHistory
OrderShare
OrderStatus
OrgDeleteRequest
OrgDeleteRequestShare
OrgMetric
OrgMetricScanResult
OrgMetricScanSummary
OrgWideEmailAddress
Organization
PackageLicense
PackagePushError
PackagePushJob
PackagePushRequest
PackageSubscriber
Partner
PartnerRole
PartyConsent
PartyConsentFeed
PartyConsentHistory
PartyConsentShare
Payment
PaymentAuthAdjustment
PaymentAuthorization
PaymentGateway
PaymentGatewayLog
PaymentGatewayProvider
PaymentGroup
PaymentLineInvoice
PaymentMethod
Period
PermissionSet
PermissionSetAssignment
PermissionSetGroup
PermissionSetGroupComponent
PermissionSetLicense
PermissionSetLicenseAssign
PermissionSetTabSetting
PlatformCachePartition
PlatformCachePartitionType
PlatformEventUsageMetric
Pricebook2
Pricebook2History
PricebookEntry
PricebookEntryHistory
ProcessDefinition
ProcessException
ProcessExceptionShare
ProcessInstance
ProcessInstanceNode
ProcessInstanceStep
ProcessInstanceWorkitem
ProcessNode
Product2
Product2Feed
Product2History
ProductConsumptionSchedule
Profile
Prompt
PromptAction
PromptActionShare
PromptError
PromptErrorShare
PromptVersion
Publisher
PushTopic
QueueSobject
QuickText
QuickTextHistory
QuickTextShare
QuickTextUsage
QuickTextUsageShare
RecentlyViewed
Recommendation
RecordAction
RecordActionHistory
RecordType
RedirectWhitelistUrl
Refund
RefundLinePayment
Report
ReportAnomalyEventStore
ReportAnomalyEventStoreFeed
ReportEvent
ReportFeed
ResourceAbsence
ResourceAbsenceFeed
ResourceAbsenceHistory
ResourcePreference
ResourcePreferenceFeed
ResourcePreferenceHistory
ReturnOrder
ReturnOrderFeed
ReturnOrderHistory
ReturnOrderItemAdjustment
ReturnOrderItemTax
ReturnOrderLineItem
ReturnOrderLineItemFeed
ReturnOrderLineItemHistory
ReturnOrderShare
SPSamlAttributes
SamlSsoConfig
Scontrol
ScratchOrgInfo
ScratchOrgInfoFeed
ScratchOrgInfoHistory
ScratchOrgInfoShare
SearchPromotionRule
SecureAgent
SecureAgentPlugin
SecureAgentPluginProperty
SecureAgentsCluster
SecurityCustomBaseline
ServiceAppointment
ServiceAppointmentFeed
ServiceAppointmentHistory
ServiceAppointmentShare
ServiceAppointmentStatus
ServiceResource
ServiceResourceFeed
ServiceResourceHistory
ServiceResourceShare
ServiceResourceSkill
ServiceResourceSkillFeed
ServiceResourceSkillHistory
ServiceSetupProvisioning
ServiceTerritory
ServiceTerritoryFeed
ServiceTerritoryHistory
ServiceTerritoryMember
ServiceTerritoryMemberFeed
ServiceTerritoryMemberHistory
ServiceTerritoryShare
ServiceTerritoryWorkType
ServiceTerritoryWorkTypeFeed
ServiceTerritoryWorkTypeHistory
SessionHijackingEventStore
SessionHijackingEventStoreFeed
SessionPermSetActivation
SetupAssistantStep
SetupAuditTrail
SetupEntityAccess
Site
SiteFeed
SiteHistory
SiteIframeWhiteListUrl
SiteRedirectMapping
Skill
SkillRequirement
SkillRequirementFeed
SkillRequirementHistory
Solution
SolutionFeed
SolutionHistory
SolutionStatus
Stamp
StampAssignment
StaticResource
StreamingChannel
StreamingChannelShare
TabDefinition
Task
TaskFeed
TaskPriority
TaskStatus
TenantUsageEntitlement
TestSuiteMembership
ThirdPartyAccountLink
ThreatDetectionFeedback
ThreatDetectionFeedbackFeed
TimeSlot
TodayGoal
TodayGoalShare
Topic
TopicAssignment
TopicFeed
TopicUserEvent
TransactionSecurityPolicy
Translation
UiFormulaCriterion
UiFormulaRule
UndecidedEventRelation
UriEvent
User
UserAppInfo
UserAppMenuCustomization
UserAppMenuCustomizationShare
UserAppMenuItem
UserEmailPreferredPerson
UserEmailPreferredPersonShare
UserFeed
UserLicense
UserListView
UserListViewCriterion
UserLogin
UserPackageLicense
UserPermissionAccess
UserPreference
UserProvAccount
UserProvAccountStaging
UserProvMockTarget
UserProvisioningConfig
UserProvisioningLog
UserProvisioningRequest
UserProvisioningRequestShare
UserRole
UserSetupEntityAccess
UserShare
VerificationHistory
VisualforceAccessMetrics
WaveAutoInstallRequest
WaveCompatibilityCheckItem
WebLink
WorkType
WorkTypeFeed
WorkTypeGroup
WorkTypeGroupFeed
WorkTypeGroupHistory
WorkTypeGroupMember
WorkTypeGroupMemberFeed
WorkTypeGroupMemberHistory
WorkTypeGroupShare
WorkTypeHistory
WorkTypeShare

Note: Using the BULK API is not possible to receive data from the following streams:

AcceptedEventRelation
AssetTokenEvent
AttachedContentNote
Attachment
CaseStatus
ContractStatus
DeclinedEventRelation
EventWhoRelation
FieldSecurityClassification
OrderStatus
PartnerRole
QuoteTemplateRichTextData
RecentlyViewed
ServiceAppointmentStatus
SolutionStatus
TaskPriority
TaskStatus
TaskWhoRelation
UndecidedEventRelation

Changelog

Version	Date	Pull Request	Subject

| 0.1.7 | 2021-11-24 | 8206 | Handling 400 error when trying to create a job for sync using Bulk API. | | 0.1.6 | 2021-11-16 | 8009 | Fix retring of BULK jobs | | 0.1.5 | 2021-11-15 | 7885 | Add Transform for output records | | 0.1.4 | 2021-11-09 | 7778 | Fix types for anyType fields | | 0.1.3 | 2021-11-06 | 7592 | Fix getting anyType fields using BULK API | | 0.1.2 | 2021-09-30 | 6438 | Annotate Oauth2 flow initialization parameters in connector specification | | 0.1.1 | 2021-09-21 | 6209 | Fix bug with pagination for BULK API | | 0.1.0 | 2021-09-08 | 5619 | Salesforce Aitbyte-Native Connector |

16 KiB Raw Blame History