5.5 KiB
Amazon SQS
Overview
The Amazon SQS source syncs the SQS API, refer: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html. It supports full refresh sync and could be viewed in builder with low-code support.
Getting started
Requirements
- AWS IAM Access Key
- AWS IAM Secret Key
- AWS SQS Queue URL
- AWS Region
- Action target
Setup guide
Supported Streams
This Source is capable of syncing the following core Action that would be received as streams for sync:
Other Actions are in beta which might require more/less parameters
Features
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental Sync | No | |
| Namespaces | No |
Performance considerations
The rate of lookup requests for RecieveMessage stream is limited to two per second, per account, per region. This connector gracefully retries when encountering a throttling error. However if the errors continue repeatedly after multiple retries (for example if you setup many instances of this connector using the same account and region), the connector sync will fail.
Output schema
This source will output one stream for the configured SQS Queue. The stream record data will have three fields:
- id (a UUIDv4 as a STRING)
- body (message body as a STRING)
- attributes (attributes of the messages as an OBJECT or NULL)
Properties
Required properties are 'Queue URL', 'AWS Region' and 'Delete Messages After Read' as noted in bold below.
- Queue URL (STRING)
- The full AWS endpoint URL of the queue e.g.
https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue-url
- The full AWS endpoint URL of the queue e.g.
- AWS Region (STRING)
- The region code for the SQS Queue e.g. eu-west-1
- Max Batch Size (INTEGER)
- The max amount of messages to consume in a single poll e.g. 5
- Minimum of 1, maximum of 10
- Default: 10
- Max Wait Time (INTEGER)
- The max amount of time (in seconds) to poll for messages before commiting a batch (or timing
out) unless we fill a batch (as per
Max Batch Size) - Minimum of 1, maximum of 20
- Default: 20
- The max amount of time (in seconds) to poll for messages before commiting a batch (or timing
out) unless we fill a batch (as per
- Message Attributes To Return (STRING)
- A comma separated list of Attributes to return for each message
- Default: All
- Message Visibility Timeout (INTEGER)
- After a message is read, how much time (in seconds) should the message be hidden from other consumers
- After this timeout, the message is not deleted and can be re-read
- Default: 20
- AWS IAM Access Key ID (STRING)
- The Access Key for the IAM User with permissions on this Queue
- AWS IAM Secret Key (STRING)
- The Secret Key for the IAM User with permissions on this Queue
- Target (STRING)
- The targeted action resource for the fetch
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 1.0.15 | 2025-05-10 | 59775 | Update dependencies |
| 1.0.14 | 2025-05-03 | 59344 | Update dependencies |
| 1.0.13 | 2025-04-26 | 58684 | Update dependencies |
| 1.0.12 | 2025-04-19 | 58293 | Update dependencies |
| 1.0.11 | 2025-04-12 | 57609 | Update dependencies |
| 1.0.10 | 2025-04-05 | 57146 | Update dependencies |
| 1.0.9 | 2025-03-29 | 56614 | Update dependencies |
| 1.0.8 | 2025-03-22 | 56083 | Update dependencies |
| 1.0.7 | 2025-03-08 | 54832 | Update dependencies |
| 1.0.6 | 2025-03-01 | 54738 | fix: Update source-amazon-sqs to use nltk 3.9.1 or higher |
| 1.0.5 | 2025-02-22 | 54271 | Update dependencies |
| 1.0.4 | 2025-02-15 | 53936 | Update dependencies |
| 1.0.3 | 2025-02-01 | 52878 | Update dependencies |
| 1.0.2 | 2025-01-25 | 52163 | Update dependencies |
| 1.0.1 | 2025-01-18 | 51742 | Update dependencies |
| 1.0.0 | 2024-11-07 | 41064 | Migrate to low code |
| 0.1.1 | 2024-01-03 | #33924 | Add new ap-southeast-3 AWS region |
| 0.1.0 | 2021-10-10 | #0000 | Initial version |