d9fa24ffff
* checkout from alex/cac * checkout from alex/cac * checkout from alex/cac * Add missing tests * Add missing files * Add missing tests * add missing file * missing file * missing file * rename * doc * doc * remove broken test * rename * jinja dependency * Add comment * comment * comment * pyjq dependency * rename file * delete unused file * Revert "delete unused file" This reverts commit758e939367. * fix * rename * abstract property * delete unused field * delete unused field * rename * pass kwargs directly * isort * Revert "isort" This reverts commit4a79223944. * isort * update state * fix imports * update dependency * format * rename file * decoder * Use decoder * Update comment * dict_state is actually backed by a dict * Add a comment * update state takes kwargs * move state out of offset paginator * update jq parameter order * update * remove incremental mixin * delete comment * update comments * update comments * remove no_state * rename package * checkout from alex/cac * Add missing tests * Add missing files * missing file * rename * jinja dependency * Add comment * comment * comment * Revert "delete unused file" This reverts commit758e939367. * delete unused field * delete unused field * rename * pass kwargs directly * isort * Revert "isort" This reverts commit4a79223944. * format * decoder * better error handling * remove nostate * isort * remove print * move test * delete duplicates * delete dead code * Update mapping type to [str, Any] * add comment * Add comment * pass parameters through kwargs * pass parameters through kwargs * update interface to pass source in interface * update interface to pass source in interface * rename to stream_slicer * Allow passing a string or an enum * Define StateType enum * convert state_type if not of type type * convert state_type if not of type type * Low code connectors: string interpolation with jinja (#12852) * checkout from alex/cac * Add missing tests * Add missing files * missing file * rename * jinja dependency * Add comment * comment * comment * Revert "delete unused file" This reverts commit758e939367. * delete unused field * delete unused field * rename * pass kwargs directly * isort * Revert "isort" This reverts commit4a79223944. * format * decoder * better error handling * remove nostate * isort * delete dead code * Update mapping type to [str, Any] * add comment * Add comment * pass parameters through kwargs * move test to right module * Add missing test * Use authbase instead of deprecated class * leverage generator * Delete dead code * rename methods * rename to declarative * rename the classes too * Try to install packages to build jq * isort * only automake * Revert "only automake" This reverts commitc8fe154ffc. * remove git * format * Add jq dependency * Use request header provider * rename * rename field * remove get_context method * rename
Introduction
Data integration made simple, secure and extensible. The new open-source standard to sync data from applications, APIs & databases to warehouses, lakes & other destinations.
Airbyte is on a mission to make data integration pipelines a commodity.
- Maintenance-free connectors you can use in minutes. Just authenticate your sources and warehouse, and get connectors that adapt to schema and API changes for you.
- Building new connectors made trivial. We make it very easy to add new connectors that you need, using the language of your choice, by offering scheduling and orchestration.
- Designed to cover the long tail of connectors and needs. Benefit from the community's battle-tested connectors and adapt them to your specific needs.
- Your data stays in your cloud. Have full control over your data, and the costs of your data transfers.
- No more security compliance process to go through as Airbyte is self-hosted.
- No more pricing indexed on volume, as cloud-based solutions offer.
Here's a list of our connectors with their health status.
Quick start
git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker-compose up
Now visit http://localhost:8000
Here is a step-by-step guide showing you how to load data from an API into a file, all on your computer.
Features
- Built for extensibility: Adapt an existing connector to your needs or build a new one with ease.
- Optional normalized schemas: Entirely customizable, start with raw data or from some suggestion of normalized data.
- Full-grade scheduler: Automate your replications with the frequency you need.
- Real-time monitoring: We log all errors in full detail to help you understand.
- Incremental updates: Automated replications are based on incremental updates to reduce your data transfer costs.
- Manual full refresh: Sometimes, you need to re-sync all your data to start again.
- Debugging autonomy: Modify and debug pipelines as you see fit, without waiting.
Contributing
We love contributions to Airbyte, big or small.
See our Contributing guide on how to get started. Not sure where to start? We’ve listed some good first issues to start with. If you have any questions, please open a draft PR or visit our slack channel where the core team can help answer your questions.
Note that you are able to create connectors using the language you want, as Airbyte connections run as Docker containers.
Also, we will never ask you to maintain your connector. The goal is that the Airbyte team and the community helps maintain it, let's call it crowdsourced maintenance!
Community support
For general help using Airbyte, please refer to the official Airbyte documentation. For additional help, you can use one of these channels to ask a question:
- Slack (For live discussion with the Community and Airbyte team)
- Forum (For deeper converstaions about features, connectors, or problems)
- GitHub (Bug reports, Contributions)
- Twitter (Get the news fast)
- Weekly office hours (Live informal 30-minute video call sessions with the Airbyte team)
Roadmap
Check out our roadmap to get informed on what we are currently working on, and what we have in mind for the next weeks, months and years.
License
See the LICENSE file for licensing information, and our FAQ for any questions you may have on that topic.