* Making single port 8000 default and update NetworkErrorBoundary.tsx response code 502
* add .env files
* Updating instructions in docs/deploying-airbyte/*
* Migrate BufferedStreamConsumer users (e.g. all JDBC destinations, MeiliSearch) (#3473)
* Add checkpointing test cases in Acceptance Tests (#3473)
* Add testing for emitting state in Destination Standard Test (#3546)
* Migrate BQ to support checkpointing (#3546)
* Migrate copy destinations support checkpointing (#3547)
* Checkpointing: Migrate CSV and JSON destinations (#3551)
Fixes issue #2122
Added icon data
Added icon filenames to connector description yaml files
Added plumbing to transfer and load data properly
Updated tests
This PR is step 5 of this tech spec - https://docs.google.com/document/d/1qFk4YqnwxE4MCGeJ9M2scGOYej6JnDy9A0zbICP_zjI/edit.
The first of (at least) 2 PRs to implement this on the source side. I made some headway before deciding to break the changes into one PR implementing this for discover schema job, and another PR implementing this for read. The combined PR would have been too big otherwise.
Also refactor MoreResources as the test method was attempting to write to the location classes where loaded out from - the issue is we cannot guarantee where the class is loaded from can be written to. Changing this to write to a random folder in the temp directory.
* Handle destination sync mode in destinations
* Source & Destination sync modes are required (#2500)
* Provide Migration script making sure it is always defined for previous sync configs
Add the latestSyncJobStatus field to the WbConnectionRead struct. Take the chance to rename the lastSync field to latestSyncJobCreatedAt to better reflect what the field captures. lastSync was confusing as I expect it to return the Id/object of the last sync and not the last sync's timestamp.
We keep the current expectation when setting status - the same job that sets the timestamp field is used to set the status field.
We surface the job's status as is. This gives the UI flexibility on displaying the various statuses.
Since the 'latest' sync job is determined by the JobHistoryHandler, I buffed up the job history handler test cases to make sure we do always return the latest sync job.
What:
Swagger UI has bad-UI and extensibility. It's not intuitive and often displays information badly (especially for more complicated structs), which leads to initial API confusion.
How:
Switch over to use Rapidoc. This is our new api docs!
Created the airbyte-public-api-docs bucket and uploaded our new html docs. Set up a CloudFront distribution to enable HTTPS.
A copy of our html docs is checked in in this PR. As you can see, this is a minimal file that relies on a React component published by Rapidoc. It is pointed to the api docs in our master branch and will automatically update itself; do not expect us to need to touch this api file unless we want to make display changes.
I consolidated all files involving api documentation into the docs/api folder for simplicity. I also moved the generated html file to the generated-api-html folder within the api folder to make it clear it was generated. Finally, added a README to document our set up.
Benefits:
UI is much cleaner.
We control the html source file, which means it'll be easier for us to make changes in the future.
Expose the latest connector version to users for better connector UX experience. The airbyte webapp currently displays the docker tag. Users can specify a custom tag to use within using the same cell. However, users do not know what the latest connector image tag is without manually checking the Airbyte github or change log. These changes will provide an endpoint for the webapp to retrieve the relevant information.
The main change is adding a list_latest endpoint to the SourceDefinition and DestinationDefinition resources. Some minor refactors here and there for code readability and maintenance. Note, we use Github as a source of truth for the latest master list.
See this google document to better understand the thought behind this: https://docs.google.com/document/d/1CeqSw9ooWDZP0yvVSmi28DgIHrlW97Tlbm9lIDSzipQ/edit
* Previously if the job failed at all the API would return a 4XX or 500. This prevented the UI from being able to display logs from the failed job.
* Instead, if the underlying job fails the API will still return a 200. The schema field will be empty however and the consumer must check the jobInfo.status field to determine if the job succeeded or not.
* Will still return 422 on invalid inputs.
* add correct but failing tests for desensitizing logs
* make sure rest of scheduler contents are written to a file
* log server requests to file
* working
* working
* clean up
* only filter out connectionConfiguratoin instead of classes
* remove syncMode from connections (sync mode is already controlled at the stream level, this is just an artifact of the original design.)
* make supportedSyncModes required in the API. (functionally it was already never null because of how we did conversions. it still can be empty.)
* Remove onboardingComplete flag (no longer used).
* Add news, anonymousDataCollection, and securityUpdates to the read struct so that we can continue to do PUT instead of PATCH when updating workspaces. workspaces/update was prevously only called once at when Airbyte was started up, but with import / export it can now be called more frequently.
* add scaffolding
* add backend api handling
* add other fields for merging
* keep defaults separate
* add good unit tests and fix bug
* remove sout
* add tests for discovery combination and fix lack of field selection propagation
* address some pr comments
* address some more pr comments
* remove wait for completion
* use first field
* fmt
* fix tests that expected sync sync jobs
* fmt
* Change api calls for source create in UI
* Change api calls for destination create in UI
* Change api calls for source and destination update in UI
* Update api calls for source and destination check connection
* Fix connector data
* Add failed logs to create source view
* Add failed logs to update source and destination view
* Add failed logs to create destination view
* Create util function for error responce
* Add failed logs to create source and destination in onboarding
* Add failed logs to connection schema view
* Change 'incomplete' status message to 'retrying' message
* Trim the spaces before and after the text fields in JSON configs
* Fix status
* Remove unused webBackend methods
