cbf9ea76c1
* initial work to parse manifest objects into pydantic models * pr feedback and some other validations * rerun manifest schema generation * remove field constraint due to bug * initial work to construct most components from greenhouse * custom components parse subcomponent fields correctly and adding a few more component constructors * construct components from gnews * first pass at posthog.yaml * Handle nested custom components with list values. Also includes updates to posthog.yaml, including autoformatting changes. * adding constructors for slicers, filters, and transformations and a few bug fixes * make sed work across multiple OS * add NoAuth component * fix handling of custom components with nested list * Autogenerate `TYPE_NAME_TO_MODEL` mapping * Handle default kwargs not defined on model for custom components * Re-add `options` for CartesianProductStreamSlicer for backwards compat with custom stream slicers * add basic unit tests for the model component factory * add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility * Remove `_get_defaults`; using actual default values on classes instead * Add backoff strategy component creation functions * add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility * add lots of tests to construct components from the pydantic models and a few bug fixes * add a few tests for the model to component factory * add catch * fix a bug where propagated schema doesn't work with old factory * clean up a few files * add type inference for custom components, more tests and some refactoring of the model factory * template, docs, manifest updates, pr feedback and some cleanup * pr feedback and polish schema a bit * fix tests from the latest rebase of master * fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow * fix CheckStream bug cleanup and a few small tweaks and polish * add additional test to cover bug case * fix formatting * 🤖 Bump minor version of Airbyte CDK Co-authored-by: Catherine Noll <noll.catherine@gmail.com> Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
40 lines
1.7 KiB
Bash
Executable File
40 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
[ -z "$ROOT_DIR" ] && exit 1
|
|
|
|
YAML_DIR=airbyte-cdk/python/airbyte_cdk/sources/declarative
|
|
OUTPUT_DIR=airbyte-cdk/python/airbyte_cdk/sources/declarative/models
|
|
|
|
function main() {
|
|
rm -rf "$ROOT_DIR/$OUTPUT_DIR"/*.py
|
|
echo "# generated by generate-component-manifest-files" > "$ROOT_DIR/$OUTPUT_DIR"/__init__.py
|
|
|
|
for f in "$ROOT_DIR/$YAML_DIR"/*.yaml; do
|
|
filename_wo_ext=$(basename "$f" | cut -d . -f 1)
|
|
echo "from .$filename_wo_ext import *" >> "$ROOT_DIR/$OUTPUT_DIR"/__init__.py
|
|
|
|
docker run --user "$(id -u):$(id -g)" -v "$ROOT_DIR":/airbyte airbyte/code-generator:dev \
|
|
--input "/airbyte/$YAML_DIR/$filename_wo_ext.yaml" \
|
|
--output "/airbyte/$OUTPUT_DIR/$filename_wo_ext.py" \
|
|
--use-title-as-name \
|
|
--disable-timestamp \
|
|
--enum-field-as-literal one
|
|
|
|
# There is a limitation of Pydantic where a model's private fields starting with an underscore are inaccessible.
|
|
# The Pydantic model generator replaces special characters like $ with the underscore which results in all
|
|
# component's $options field resulting in _options. We have been unable to find a workaround in the model generator
|
|
# or while reading the field. There is no estimated timeline on the fix even though it is widely debated here:
|
|
# https://github.com/pydantic/pydantic/issues/288.
|
|
#
|
|
# Our low effort way to address this is to perform additional post-processing to rename _options to options.
|
|
# We can revisit this if there is movement on a fix.
|
|
temp_file=$(mktemp)
|
|
sed 's/_options:/options:/g' "$ROOT_DIR/$OUTPUT_DIR/$filename_wo_ext.py" > "${temp_file}"
|
|
mv "${temp_file}" "$ROOT_DIR/$OUTPUT_DIR/$filename_wo_ext.py"
|
|
done
|
|
}
|
|
|
|
main "$@"
|