From b928cf10730b97edc2f2eb265d3fec92eb42dd1e Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Mon, 28 Jan 2019 15:53:39 +0200 Subject: [PATCH 01/34] Initial commit with proposed structure and files. --- .circleci/config.yml | 1 + .gitignore | 26 +++++++ CONTRIBUTING.md | 99 +++++++++++++++++++++++++ GRUNTWORK_PHILOSOPHY.md | 64 ++++++++++++++++ LICENSE => LICENSE.txt | 0 NOTICE | 4 + README.md | 53 +++++++++++++ examples/cloud-sql-mysql/.gitkeep | 0 examples/cloud-sql-postgres/.gitkeep | 0 modules/cloud-sql/.gitkeep | 0 test/Gopkg.lock | 33 +++++++++ test/Gopkg.toml | 25 +++++++ test/README.md | 59 +++++++++++++++ test/example_cloud_sql_mysql_test.go | 16 ++++ test/example_cloud_sql_postgres_test.go | 16 ++++ 15 files changed, 396 insertions(+) create mode 100644 .circleci/config.yml create mode 100644 .gitignore create mode 100644 CONTRIBUTING.md create mode 100644 GRUNTWORK_PHILOSOPHY.md rename LICENSE => LICENSE.txt (100%) create mode 100644 NOTICE create mode 100644 README.md create mode 100644 examples/cloud-sql-mysql/.gitkeep create mode 100644 examples/cloud-sql-postgres/.gitkeep create mode 100644 modules/cloud-sql/.gitkeep create mode 100644 test/Gopkg.lock create mode 100644 test/Gopkg.toml create mode 100644 test/README.md create mode 100644 test/example_cloud_sql_mysql_test.go create mode 100644 test/example_cloud_sql_postgres_test.go diff --git a/.circleci/config.yml b/.circleci/config.yml new file mode 100644 index 0000000..199337b --- /dev/null +++ b/.circleci/config.yml @@ -0,0 +1 @@ +# NO-OP \ No newline at end of file diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..190168e --- /dev/null +++ b/.gitignore @@ -0,0 +1,26 @@ +# Terraform files +.terraform +terraform.tfstate +terraform.tfvars +*.tfstate* +*.zip + +# OS X files +.history +.DS_Store + +# IntelliJ files +.idea_modules +*.iml +*.iws +*.ipr +.idea/ +build/ +*/build/ +out/ + +# Go best practices dictate that libraries should not include the vendor directory +vendor + +#VIM swap files +*.swp diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..20b9986 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,99 @@ +# Contribution Guidelines + +Contributions to this Module are very welcome! We follow a fairly standard [pull request process]( +https://help.github.com/articles/about-pull-requests/) for contributions, subject to the following guidelines: + +1. [File a GitHub issue](#file-a-github-issue) +1. [Update the documentation](#update-the-documentation) +1. [Update the tests](#update-the-tests) +1. [Update the code](#update-the-code) +1. [Create a pull request](#create-a-pull-request) +1. [Merge and release](#merge-and-release) + +## File a GitHub issue + +Before starting any work, we recommend filing a GitHub issue in this repo. This is your chance to ask questions and +get feedback from the maintainers and the community before you sink a lot of time into writing (possibly the wrong) +code. If there is anything you're unsure about, just ask! + +## Update the documentation + +We recommend updating the documentation *before* updating any code (see [Readme Driven +Development](http://tom.preston-werner.com/2010/08/23/readme-driven-development.html)). This ensures the documentation +stays up to date and allows you to think through the problem at a high level before you get lost in the weeds of +coding. + +## Update the tests + +We also recommend updating the automated tests *before* updating any code (see [Test Driven +Development](https://en.wikipedia.org/wiki/Test-driven_development)). That means you add or update a test case, +verify that it's failing with a clear error message, and *then* make the code changes to get that test to pass. This +ensures the tests stay up to date and verify all the functionality in this Module, including whatever new +functionality you're adding in your contribution. Check out the [tests](https://github.com/gruntwork-io/terraform-google-sql/tree/master/test) folder for instructions on running the +automated tests. + +## Update the code + +At this point, make your code changes and use your new test case to verify that everything is working. As you work, +keep in mind two things: + +1. Backwards compatibility +1. Downtime + +### Backwards compatibility + +Please make every effort to avoid unnecessary backwards incompatible changes. With Terraform code, this means: + +1. Do not delete, rename, or change the type of input variables. +1. If you add an input variable, it should have a `default`. +1. Do not delete, rename, or change the type of output variables. +1. Do not delete or rename a module in the `modules` folder. + +If a backwards incompatible change cannot be avoided, please make sure to call that out when you submit a pull request, +explaining why the change is absolutely necessary. + +### Downtime + +Bear in mind that the Terraform code in this Module is used by real companies to run real infrastructure in +production, and certain types of changes could cause downtime. For example, consider the following: + +1. If you rename a resource (e.g. `google_sql_database_instance "foo"` -> `google_sql_database_instance "bar"`), Terraform will see that as deleting + the old resource and creating a new one. +1. If you change certain attributes of a resource (e.g. the `name` of an `google_compute_instance`), the cloud provider (e.g. Google) may + treat that as an instruction to delete the old resource and a create a new one. + +Deleting certain types of resources (e.g. virtual servers, load balancers) can cause downtime, so when making code +changes, think carefully about how to avoid that. For example, can you avoid downtime by using +[create_before_destroy](https://www.terraform.io/docs/configuration/resources.html#create_before_destroy)? Or via +the `terraform state` command? If so, make sure to note this in our pull request. If downtime cannot be avoided, +please make sure to call that out when you submit a pull request. + + +### Formatting and pre-commit hooks + +You must run `terraform fmt` on the code before committing. You can configure your computer to do this automatically +using pre-commit hooks managed using [pre-commit](http://pre-commit.com/): + +1. [Install pre-commit](http://pre-commit.com/#install). E.g.: `brew install pre-commit`. +1. Install the hooks: `pre-commit install`. + +That's it! Now just write your code, and every time you commit, `terraform fmt` will be run on the files you're +committing. + + +## Create a pull request + +[Create a pull request](https://help.github.com/articles/creating-a-pull-request/) with your changes. Please make sure +to include the following: + +1. A description of the change, including a link to your GitHub issue. +1. The output of your automated test run, preferably in a [GitHub Gist](https://gist.github.com/). We cannot run + automated tests for pull requests automatically due to [security + concerns](https://circleci.com/docs/fork-pr-builds/#security-implications), so we need you to manually provide this + test output so we can verify that everything is working. +1. Any notes on backwards incompatibility or downtime. + +## Merge and release + +The maintainers for this repo will review your code and provide feedback. If everything looks good, they will merge the +code and release a new version, which you'll be able to find in the [releases page](../../releases). \ No newline at end of file diff --git a/GRUNTWORK_PHILOSOPHY.md b/GRUNTWORK_PHILOSOPHY.md new file mode 100644 index 0000000..22a28a8 --- /dev/null +++ b/GRUNTWORK_PHILOSOPHY.md @@ -0,0 +1,64 @@ +# Gruntwork Philosophy + +At Gruntwork, we strive to accelerate the deployment of production grade infrastructure by prodiving a library of +stable, reusable, and battle tested infrastructure as code organized into a series of [modules](#what-is-a-module) with +[submodules](#what-is-a-submodule). Each module represents a particular set of infrastructure that is componentized into +smaller pieces represented by the submodules within the module. By doing so, we have built a composable library that can +be combined into building out everything from simple single service deployments to complicated microservice setups so +that your infrastructure can grow with your business needs. Every module we provide is built with the [production grade +infrastruture checklist](#production-grade-infrastructure-checklist) in mind, ensuring that the services you deploy are +resilient, fault tolerant, and scalable. + + +## What is a Module? + +A Module is a reusable, tested, documented, configurable, best-practices definition of a single piece of Infrastructure +(e.g., Docker cluster, VPC, Jenkins, Consul), written using a combination of [Terraform](https://www.terraform.io/), Go, +and Bash. A module contains a set of automated tests, documentation, and examples that have been proven in production, +providing the underlying infrastructure for [Gruntwork's customers](https://www.gruntwork.io/customers). + +Instead of figuring out the details of how to run a piece of infrastructure from scratch, you can reuse existing code +that has been proven in production. And instead of maintaining all that infrastructure code yourself, you can leverage +the work of the community to pick up infrastructure improvements through a version number bump. + + +## What is a Submodule? + +Each Infrastructure Module consists of one or more orthogonal Submodules that handle some specific aspect of that +Infrastructure Module's functionality. Breaking the code up into multiple submodules makes it easier to reuse and +compose to handle many different use cases. Although Modules are designed to provide an end to end solution to manage +the relevant infrastructure by combining the Submodules defined in the Module, Submodules can be used independently for +specific functionality that you need in your infrastructure code. + + +## Production Grade Infrastructure Checklist + +At Gruntwork, we have learned over the years that it is not enough to just get the services up and running in a publicly +accessible space to call your application "production-ready." There are many more things to consider, and oftentimes +many of these considerations are missing in the deployment plan of applications. These topics come up as afterthoughts, +and are learned the hard way after the fact. That is why we codified all of them into a checklist that can be used as a +reference to help ensure that they are considered before your application goes to production, and conscious decisions +are made to neglect particular components if needed, as opposed to accidentally omitting them from consideration. + + + +| Task | Description | Example tools | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| +| Install | Install the software binaries and all dependencies. | Bash, Chef, Ansible, Puppet | +| Configure | Configure the software at runtime. Includes port settings, TLS certs, service discovery, leaders, followers, replication, etc. | Bash, Chef, Ansible, Puppet | +| Provision | Provision the infrastructure. Includes EC2 instances, load balancers, network topology, security gr oups, IAM permissions, etc. | Terraform, CloudFormation | +| Deploy | Deploy the service on top of the infrastructure. Roll out updates with no downtime. Includes blue-green, rolling, and canary deployments. | Scripts, Orchestration tools (ECS, k8s, Nomad) | +| High availability | Withstand outages of individual processes, EC2 instances, services, Availability Zones, and regions. | Multi AZ, multi-region, replication, ASGs, ELBs | +| Scalability | Scale up and down in response to load. Scale horizontally (more servers) and/or vertically (bigger servers). | ASGs, replication, sharding, caching, divide and conquer | +| Performance | Optimize CPU, memory, disk, network, GPU, and usage. Includes query tuning, benchmarking, load testing, and profiling. | Dynatrace, valgrind, VisualVM, ab, Jmeter | +| Networking | Configure static and dynamic IPs, ports, service discovery, firewalls, DNS, SSH access, and VPN access. | EIPs, ENIs, VPCs, NACLs, SGs, Route 53, OpenVPN | +| Security | Encryption in transit (TLS) and on disk, authentication, authorization, secrets management, server hardening. | ACM, EBS Volumes, Cognito, Vault, CIS | +| Metrics | Availability metrics, business metrics, app metrics, server metrics, events, observability, tracing, and alerting. | CloudWatch, DataDog, New Relic, Honeycomb | +| Logs | Rotate logs on disk. Aggregate log data to a central location. | CloudWatch logs, ELK, Sumo Logic, Papertrail | +| Backup and Restore | Make backups of DBs, caches, and other data on a scheduled basis. Replicate to separate region/account. | RDS, ElastiCache, ec2-snapper, Lambda | +| Cost optimization | Pick proper instance types, use spot and reserved instances, use auto scaling, and nuke unused resources. | ASGs, spot instances, reserved instances | +| Documentation | Document your code, architecture, and practices. Create playbooks to respond to incidents. | READMEs, wikis, Slack | +| Tests | Write automated tests for your infrastructure code. Run tests after every commit and nightly. | Terratest | \ No newline at end of file diff --git a/LICENSE b/LICENSE.txt similarity index 100% rename from LICENSE rename to LICENSE.txt diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..24a7708 --- /dev/null +++ b/NOTICE @@ -0,0 +1,4 @@ +terraform-google-sql +Copyright 2019 Gruntwork, Inc. + +This product includes software developed at Gruntwork (https://www.gruntwork.io/). diff --git a/README.md b/README.md new file mode 100644 index 0000000..6a92293 --- /dev/null +++ b/README.md @@ -0,0 +1,53 @@ +[![Maintained by Gruntwork.io](https://img.shields.io/badge/maintained%20by-gruntwork.io-%235849a6.svg)](https://gruntwork.io/?ref=repo_google_cloudsql) + +# Cloud SQL Modules + +This repo contains modules for running relational databases such as MySQL and PostgreSQL on Google's +[Cloud SQL](https://cloud.google.com/sql/) on [GCP](https://cloud.google.com/). + +## Code included in this Module + +* [cloud-sql](/modules/cloud-sql): Deploy a Cloud SQL cluster. + + +## What is Cloud SQL? + +Cloud SQL is Google's fully-managed database service that makes it easy to set up, maintain, manage, and administer +your relational databases on Google Cloud Platform. Cloud SQL automatically includes Data replication between multiple +zones with automatic failover, automated and on-demand backups, and point-in-time recovery. + +You can learn more Cloud SQL from [the official documentation](https://cloud.google.com/sql/docs/). + +## Who maintains this Module? + +This Module and its Submodules are maintained by [Gruntwork](http://www.gruntwork.io/). If you are looking for help or +commercial support, send an email to +[support@gruntwork.io](mailto:support@gruntwork.io?Subject=Google%20SQL%20Module). + +Gruntwork can help with: + +* Setup, customization, and support for this Module. +* Modules and submodules for other types of infrastructure, such as VPCs, Docker clusters, databases, and continuous + integration. +* Modules and Submodules that meet compliance requirements, such as HIPAA. +* Consulting & Training on GCP, AWS, Terraform, and DevOps. + + +## How do I contribute to this Module? + +Contributions are very welcome! Check out the [Contribution Guidelines](/CONTRIBUTING.md) for instructions. + + +## How is this Module versioned? + +This Module follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, along +with the changelog, in the [Releases Page](../../releases). + +During initial development, the major version will be 0 (e.g., `0.x.y`), which indicates the code does not yet have a +stable API. Once we hit `1.0.0`, we will make every effort to maintain a backwards compatible API and use the MAJOR, +MINOR, and PATCH versions on each release to indicate any incompatibilities. + + +## License + +Please see [LICENSE.txt](/LICENSE.txt) for details on how the code in this repo is licensed. diff --git a/examples/cloud-sql-mysql/.gitkeep b/examples/cloud-sql-mysql/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/examples/cloud-sql-postgres/.gitkeep b/examples/cloud-sql-postgres/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/modules/cloud-sql/.gitkeep b/modules/cloud-sql/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/test/Gopkg.lock b/test/Gopkg.lock new file mode 100644 index 0000000..3c7bff3 --- /dev/null +++ b/test/Gopkg.lock @@ -0,0 +1,33 @@ +# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'. + + +[[projects]] + digest = "1:0deddd908b6b4b768cfc272c16ee61e7088a60f7fe2f06c547bd3d8e1f8b8e77" + name = "github.com/davecgh/go-spew" + packages = ["spew"] + pruneopts = "" + revision = "8991bc29aa16c548c550c7ff78260e27b9ab7c73" + version = "v1.1.1" + +[[projects]] + digest = "1:256484dbbcd271f9ecebc6795b2df8cad4c458dd0f5fd82a8c2fa0c29f233411" + name = "github.com/pmezard/go-difflib" + packages = ["difflib"] + pruneopts = "" + revision = "792786c7400a136282c1664665ae0a8db921c6c2" + version = "v1.0.0" + +[[projects]] + digest = "1:381bcbeb112a51493d9d998bbba207a529c73dbb49b3fd789e48c63fac1f192c" + name = "github.com/stretchr/testify" + packages = ["assert"] + pruneopts = "" + revision = "ffdc059bfe9ce6a4e144ba849dbedead332c6053" + version = "v1.3.0" + +[solve-meta] + analyzer-name = "dep" + analyzer-version = 1 + input-imports = ["github.com/stretchr/testify/assert"] + solver-name = "gps-cdcl" + solver-version = 1 diff --git a/test/Gopkg.toml b/test/Gopkg.toml new file mode 100644 index 0000000..7d7581c --- /dev/null +++ b/test/Gopkg.toml @@ -0,0 +1,25 @@ +# Gopkg.toml example +# +# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md +# for detailed Gopkg.toml documentation. +# +# required = ["github.com/user/thing/cmd/thing"] +# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"] +# +# [[constraint]] +# name = "github.com/user/project" +# version = "1.0.0" +# +# [[constraint]] +# name = "github.com/user/project2" +# branch = "dev" +# source = "github.com/myfork/project2" +# +# [[override]] +# name = "github.com/x/y" +# version = "2.4.0" + +[[constraint]] + name = "github.com/gruntwork-io/terratest" + source = "git@github.com:gruntwork-io/terratest" + version = "0.13.22" diff --git a/test/README.md b/test/README.md new file mode 100644 index 0000000..3f0f0d9 --- /dev/null +++ b/test/README.md @@ -0,0 +1,59 @@ +# Tests + +This folder contains automated tests for this Module. All of the tests are written in [Go](https://golang.org/). +Most of these are "integration tests" that deploy real infrastructure using Terraform and verify that infrastructure +works as expected using a helper library called [Terratest](https://github.com/gruntwork-io/terratest). + + + +## WARNING WARNING WARNING + +**Note #1**: Many of these tests create real resources in an AWS account and then try to clean those resources up at +the end of a test run. That means these tests may cost you money to run! When adding tests, please be considerate of +the resources you create and take extra care to clean everything up when you're done! + +**Note #2**: Never forcefully shut the tests down (e.g. by hitting `CTRL + C`) or the cleanup tasks won't run! + +**Note #3**: We set `-timeout 60m` on all tests not because they necessarily take that long, but because Go has a +default test timeout of 10 minutes, after which it forcefully kills the tests with a `SIGQUIT`, preventing the cleanup +tasks from running. Therefore, we set an overlying long timeout to make sure all tests have enough time to finish and +clean up. + + + +## Running the tests + +### Prerequisites + +- Install the latest version of [Go](https://golang.org/). +- Install [dep](https://github.com/golang/dep) for Go dependency management. +- Install [Terraform](https://www.terraform.io/downloads.html). +- Configure your Google credentials using one of the [options supported by GCP](https://cloud.google.com/docs/authentication/getting-started). + + +### One-time setup + +Download Go dependencies using dep: + +``` +cd test +dep ensure +``` + + +### Run all the tests + +```bash +cd test +go test -v -timeout 60m +``` + + +### Run a specific test + +To run a specific test called `TestFoo`: + +```bash +cd test +go test -v -timeout 60m -run TestFoo +``` \ No newline at end of file diff --git a/test/example_cloud_sql_mysql_test.go b/test/example_cloud_sql_mysql_test.go new file mode 100644 index 0000000..5310c38 --- /dev/null +++ b/test/example_cloud_sql_mysql_test.go @@ -0,0 +1,16 @@ +package test + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in +// the templates +// TODO: try to actually connect to the RDS DBs and check they are working +func TestCloudSQLMySql(t *testing.T) { + t.Parallel() + + assert.Equal(t, "3306", "3306") +} diff --git a/test/example_cloud_sql_postgres_test.go b/test/example_cloud_sql_postgres_test.go new file mode 100644 index 0000000..c51466c --- /dev/null +++ b/test/example_cloud_sql_postgres_test.go @@ -0,0 +1,16 @@ +package test + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in +// the templates +// TODO: try to actually connect to the RDS DBs and check they are working +func TestCloudSQLPostgres(t *testing.T) { + t.Parallel() + + assert.Equal(t, "5432", "5432") +} From b7aa325fa024144334a33e095211c5eadd24cf02 Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Mon, 28 Jan 2019 22:01:01 +0200 Subject: [PATCH 02/34] Update README.md Co-Authored-By: autero1 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6a92293..201dacb 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ You can learn more Cloud SQL from [the official documentation](https://cloud.goo ## Who maintains this Module? -This Module and its Submodules are maintained by [Gruntwork](http://www.gruntwork.io/). If you are looking for help or +This Module and its Submodules are maintained by [Gruntwork](http://www.gruntwork.io/). Read the [Gruntwork Philosophy](/GRUNTWORK_PHILOSOPHY.md) document to learn more about how Gruntwork builds production grade infrastructure code. If you are looking for help or commercial support, send an email to [support@gruntwork.io](mailto:support@gruntwork.io?Subject=Google%20SQL%20Module). From e67085156e1e8c584abd02d938d13b9db66ecfad Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 30 Jan 2019 18:08:07 +0200 Subject: [PATCH 03/34] Add separate modules for Postgres and MySQL. More detailed documentation of core capabilities. --- README.md | 13 +++++++++---- modules/mysql/.gitkeep | 0 modules/postgresql/.gitkeep | 0 3 files changed, 9 insertions(+), 4 deletions(-) create mode 100644 modules/mysql/.gitkeep create mode 100644 modules/postgresql/.gitkeep diff --git a/README.md b/README.md index 201dacb..cf41fe7 100644 --- a/README.md +++ b/README.md @@ -7,16 +7,21 @@ This repo contains modules for running relational databases such as MySQL and Po ## Code included in this Module -* [cloud-sql](/modules/cloud-sql): Deploy a Cloud SQL cluster. +* [mysql](/modules/mysql): Deploy a Cloud SQL MySQL cluster. +* [postgresql](/modules/mysql): Deploy a Cloud SQL PostgreSQL cluster. ## What is Cloud SQL? Cloud SQL is Google's fully-managed database service that makes it easy to set up, maintain, manage, and administer -your relational databases on Google Cloud Platform. Cloud SQL automatically includes Data replication between multiple -zones with automatic failover, automated and on-demand backups, and point-in-time recovery. +your relational databases on Google Cloud Platform. Cloud SQL automatically includes: -You can learn more Cloud SQL from [the official documentation](https://cloud.google.com/sql/docs/). +* Data replication between multiple zones with automatic failover. +* Automated and on-demand backups, and point-in-time recovery. +* Data encryption on networks, database tables, temporary files, and backups. +* Secure external connections with the [Cloud SQL Proxy](https://cloud.google.com/sql/docs/mysql/sql-proxy) or with the SSL/TLS protocol. + +You can learn more about Cloud SQL from [the official documentation](https://cloud.google.com/sql/docs/). ## Who maintains this Module? diff --git a/modules/mysql/.gitkeep b/modules/mysql/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/modules/postgresql/.gitkeep b/modules/postgresql/.gitkeep new file mode 100644 index 0000000..e69de29 From 3cd7599c94f79f7ef5afe3894f99ca73903ede8e Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 30 Jan 2019 22:42:10 +0200 Subject: [PATCH 04/34] Readme for cloud sql module --- modules/cloud-sql/README.md | 52 +++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 modules/cloud-sql/README.md diff --git a/modules/cloud-sql/README.md b/modules/cloud-sql/README.md new file mode 100644 index 0000000..eb8bfed --- /dev/null +++ b/modules/cloud-sql/README.md @@ -0,0 +1,52 @@ +# Cloud SQL Module + +This module creates a [Google Cloud SQL](https://cloud.google.com/sql/) cluster. The cluster is managed by Google, +automating backups, replication, patches, and updates. + +You can use Cloud SQL with either [MySQL](https://cloud.google.com/sql/docs/mysql/) or [PostgreSQL](https://cloud.google.com/sql/docs/postgres/). + +## How do you use this module? + +See the [examples](/examples) folder for an example. + +## How do you connect to the database? + +This module provides the connection details as [Terraform output +variables](https://www.terraform.io/intro/getting-started/outputs.html): + +1. **Cluster endpoint**: The endpoint for the whole cluster. You should always use this URL for writes, as it points to + the primary. +1. **Instance endpoints**: A comma-separated list of all DB instance URLs in the cluster, including the primary and all + read replicas. Use these URLs for reads (see "How do you scale this DB?" below). +1. **Port**: The port to use to connect to the endpoints above. + +TODO: Connectivity and output + +For more info, see [Connecting to Cloud SQL from External Applications](https://cloud.google.com/sql/docs/mysql/connect-external-app). + +You can programmatically extract these variables in your Terraform templates and pass them to other resources (e.g. +pass them to User Data in your EC2 instances). You'll also see the variables at the end of each `terraform apply` call +or if you run `terraform output`. + +## How do you scale this database? + +* **Storage**: Cloud SQL manages storage for you, automatically growing cluster volume up to 10TB. +* **Vertical scaling**: To scale vertically (i.e. bigger DB instances with more CPU and RAM), use the `machine_type` + input variable. For a list of Cloud SQL Machine Types, see [Cloud SQL Pricing](https://cloud.google.com/sql/pricing#2nd-gen-pricing). +* **Horizontal scaling**: To scale horizontally, you can add more replicas using the `instance_count` input variable, + and Aurora will automatically deploy the new instances, sync them to the master, and make them available as read + replicas. + +## How do you configure this module? + +This module allows you to configure a number of parameters, such as backup windows, maintenance window, port number, +and encryption. For a list of all available variables and their descriptions, see [variables.tf](./variables.tf). + +## Known Issues + +### Instance Recovery + +Due to limitations on the current `terraform` provider for Google, it is not possible to restore backups with `terraform`. +See https://github.com/terraform-providers/terraform-provider-google/issues/2446 + + From 941c7c10f9e2e231e498ab9bb53db24570897bd6 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 30 Jan 2019 22:44:30 +0200 Subject: [PATCH 05/34] Readme for cloud sql module --- modules/cloud-sql/variables.tf | 140 +++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 modules/cloud-sql/variables.tf diff --git a/modules/cloud-sql/variables.tf b/modules/cloud-sql/variables.tf new file mode 100644 index 0000000..be32553 --- /dev/null +++ b/modules/cloud-sql/variables.tf @@ -0,0 +1,140 @@ +# --------------------------------------------------------------------------------------------------------------------- +# REQUIRED PARAMETERS +# These variables are expected to be passed in by the operator +# --------------------------------------------------------------------------------------------------------------------- + +variable "project" { + description = "The project ID to host the database in." +} + +variable "region" { + description = "The region to host the database in." +} + +variable "name" { + description = "The name of the database instance." +} + +variable "engine" { + description = "The engine version of the database, e.g. `MYSQL_5_7` or `POSTGRES_9_6`." +} + +variable "master_instance_name" { + description = "The name of the instance that will act as the master in the replication setup. Note, this requires the master to have binary_log_enabled set, as well as existing backups." + default = "" +} + +variable "machine_type" { + description = "The machine tier (First Generation) or type (Second Generation). See this page for supported tiers and pricing: https://cloud.google.com/sql/pricing" + default = "db-f1-micro" +} + +variable "db_name" { + description = "Name of the default database to create" + default = "default" +} + +variable "db_charset" { + description = "The charset for the default database" + default = "" +} + +variable "db_collation" { + description = "The collation for the default database. Example for MySQL databases: 'utf8_general_ci', and Postgres: 'en_US.UTF8'" + default = "" +} + +variable "master_user_name" { + description = "The name of the default user" + default = "default" +} + +variable "master_user_host" { + description = "The host for the default user" + default = "%" +} + +variable "master_user_password" { + description = "The password for the default user." + default = "" +} + +variable "activation_policy" { + description = "This specifies when the instance should be active. Can be either `ALWAYS`, `NEVER` or `ON_DEMAND`." + default = "ALWAYS" +} + +variable "authorized_gae_applications" { + description = "A list of Google App Engine (GAE) project names that are allowed to access this instance." + type = "list" + default = [] +} + +variable "disk_autoresize" { + description = "Second Generation only. Configuration to increase storage size automatically." + default = true +} + +variable "disk_size" { + description = "Second generation only. The size of data disk, in GB. Size of a running instance cannot be reduced but can be increased." + default = 10 +} + +variable "disk_type" { + description = "Second generation only. The type of data disk: `PD_SSD` or `PD_HDD`." + default = "PD_SSD" +} + +variable "pricing_plan" { + description = "First generation only. Pricing plan for this instance, can be one of `PER_USE` or `PACKAGE`." + default = "PER_USE" +} + +variable "replication_type" { + description = "Replication type for this instance, can be one of `ASYNCHRONOUS` or `SYNCHRONOUS`." + default = "SYNCHRONOUS" +} + +variable "flags" { + description = "List of Cloud SQL flags that are applied to the database server" + default = [] + type = "list" +} + +# IGNORE EVERYTHING BELOW + +variable backup_configuration { + description = "The backup_configuration settings subblock for the database setings" + type = "map" + default = {} +} + +variable ip_configuration { + description = "The ip_configuration settings subblock" + type = "list" + default = [{}] +} + +variable location_preference { + description = "The location_preference settings subblock" + type = "list" + default = [] +} + +variable maintenance_window { + description = "The maintenance_window settings subblock" + type = "list" + default = [] +} + +variable replica_configuration { + description = "The optional replica_configuration block for the database instance" + type = "list" + default = [] +} + +# --------------------------------------------------------------------------------------------------------------------- +# OPTIONAL PARAMETERS +# Generally, these values won't need to be changed. +# --------------------------------------------------------------------------------------------------------------------- +# TODO: \ No newline at end of file From dfb4eb8538174cc827001b47ee7e470d90a55bcc Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Fri, 1 Feb 2019 12:09:36 +0200 Subject: [PATCH 06/34] Add sandbox for experimentation --- examples/cloud-sql-mysql/.gitkeep | 0 examples/cloud-sql-mysql/main.tf | 78 +++++++++++++++++++++++++++++++ examples/sandbox/main.tf | 45 ++++++++++++++++++ modules/cloud-sql/main.tf | 61 ++++++++++++++++++++++++ modules/cloud-sql/outputs.tf | 19 ++++++++ modules/cloud-sql/variables.tf | 11 +++-- 6 files changed, 211 insertions(+), 3 deletions(-) delete mode 100644 examples/cloud-sql-mysql/.gitkeep create mode 100644 examples/cloud-sql-mysql/main.tf create mode 100644 examples/sandbox/main.tf create mode 100644 modules/cloud-sql/main.tf create mode 100644 modules/cloud-sql/outputs.tf diff --git a/examples/cloud-sql-mysql/.gitkeep b/examples/cloud-sql-mysql/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf new file mode 100644 index 0000000..29bbd7e --- /dev/null +++ b/examples/cloud-sql-mysql/main.tf @@ -0,0 +1,78 @@ +provider "google-beta" { + region = "${var.region}" + project = "${var.project}" +} + +# Use Terraform 0.10.x so that we can take advantage of Terraform GCP functionality as a separate provider via +# https://github.com/terraform-providers/terraform-provider-google +terraform { + required_version = ">= 0.10.3" +} + +variable "region" { + default = "europe-north1" +} + +variable "project" { + default = "dev-sandbox-228703" +} + + +variable "zone" { + default = "europe-north1-a" +} + +variable "mysql_version" { + default = "MYSQL_5_6" +} + +resource "random_id" "name" { + byte_length = 2 +} + +resource "google_compute_network" "private_network" { + provider = "google-beta" + name = "private-network" +} + +resource "google_compute_global_address" "private_ip_address" { + provider = "google-beta" + name = "private-ip-address" + purpose = "VPC_PEERING" + address_type = "INTERNAL" + prefix_length = 16 + network = "${google_compute_network.private_network.self_link}" +} + +resource "google_service_networking_connection" "private_vpc_connection" { + provider = "google-beta" + network = "${google_compute_network.private_network.self_link}" + service = "servicenetworking.googleapis.com" + reserved_peering_ranges = ["${google_compute_global_address.private_ip_address.name}"] +} + + +module "mysql-db" { + source = "../../modules/cloud-sql" + name = "example-mysql-${random_id.name.hex}" + region = "${var.region}" + engine = "${var.mysql_version}" + project = "${var.project}" + machine_type = "db-f1-micro" + + ip_configuration = [ + { + ipv4_enabled = "true" + private_network = "${google_compute_network.private_network.self_link}" + } + ] + + # https://cloud.google.com/sql/docs/mysql/flags + flags = [ + ] +} + +output "mysql_conn" { + value = "${var.project}:${var.region}:${module.mysql-db.instance_name}" +} + diff --git a/examples/sandbox/main.tf b/examples/sandbox/main.tf new file mode 100644 index 0000000..2551ea2 --- /dev/null +++ b/examples/sandbox/main.tf @@ -0,0 +1,45 @@ +provider "google-beta" { + region = "${var.region}" +} + +variable "region" { + default = "europe-north1" +} + +variable "project" { + default = "petri-sandbox" +} + +variable "endpoints" { + type = "list" + default = ["192.168.11.1", "192.168.11.2"] +} + +data "template_file" "single_ip" { + count = "${length(var.endpoints)}" + template = < Date: Fri, 1 Feb 2019 12:30:23 +0200 Subject: [PATCH 07/34] Documentation updates --- modules/cloud-sql/README.md | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/modules/cloud-sql/README.md b/modules/cloud-sql/README.md index eb8bfed..7686730 100644 --- a/modules/cloud-sql/README.md +++ b/modules/cloud-sql/README.md @@ -9,24 +9,38 @@ You can use Cloud SQL with either [MySQL](https://cloud.google.com/sql/docs/mysq See the [examples](/examples) folder for an example. +## How do you configure this module? + +This module allows you to configure a number of parameters, such as backup windows, maintenance window, replicas +and encryption. For a list of all available variables and their descriptions, see [variables.tf](./variables.tf). + ## How do you connect to the database? +**Cloud SQL instances are created in a producer network (a VPC network internal to Google). They are not created in your VPC network.** + +You can use both [public IP](https://cloud.google.com/sql/docs/mysql/connect-admin-ip) and [private IP](https://cloud.google.com/sql/docs/mysql/private-ip) to connect to a Cloud SQL instance. +Neither connection method affects the other; you must protect the public IP connection whether the instance is configured to use private IP or not. + +You can also use the [Cloud SQL Proxy](https://cloud.google.com/sql/docs/mysql/connect-admin-proxy) to connect to an instance that is also configured to use private IP. The proxy can connect using either the private IP address or a public IP address. + This module provides the connection details as [Terraform output variables](https://www.terraform.io/intro/getting-started/outputs.html): -1. **Cluster endpoint**: The endpoint for the whole cluster. You should always use this URL for writes, as it points to - the primary. -1. **Instance endpoints**: A comma-separated list of all DB instance URLs in the cluster, including the primary and all +**TODO**: Connectivity and outputs below + + +1. **Public IP** `private_ip`: The public endpoint for the cluster. +1. **Public IP** `public_ip`: The public endpoint for the cluster. +1. **Connection name** `connection_name`: The private endpoint for the cluster. +1. **Replica endpoints** `replica_endpoints`: A comma-separated list of all DB instance URLs in the cluster, including the primary and all read replicas. Use these URLs for reads (see "How do you scale this DB?" below). -1. **Port**: The port to use to connect to the endpoints above. -TODO: Connectivity and output -For more info, see [Connecting to Cloud SQL from External Applications](https://cloud.google.com/sql/docs/mysql/connect-external-app). -You can programmatically extract these variables in your Terraform templates and pass them to other resources (e.g. -pass them to User Data in your EC2 instances). You'll also see the variables at the end of each `terraform apply` call -or if you run `terraform output`. +You can programmatically extract these variables in your Terraform templates and pass them to other resources. +You'll also see the variables at the end of each `terraform apply` call or if you run `terraform output`. + +For full connectivity options and detailed documentation, see [Connecting to Cloud SQL from External Applications](https://cloud.google.com/sql/docs/mysql/connect-external-app). ## How do you scale this database? @@ -37,11 +51,6 @@ or if you run `terraform output`. and Aurora will automatically deploy the new instances, sync them to the master, and make them available as read replicas. -## How do you configure this module? - -This module allows you to configure a number of parameters, such as backup windows, maintenance window, port number, -and encryption. For a list of all available variables and their descriptions, see [variables.tf](./variables.tf). - ## Known Issues ### Instance Recovery From bdca995e244a9679aedea0d90fc51a51facb4981 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 10:33:24 +0200 Subject: [PATCH 08/34] Move impl files under mysql -module. --- modules/{cloud-sql => mysql}/README.md | 0 modules/{cloud-sql => mysql}/main.tf | 0 modules/{cloud-sql => mysql}/outputs.tf | 0 modules/{cloud-sql => mysql}/variables.tf | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename modules/{cloud-sql => mysql}/README.md (100%) rename modules/{cloud-sql => mysql}/main.tf (100%) rename modules/{cloud-sql => mysql}/outputs.tf (100%) rename modules/{cloud-sql => mysql}/variables.tf (100%) diff --git a/modules/cloud-sql/README.md b/modules/mysql/README.md similarity index 100% rename from modules/cloud-sql/README.md rename to modules/mysql/README.md diff --git a/modules/cloud-sql/main.tf b/modules/mysql/main.tf similarity index 100% rename from modules/cloud-sql/main.tf rename to modules/mysql/main.tf diff --git a/modules/cloud-sql/outputs.tf b/modules/mysql/outputs.tf similarity index 100% rename from modules/cloud-sql/outputs.tf rename to modules/mysql/outputs.tf diff --git a/modules/cloud-sql/variables.tf b/modules/mysql/variables.tf similarity index 100% rename from modules/cloud-sql/variables.tf rename to modules/mysql/variables.tf From e1c786c2e9cfabb5cdbcbca8f9e8c23eaedcb5ca Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 17:43:14 +0200 Subject: [PATCH 09/34] Ignore test data --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 190168e..6d7a06c 100644 --- a/.gitignore +++ b/.gitignore @@ -24,3 +24,5 @@ vendor #VIM swap files *.swp + +.test-data \ No newline at end of file From 97e7571b0359615224ee2e3779e8c3183eaecf2f Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 17:43:52 +0200 Subject: [PATCH 10/34] Basic functionality --- modules/mysql/README.md | 12 +-- modules/mysql/main.tf | 40 ++++----- modules/mysql/outputs.tf | 32 ++++--- modules/mysql/variables.tf | 170 ++++++++++++++++++------------------- 4 files changed, 131 insertions(+), 123 deletions(-) diff --git a/modules/mysql/README.md b/modules/mysql/README.md index 7686730..eaad1fe 100644 --- a/modules/mysql/README.md +++ b/modules/mysql/README.md @@ -1,7 +1,9 @@ -# Cloud SQL Module +# MySQL Module -This module creates a [Google Cloud SQL](https://cloud.google.com/sql/) cluster. The cluster is managed by Google, -automating backups, replication, patches, and updates. +This module creates a [Google Cloud SQL](https://cloud.google.com/sql/) [MySQL](https://cloud.google.com/sql/docs/mysql/) cluster. +The cluster is managed by Google, automating backups, replication, patches, and updates. + +TODO: Figure out documentation format for separate modules. You can use Cloud SQL with either [MySQL](https://cloud.google.com/sql/docs/mysql/) or [PostgreSQL](https://cloud.google.com/sql/docs/postgres/). @@ -26,10 +28,8 @@ You can also use the [Cloud SQL Proxy](https://cloud.google.com/sql/docs/mysql/c This module provides the connection details as [Terraform output variables](https://www.terraform.io/intro/getting-started/outputs.html): -**TODO**: Connectivity and outputs below - -1. **Public IP** `private_ip`: The public endpoint for the cluster. +1. TODO: **Private IP** `private_ip`: The public endpoint for the cluster. 1. **Public IP** `public_ip`: The public endpoint for the cluster. 1. **Connection name** `connection_name`: The private endpoint for the cluster. 1. **Replica endpoints** `replica_endpoints`: A comma-separated list of all DB instance URLs in the cluster, including the primary and all diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index 0bc76c8..4334b91 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -1,41 +1,43 @@ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # DEPLOY A CLOUD SQL CLUSTER -# This module deploys an Cloud SQL cluster. The cluster is managed by Google and automatically handles leader +# This module deploys a Cloud SQL MySQL cluster. The cluster is managed by Google and automatically handles leader # election, replication, failover, backups, patching, and encryption. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # ------------------------------------------------------------------------------ -# CREATE THE CLOUD SQL CLUSTER +# CREATE THE CLOUD SQL MYSQL CLUSTER # # NOTE: We have multiple google_sql_database_instance resources, based on # HA, encryption and replication configuration options. # ------------------------------------------------------------------------------ -resource "google_sql_database_instance" "default" { +resource "google_sql_database_instance" "master" { name = "${var.name}" project = "${var.project}" region = "${var.region}" database_version = "${var.engine}" - master_instance_name = "${var.master_instance_name}" settings { tier = "${var.machine_type}" activation_policy = "${var.activation_policy}" authorized_gae_applications = ["${var.authorized_gae_applications}"] disk_autoresize = "${var.disk_autoresize}" - backup_configuration = ["${var.backup_configuration}"] - ip_configuration = ["${var.ip_configuration}"] - location_preference = ["${var.location_preference}"] - maintenance_window = ["${var.maintenance_window}"] + + ip_configuration { + authorized_networks = ["${var.authorized_networks}"], + ipv4_enabled = "${var.publicly_accessible}" + } + + location_preference { + follow_gae_application = "${var.follow_gae_application}" + zone = "${var.zone}" + } + disk_size = "${var.disk_size}" disk_type = "${var.disk_type}" - pricing_plan = "${var.pricing_plan}" - replication_type = "${var.replication_type}" - database_flags = ["${var.flags}"] + database_flags = ["${var.database_flags}"] availability_type = "${var.availability_type}" } - - replica_configuration = ["${var.replica_configuration}"] } # ------------------------------------------------------------------------------ @@ -43,19 +45,17 @@ resource "google_sql_database_instance" "default" { # ------------------------------------------------------------------------------ resource "google_sql_database" "default" { - count = "${var.master_instance_name == "" ? 1 : 0}" name = "${var.db_name}" project = "${var.project}" - instance = "${google_sql_database_instance.default.name}" + instance = "${google_sql_database_instance.master.name}" charset = "${var.db_charset}" collation = "${var.db_collation}" } resource "google_sql_user" "default" { - count = "${var.master_instance_name == "" ? 1 : 0}" - name = "${var.db_name}" + name = "${var.master_username}" project = "${var.project}" - instance = "${google_sql_database_instance.default.name}" - host = "${var.db_user_host}" - password = "${var.db_password}" + instance = "${google_sql_database_instance.master.name}" + host = "${var.master_host}" + password = "${var.master_password}" } \ No newline at end of file diff --git a/modules/mysql/outputs.tf b/modules/mysql/outputs.tf index 925b47e..26971f0 100644 --- a/modules/mysql/outputs.tf +++ b/modules/mysql/outputs.tf @@ -1,19 +1,29 @@ -output instance_name { +output "instance_name" { description = "The name of the database instance" - value = "${google_sql_database_instance.default.name}" + value = "${google_sql_database_instance.master.name}" } -output instance_address { +output "public_ip" { description = "The IPv4 address of the master database instance" - value = "${google_sql_database_instance.default.ip_address.0.ip_address}" + value = "${var.publicly_accessible ? google_sql_database_instance.master.ip_address.0.ip_address : ""}" } -output instance_address_time_to_retire { - description = "The time the master instance IP address will be reitred. RFC 3339 format." - value = "${google_sql_database_instance.default.ip_address.0.time_to_retire}" -} - -output self_link { +output "instance_self_link" { description = "Self link to the master instance" - value = "${google_sql_database_instance.default.self_link}" + value = "${google_sql_database_instance.master.self_link}" } + +output "db_name" { + description = "Name of the default database" + value = "${google_sql_database.default.name}" +} + +output "proxy_connection" { + value = "${var.project}:${var.region}:${google_sql_database_instance.master.name}" +} + +output "db_self_link" { + description = "Self link to the default database" + value = "${google_sql_database.default.self_link}" +} + diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index 49e3603..0f70f3f 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -12,21 +12,64 @@ variable "region" { } variable "name" { - description = "The name of the database instance." + description = "The name of the database instance. Note, after a name is used, it cannot be reused for up to one week. Use lowercase letters, numbers, and hyphens. Start with a letter." } variable "engine" { - description = "The engine version of the database, e.g. `MYSQL_5_7` or `POSTGRES_9_6`." + description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`." } -variable "master_instance_name" { - description = "The name of the instance that will act as the master in the replication setup. Note, this requires the master to have binary_log_enabled set, as well as existing backups." +# TODO: Depending on how the replicas are set up, tweak this. +#variable "master_instance_name" { +# description = "The name of the instance that will act as the master in the replication setup. Note, this requires the master to have binary_log_enabled set, as well as existing backups." +# default = "" +#} + +variable "machine_type" { + description = "The machine type for the instance. See this page for supported tiers and pricing: https://cloud.google.com/sql/pricing" +} + +variable "db_name" { + description = "Name of for your database of up to 8 alpha-numeric characters." default = "" } -variable "machine_type" { - description = "The machine tier (First Generation) or type (Second Generation). See this page for supported tiers and pricing: https://cloud.google.com/sql/pricing" - default = "db-f1-micro" +variable "master_username" { + description = "The username for the master user." +} + +variable "master_password" { + description = "The password for the master user." +} + +# --------------------------------------------------------------------------------------------------------------------- +# OPTIONAL PARAMETERS +# Generally, these values won't need to be changed. +# --------------------------------------------------------------------------------------------------------------------- + +variable "activation_policy" { + description = "This specifies when the instance should be active. Can be either `ALWAYS`, `NEVER` or `ON_DEMAND`." + default = "ALWAYS" +} + +variable "authorized_networks" { + description = "A list of authorized CIDR-formatted IP address ranges that can connect to this DB." + type = "list" + default = [] + # Example: + # + # authorized_networks = [ + # { + # name = "all-inbound" # optional + # value = "0.0.0.0/0" + # } + # ] +} + +variable "authorized_gae_applications" { + description = "A list of Google App Engine (GAE) project names that are allowed to access this instance." + type = "list" + default = [] } variable "availability_type" { @@ -34,45 +77,33 @@ variable "availability_type" { default = "ZONAL" } -variable "db_name" { - description = "Name of the default database to create" - default = "default" -} - variable "db_charset" { - description = "The charset for the default database" + description = "The charset for the default database." default = "" } variable "db_collation" { - description = "The collation for the default database. Example for MySQL databases: 'utf8_general_ci', and Postgres: 'en_US.UTF8'" + description = "The collation for the default database. Example for MySQL databases: 'utf8_general_ci'." default = "" } -variable "db_user" { - description = "The name of the default user" - default = "default" -} - -variable "db_user_host" { - description = "The host for the default user" - default = "%" -} - -variable "db_password" { - description = "The password for the default user." - default = "" -} - -variable "activation_policy" { - description = "This specifies when the instance should be active. Can be either `ALWAYS`, `NEVER` or `ON_DEMAND`." - default = "ALWAYS" -} - -variable "authorized_gae_applications" { - description = "A list of Google App Engine (GAE) project names that are allowed to access this instance." - type = "list" +variable "database_flags" { + description = "List of Cloud SQL flags that are applied to the database server" + type = "list" default = [] + + # Example: + # + # database_flags = [ + # { + # name = "auto_increment_increment" + # value = "10" + # }, + # { + # name = "auto_increment_offset" + # value = "5" + # }, + #] } variable "disk_autoresize" { @@ -86,60 +117,27 @@ variable "disk_size" { } variable "disk_type" { - description = "Second generation only. The type of data disk: `PD_SSD` or `PD_HDD`." - default = "PD_SSD" + description = "The type of storage to use. Must be one of `PD_SSD` or `PD_HDD`." + default = "PD_HDD" } -variable "pricing_plan" { - description = "First generation only. Pricing plan for this instance, can be one of `PER_USE` or `PACKAGE`." - default = "PER_USE" +variable "follow_gae_application" { + description = "A GAE application whose zone to remain in. Must be in the same region as this instance." + default = "" } -variable "replication_type" { - description = "Replication type for this instance, can be one of `ASYNCHRONOUS` or `SYNCHRONOUS`." - default = "SYNCHRONOUS" +variable "zone" { + description = "Preferred zone for the instance." + default = "" } -variable "flags" { - description = "List of Cloud SQL flags that are applied to the database server" - default = [] - type = "list" +variable "master_host" { + description = "The host for the default user" + default = "%" } -# IGNORE EVERYTHING BELOW - -variable backup_configuration { - description = "The backup_configuration settings subblock for the database setings" - type = "map" - default = {} -} - -variable ip_configuration { - description = "The ip_configuration settings subblock" - type = "list" - default = [{}] -} - -variable location_preference { - description = "The location_preference settings subblock" - type = "list" - default = [] -} - -variable maintenance_window { - description = "The maintenance_window settings subblock" - type = "list" - default = [] -} - -variable replica_configuration { - description = "The optional replica_configuration block for the database instance" - type = "list" - default = [] -} - -# --------------------------------------------------------------------------------------------------------------------- -# OPTIONAL PARAMETERS -# Generally, these values won't need to be changed. -# --------------------------------------------------------------------------------------------------------------------- -# TODO: \ No newline at end of file +# In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to AWS, this needs to remain open to the internet. +variable "publicly_accessible" { + description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." + default = false +} \ No newline at end of file From ff0a275ddc2d0a7e954212e66ecb758972fad92a Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 17:44:24 +0200 Subject: [PATCH 11/34] Add first example --- examples/cloud-sql-mysql/main.tf | 88 ++++++++++----------------- examples/cloud-sql-mysql/outputs.tf | 29 +++++++++ examples/cloud-sql-mysql/variables.tf | 46 ++++++++++++++ 3 files changed, 106 insertions(+), 57 deletions(-) create mode 100644 examples/cloud-sql-mysql/outputs.tf create mode 100644 examples/cloud-sql-mysql/variables.tf diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 29bbd7e..504961a 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -9,70 +9,44 @@ terraform { required_version = ">= 0.10.3" } -variable "region" { - default = "europe-north1" -} +module "mysql" { + source = "../../modules/mysql" -variable "project" { - default = "dev-sandbox-228703" -} - - -variable "zone" { - default = "europe-north1-a" -} - -variable "mysql_version" { - default = "MYSQL_5_6" -} - -resource "random_id" "name" { - byte_length = 2 -} - -resource "google_compute_network" "private_network" { - provider = "google-beta" - name = "private-network" -} - -resource "google_compute_global_address" "private_ip_address" { - provider = "google-beta" - name = "private-ip-address" - purpose = "VPC_PEERING" - address_type = "INTERNAL" - prefix_length = 16 - network = "${google_compute_network.private_network.self_link}" -} - -resource "google_service_networking_connection" "private_vpc_connection" { - provider = "google-beta" - network = "${google_compute_network.private_network.self_link}" - service = "servicenetworking.googleapis.com" - reserved_peering_ranges = ["${google_compute_global_address.private_ip_address.name}"] -} - - -module "mysql-db" { - source = "../../modules/cloud-sql" - name = "example-mysql-${random_id.name.hex}" - region = "${var.region}" - engine = "${var.mysql_version}" project = "${var.project}" - machine_type = "db-f1-micro" + region = "${var.region}" + name = "${var.name}" + db_name = "${var.db_name}" - ip_configuration = [ + engine = "${var.mysql_version}" + machine_type = "${var.machine_type}" + + master_password = "${var.master_password}" + master_username = "${var.master_username}" + + master_host = "%" + publicly_accessible = "${var.publicly_accessible}" + + # Never do this in production! + # We're setting permissive network rules to make + # it easier to test the instance + authorized_networks = [ { - ipv4_enabled = "true" - private_network = "${google_compute_network.private_network.self_link}" + name = "allow-all-inbound", + value = "0.0.0.0/0" } ] - # https://cloud.google.com/sql/docs/mysql/flags - flags = [ + # Set auto-increment flags to test the + # feature in during automated testing + database_flags = [ + { + name = "auto_increment_increment" + value = "10" + }, + { + name = "auto_increment_offset" + value = "5" + } ] } -output "mysql_conn" { - value = "${var.project}:${var.region}:${module.mysql-db.instance_name}" -} - diff --git a/examples/cloud-sql-mysql/outputs.tf b/examples/cloud-sql-mysql/outputs.tf new file mode 100644 index 0000000..7565f1a --- /dev/null +++ b/examples/cloud-sql-mysql/outputs.tf @@ -0,0 +1,29 @@ +output "instance_name" { + description = "The name of the database instance" + value = "${module.mysql.instance_name}" +} + +output "public_ip" { + description = "The IPv4 address of the master database instance" + value = "${module.mysql.public_ip}" +} + +output "instance_self_link" { + description = "Self link to the master instance" + value = "${module.mysql.instance_self_link}" +} + +output "db_name" { + description = "Name of the default database" + value = "${module.mysql.db_name}" +} + +output "proxy_connection" { + value = "${module.mysql.proxy_connection}" +} + +output "db_self_link" { + description = "Self link to the default database" + value = "${module.mysql.db_self_link}" +} + diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf new file mode 100644 index 0000000..054a089 --- /dev/null +++ b/examples/cloud-sql-mysql/variables.tf @@ -0,0 +1,46 @@ +# --------------------------------------------------------------------------------------------------------------------- +# REQUIRED PARAMETERS +# These variables are expected to be passed in by the operator +# --------------------------------------------------------------------------------------------------------------------- + +variable "project" { + description = "The project ID to host the database in." +} + +variable "region" { + description = "The region to host the database in." +} + +# Note, after a name is used, it cannot be reused for up to one week. +variable "name" { + description = "The name of the database instance. Use lowercase letters, numbers, and hyphens. Start with a letter." +} + +variable "master_username" { + description = "The username for the master user." +} + +variable "master_password" { + description = "The password for the master user." +} + +# --------------------------------------------------------------------------------------------------------------------- +# OPTIONAL PARAMETERS +# Generally, these values won't need to be changed. +# --------------------------------------------------------------------------------------------------------------------- +variable "publicly_accessible" { + default = "true" +} + +variable "mysql_version" { + description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`." + default = "MYSQL_5_7" +} + +variable "machine_type" { + default = "db-f1-micro" +} + +variable "db_name" { + default = "default" +} From ea66b2fe48d28c31c8e32c33a414878417cf3f82 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 17:44:49 +0200 Subject: [PATCH 12/34] Add proper test case. --- test/Gopkg.lock | 379 ++++++++++++++++++++++++++- test/Gopkg.toml | 7 +- test/example_cloud_sql_mysql_test.go | 149 ++++++++++- test/test_util.go | 16 ++ 4 files changed, 543 insertions(+), 8 deletions(-) create mode 100644 test/test_util.go diff --git a/test/Gopkg.lock b/test/Gopkg.lock index 3c7bff3..f1061a0 100644 --- a/test/Gopkg.lock +++ b/test/Gopkg.lock @@ -1,6 +1,91 @@ # This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'. +[[projects]] + digest = "1:8b95956b70e181b19025c7ba3578fdfd8efbec4ce916490700488afb9218972c" + name = "cloud.google.com/go" + packages = [ + "compute/metadata", + "iam", + "internal", + "internal/optional", + "internal/trace", + "internal/version", + "storage", + ] + pruneopts = "" + revision = "64a2037ec6be8a4b0c1d1f706ed35b428b989239" + version = "v0.26.0" + +[[projects]] + digest = "1:4b6ba994aa18b6c6a871cd700a66525a182f2c0cd78022e25e01508fc8559122" + name = "github.com/aws/aws-sdk-go" + packages = [ + "aws", + "aws/awserr", + "aws/awsutil", + "aws/client", + "aws/client/metadata", + "aws/corehandlers", + "aws/credentials", + "aws/credentials/ec2rolecreds", + "aws/credentials/endpointcreds", + "aws/credentials/processcreds", + "aws/credentials/stscreds", + "aws/csm", + "aws/defaults", + "aws/ec2metadata", + "aws/endpoints", + "aws/request", + "aws/session", + "aws/signer/v4", + "internal/ini", + "internal/s3err", + "internal/sdkio", + "internal/sdkrand", + "internal/sdkuri", + "internal/shareddefaults", + "private/protocol", + "private/protocol/ec2query", + "private/protocol/eventstream", + "private/protocol/eventstream/eventstreamapi", + "private/protocol/json/jsonutil", + "private/protocol/jsonrpc", + "private/protocol/query", + "private/protocol/query/queryutil", + "private/protocol/rest", + "private/protocol/restxml", + "private/protocol/xml/xmlutil", + "service/acm", + "service/autoscaling", + "service/cloudwatchlogs", + "service/ec2", + "service/iam", + "service/kms", + "service/rds", + "service/s3", + "service/s3/s3iface", + "service/s3/s3manager", + "service/sns", + "service/sqs", + "service/sts", + ] + pruneopts = "" + revision = "949cbce4e4443b72f6da12adbeb5d416dd506fbe" + version = "v1.16.27" + +[[projects]] + digest = "1:b529f4bf748979caa18b599d40d13e8b6e591a74b340f315ce4f95e119c288c2" + name = "github.com/boombuler/barcode" + packages = [ + ".", + "qr", + "utils", + ] + pruneopts = "" + revision = "3cfea5ab600ae37946be2b763b8ec2c1cf2d272d" + version = "v1.0.0" + [[projects]] digest = "1:0deddd908b6b4b768cfc272c16ee61e7088a60f7fe2f06c547bd3d8e1f8b8e77" name = "github.com/davecgh/go-spew" @@ -9,6 +94,80 @@ revision = "8991bc29aa16c548c550c7ff78260e27b9ab7c73" version = "v1.1.1" +[[projects]] + digest = "1:e692d16fdfbddb94e9e4886aaf6c08bdbae5cb4ac80651445de9181b371c6e46" + name = "github.com/go-sql-driver/mysql" + packages = ["."] + pruneopts = "" + revision = "72cd26f257d44c1114970e19afddcd812016007e" + source = "git@github.com:go-sql-driver/mysql" + version = "v1.4.1" + +[[projects]] + digest = "1:3dd078fda7500c341bc26cfbc6c6a34614f295a2457149fc1045cab767cbcf18" + name = "github.com/golang/protobuf" + packages = [ + "proto", + "protoc-gen-go/descriptor", + "ptypes", + "ptypes/any", + "ptypes/duration", + "ptypes/timestamp", + ] + pruneopts = "" + revision = "aa810b61a9c79d51363740d207bb46cf8e620ed5" + version = "v1.2.0" + +[[projects]] + digest = "1:c1d7e883c50a26ea34019320d8ae40fad86c9e5d56e63a1ba2cb618cef43e986" + name = "github.com/google/uuid" + packages = ["."] + pruneopts = "" + revision = "064e2069ce9c359c118179501254f67d7d37ba24" + version = "0.2" + +[[projects]] + digest = "1:55c1b46a80db2baf4d762c1d0b5cb4946e46125baa02b8959310abab15b54aee" + name = "github.com/googleapis/gax-go" + packages = [ + ".", + "v2", + ] + pruneopts = "" + revision = "c8a15bac9b9fe955bd9f900272f9a306465d28cf" + version = "v2.0.3" + +[[projects]] + digest = "1:4b6c36dd91add683751cb4670cc26d63c7e7db259695ffd723670c666797c62c" + name = "github.com/gruntwork-io/terratest" + packages = [ + "modules/aws", + "modules/collections", + "modules/customerrors", + "modules/environment", + "modules/files", + "modules/gcp", + "modules/logger", + "modules/packer", + "modules/random", + "modules/retry", + "modules/shell", + "modules/ssh", + "modules/terraform", + "modules/test-structure", + ] + pruneopts = "" + revision = "425f7f1c2c72c607142c67aee02d1c14f874cf0a" + source = "git@github.com:gruntwork-io/terratest" + version = "v0.13.24" + +[[projects]] + digest = "1:13fe471d0ed891e8544eddfeeb0471fd3c9f2015609a1c000aefdedf52a19d40" + name = "github.com/jmespath/go-jmespath" + packages = ["."] + pruneopts = "" + revision = "c2b33e84" + [[projects]] digest = "1:256484dbbcd271f9ecebc6795b2df8cad4c458dd0f5fd82a8c2fa0c29f233411" name = "github.com/pmezard/go-difflib" @@ -17,17 +176,233 @@ revision = "792786c7400a136282c1664665ae0a8db921c6c2" version = "v1.0.0" +[[projects]] + digest = "1:d8c398c75a666d415196ba289402c3f52b595d8cede451a9e57278354e327a93" + name = "github.com/pquerna/otp" + packages = [ + ".", + "hotp", + "totp", + ] + pruneopts = "" + revision = "be78767b3e392ce45ea73444451022a6fc32ad0d" + version = "v1.1.0" + [[projects]] digest = "1:381bcbeb112a51493d9d998bbba207a529c73dbb49b3fd789e48c63fac1f192c" name = "github.com/stretchr/testify" - packages = ["assert"] + packages = [ + "assert", + "require", + ] pruneopts = "" revision = "ffdc059bfe9ce6a4e144ba849dbedead332c6053" version = "v1.3.0" +[[projects]] + digest = "1:b1bb9332f6cb7821a730e1681819b2813340eba5e873f05381815a7c6807d172" + name = "go.opencensus.io" + packages = [ + ".", + "exemplar", + "internal", + "internal/tagencoding", + "plugin/ochttp", + "plugin/ochttp/propagation/b3", + "stats", + "stats/internal", + "stats/view", + "tag", + "trace", + "trace/internal", + "trace/propagation", + "trace/tracestate", + ] + pruneopts = "" + revision = "2b5032d79456124f42db6b7eb19ac6c155449dc2" + version = "v0.19.0" + +[[projects]] + branch = "master" + digest = "1:cbc1ebc01ec2ceb2c3cc7b9e33357dbc3eff173335f02d9f01603f14102602a7" + name = "golang.org/x/crypto" + packages = [ + "curve25519", + "ed25519", + "ed25519/internal/edwards25519", + "internal/chacha20", + "internal/subtle", + "poly1305", + "ssh", + "ssh/agent", + ] + pruneopts = "" + revision = "b8fe1690c61389d7d2a8074a507d1d40c5d30448" + +[[projects]] + branch = "master" + digest = "1:1e9704e5379e68ac473c28aeb3f7e7cd4036ae8a246bf0285b5ebdbb8e0cfacf" + name = "golang.org/x/net" + packages = [ + "context", + "context/ctxhttp", + "http/httpguts", + "http2", + "http2/hpack", + "idna", + "internal/timeseries", + "trace", + ] + pruneopts = "" + revision = "d26f9f9a57f3fab6a695bec0d84433c2c50f8bbf" + +[[projects]] + digest = "1:b697592485cb412be4188c08ca0beed9aab87f36b86418e21acc4a3998f63734" + name = "golang.org/x/oauth2" + packages = [ + ".", + "google", + "internal", + "jws", + "jwt", + ] + pruneopts = "" + revision = "d2e6202438beef2727060aa7cabdd924d92ebfd9" + +[[projects]] + branch = "master" + digest = "1:489610147902fe0c7229218c749bb25a8a9ecce0d726ae4f8662517319f32554" + name = "golang.org/x/sys" + packages = ["unix"] + pruneopts = "" + revision = "41f3e6584952bb034a481797859f6ab34b6803bd" + +[[projects]] + digest = "1:5acd3512b047305d49e8763eef7ba423901e85d5dd2fd1e71778a0ea8de10bd4" + name = "golang.org/x/text" + packages = [ + "collate", + "collate/build", + "internal/colltab", + "internal/gen", + "internal/tag", + "internal/triegen", + "internal/ucd", + "language", + "secure/bidirule", + "transform", + "unicode/bidi", + "unicode/cldr", + "unicode/norm", + "unicode/rangetable", + ] + pruneopts = "" + revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0" + version = "v0.3.0" + +[[projects]] + branch = "master" + digest = "1:e53b3650fadc76377e90fa596a939233cff79f8606cae71e0b6c73e5e6829ecd" + name = "google.golang.org/api" + packages = [ + "compute/v1", + "gensupport", + "googleapi", + "googleapi/internal/uritemplates", + "googleapi/transport", + "internal", + "iterator", + "option", + "storage/v1", + "transport/http", + "transport/http/internal/propagation", + ] + pruneopts = "" + revision = "697424a9b4245c525a11d14a2460c76f35f8db55" + +[[projects]] + digest = "1:bc09e719c4e2a15d17163f5272d9a3131c45d77542b7fdc53ff518815bc19ab3" + name = "google.golang.org/appengine" + packages = [ + ".", + "cloudsql", + "internal", + "internal/app_identity", + "internal/base", + "internal/datastore", + "internal/log", + "internal/modules", + "internal/remote_api", + "internal/urlfetch", + "urlfetch", + ] + pruneopts = "" + revision = "e9657d882bb81064595ca3b56cbe2546bbabf7b1" + version = "v1.4.0" + +[[projects]] + branch = "master" + digest = "1:91f75dc679abcf04b29b064a5b9c40a0b0561c86c322010fbf6fc08040cc48bc" + name = "google.golang.org/genproto" + packages = [ + "googleapis/api/annotations", + "googleapis/iam/v1", + "googleapis/rpc/code", + "googleapis/rpc/status", + ] + pruneopts = "" + revision = "4b09977fb92221987e99d190c8f88f2c92727a29" + +[[projects]] + digest = "1:39d4d828b87d58d114fdc211f0638f32dcae84019fe17d6b48f9b697f4b60213" + name = "google.golang.org/grpc" + packages = [ + ".", + "balancer", + "balancer/base", + "balancer/roundrobin", + "binarylog/grpc_binarylog_v1", + "codes", + "connectivity", + "credentials", + "credentials/internal", + "encoding", + "encoding/proto", + "grpclog", + "internal", + "internal/backoff", + "internal/binarylog", + "internal/channelz", + "internal/envconfig", + "internal/grpcrand", + "internal/grpcsync", + "internal/syscall", + "internal/transport", + "keepalive", + "metadata", + "naming", + "peer", + "resolver", + "resolver/dns", + "resolver/passthrough", + "stats", + "status", + "tap", + ] + pruneopts = "" + revision = "a02b0774206b209466313a0b525d2c738fe407eb" + version = "v1.18.0" + [solve-meta] analyzer-name = "dep" analyzer-version = 1 - input-imports = ["github.com/stretchr/testify/assert"] + input-imports = [ + "github.com/go-sql-driver/mysql", + "github.com/gruntwork-io/terratest/modules/gcp", + "github.com/gruntwork-io/terratest/modules/random", + "github.com/gruntwork-io/terratest/modules/terraform", + "github.com/gruntwork-io/terratest/modules/test-structure", + "github.com/stretchr/testify/assert", + ] solver-name = "gps-cdcl" solver-version = 1 diff --git a/test/Gopkg.toml b/test/Gopkg.toml index 7d7581c..75c9934 100644 --- a/test/Gopkg.toml +++ b/test/Gopkg.toml @@ -22,4 +22,9 @@ [[constraint]] name = "github.com/gruntwork-io/terratest" source = "git@github.com:gruntwork-io/terratest" - version = "0.13.22" + version = "0.13.24" + +[[constraint]] + name = "github.com/go-sql-driver/mysql" + source = "git@github.com:go-sql-driver/mysql" + version = "1.4.1" diff --git a/test/example_cloud_sql_mysql_test.go b/test/example_cloud_sql_mysql_test.go index 5310c38..b1da0f9 100644 --- a/test/example_cloud_sql_mysql_test.go +++ b/test/example_cloud_sql_mysql_test.go @@ -1,16 +1,155 @@ package test import ( - "testing" - + "database/sql" + "fmt" + _ "github.com/go-sql-driver/mysql" + "github.com/gruntwork-io/terratest/modules/gcp" + "github.com/gruntwork-io/terratest/modules/logger" + "github.com/gruntwork-io/terratest/modules/random" + "github.com/gruntwork-io/terratest/modules/terraform" + "github.com/gruntwork-io/terratest/modules/test-structure" "github.com/stretchr/testify/assert" + "log" + "path/filepath" + "strings" + "testing" ) -// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in -// the templates +const DB_NAME = "testdb" +const DB_USER = "testuser" +const DB_PASS = "testpassword" + // TODO: try to actually connect to the RDS DBs and check they are working func TestCloudSQLMySql(t *testing.T) { t.Parallel() - assert.Equal(t, "3306", "3306") + //os.Setenv("SKIP_bootstrap", "true") + //os.Setenv("SKIP_deploy", "true") + //os.Setenv("SKIP_validate_outputs", "true") + //os.Setenv("SKIP_sql_tests", "true") + //os.Setenv("SKIP_teardown", "true") + + _examplesDir := test_structure.CopyTerraformFolderToTemp(t, "../", "examples") + exampleDir := filepath.Join(_examplesDir, "cloud-sql-mysql") + rootDir := filepath.Join(_examplesDir, "../") + + test_structure.RunTestStage(t, "bootstrap", func() { + uniqueId := strings.ToLower(random.UniqueId()) + instanceName := fmt.Sprintf("mysql-test-%s", uniqueId) + projectId := gcp.GetGoogleProjectIDFromEnvVar(t) + region := getRandomRegion(t, projectId) + + test_structure.SaveString(t, exampleDir, "instance-name", instanceName) + test_structure.SaveString(t, exampleDir, "region", region) + test_structure.SaveString(t, exampleDir, "project-id", projectId) + }) + + // At the end of the test, run `terraform destroy` to clean up any resources that were created + defer test_structure.RunTestStage(t, "teardown", func() { + terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir) + terraform.Destroy(t, terraformOptions) + }) + + test_structure.RunTestStage(t, "deploy", func() { + region := test_structure.LoadString(t, exampleDir, "region") + projectId := test_structure.LoadString(t, exampleDir, "project-id") + instanceName := test_structure.LoadString(t, exampleDir, "instance-name") + terraformOptions := createTerratestOptionsForMySql(projectId, region, instanceName, rootDir) + test_structure.SaveTerraformOptions(t, exampleDir, terraformOptions) + + terraform.InitAndApply(t, terraformOptions) + }) + + test_structure.RunTestStage(t, "validate_outputs", func() { + terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir) + instanceName := test_structure.LoadString(t, exampleDir, "instance-name") + + region := test_structure.LoadString(t, exampleDir, "region") + projectId := test_structure.LoadString(t, exampleDir, "project-id") + + expectedDBConn := fmt.Sprintf("%s:%s:%s", projectId, region, instanceName) + + assert.Equal(t, instanceName, terraform.Output(t, terraformOptions, "instance_name")) + assert.Equal(t, "testdb", terraform.Output(t, terraformOptions, "db_name")) + assert.Equal(t, expectedDBConn, terraform.Output(t, terraformOptions, "proxy_connection")) + }) + + test_structure.RunTestStage(t, "sql_tests", func() { + terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir) + + publicIp := terraform.Output(t, terraformOptions, "public_ip") + + connectionString := fmt.Sprintf("%s:%s@tcp(%s:3306)/%s", DB_USER, DB_PASS, publicIp, DB_NAME) + + // Does not actually open up the connection - just returns a DB ref + logger.Logf(t, "Connecting to: %s", publicIp) + db, err := sql.Open("mysql", + connectionString) + + if err != nil { + t.Fatalf("Failed to open DB connection: %v", err) + } + + // Make sure we clean up properly + defer db.Close() + + // Run ping to actually test the connection + logger.Log(t, "Ping the DB") + if err = db.Ping(); err != nil { + t.Fatalf("Failed to ping DB: %v", err) + } + + // Create table if not exists + logger.Logf(t, "Create table: %s", MYSQL_CREATE_TEST_TABLE_WITH_AUTO_INCREMENT_STATEMENT) + if _, err = db.Exec(MYSQL_CREATE_TEST_TABLE_WITH_AUTO_INCREMENT_STATEMENT); err != nil { + t.Fatalf("Failed to create table: %v", err) + } + + // Clean up + logger.Logf(t, "Empty table: %s", MYSQL_EMPTY_TEST_TABLE_STATEMENT) + if _, err = db.Exec(MYSQL_EMPTY_TEST_TABLE_STATEMENT); err != nil { + t.Fatalf("Failed to clean up table: %v", err) + } + + // Insert data to check that our auto-increment flags worked + logger.Logf(t, "Insert data: %s", MYSQL_INSERT_TEST_ROW) + stmt, err := db.Prepare(MYSQL_INSERT_TEST_ROW) + if err != nil { + t.Fatalf("Failed to prepare statement: %v", err) + } + + // Execute the statement + res, err := stmt.Exec("Grunt") + if err != nil { + t.Fatalf("Failed to execute statement: %v", err) + } + + // Get the last insert id + lastId, err := res.LastInsertId() + if err != nil { + log.Fatal(err) + } + // Since we set the auto increment to 5, modulus should always be 0 + assert.Equal(t, int64(0), int64(lastId%5)) + }) +} + +func createTerratestOptionsForMySql(projectId string, region string, instanceName string, repoPath string) *terraform.Options { + + terratestOptions := &terraform.Options{ + // The path to where your Terraform code is located + TerraformDir: filepath.Join(repoPath, "examples", "cloud-sql-mysql"), + Vars: map[string]interface{}{ + "region": region, + "project": projectId, + "name": instanceName, + "mysql_version": "MYSQL_5_7", + "db_name": DB_NAME, + "master_username": DB_USER, + "master_password": DB_PASS, + }, + } + + return terratestOptions } diff --git a/test/test_util.go b/test/test_util.go new file mode 100644 index 0000000..c6b482c --- /dev/null +++ b/test/test_util.go @@ -0,0 +1,16 @@ +package test + +import ( + "github.com/gruntwork-io/terratest/modules/gcp" + "testing" +) + +const MYSQL_CREATE_TEST_TABLE_WITH_AUTO_INCREMENT_STATEMENT = "CREATE TABLE IF NOT EXISTS test (id int NOT NULL AUTO_INCREMENT, name varchar(10) NOT NULL, PRIMARY KEY (ID))" +const MYSQL_EMPTY_TEST_TABLE_STATEMENT = "DELETE FROM test" +const MYSQL_INSERT_TEST_ROW = "INSERT INTO test(name) VALUES(?)" + +func getRandomRegion(t *testing.T, projectID string) string { + //approvedRegions := []string{"europe-north1", "europe-west1", "europe-west2", "europe-west3", "us-central1", "us-east1", "us-west1"} + approvedRegions := []string{"europe-north1"} + return gcp.GetRandomRegion(t, projectID, approvedRegions, []string{}) +} From 0b6dc4d46e332bc8cd2ffc8283d56f812cecb3b8 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 17:54:59 +0200 Subject: [PATCH 13/34] Documentation updates --- modules/mysql/variables.tf | 2 +- test/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index 0f70f3f..9695e21 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -136,7 +136,7 @@ variable "master_host" { default = "%" } -# In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to AWS, this needs to remain open to the internet. +# In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to GCP, this needs to remain open to the internet. variable "publicly_accessible" { description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." default = false diff --git a/test/README.md b/test/README.md index 3f0f0d9..b428e57 100644 --- a/test/README.md +++ b/test/README.md @@ -8,7 +8,7 @@ works as expected using a helper library called [Terratest](https://github.com/g ## WARNING WARNING WARNING -**Note #1**: Many of these tests create real resources in an AWS account and then try to clean those resources up at +**Note #1**: Many of these tests create real resources in a GCP project and then try to clean those resources up at the end of a test run. That means these tests may cost you money to run! When adding tests, please be considerate of the resources you create and take extra care to clean everything up when you're done! From 47beb8510aaac65fd0d3ec6e14d490808ce2eb98 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 19:28:08 +0200 Subject: [PATCH 14/34] Naming changes based on PR review. --- examples/cloud-sql-mysql/outputs.tf | 8 ++++---- modules/mysql/outputs.tf | 6 ++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/examples/cloud-sql-mysql/outputs.tf b/examples/cloud-sql-mysql/outputs.tf index 7565f1a..e5a70af 100644 --- a/examples/cloud-sql-mysql/outputs.tf +++ b/examples/cloud-sql-mysql/outputs.tf @@ -8,9 +8,9 @@ output "public_ip" { value = "${module.mysql.public_ip}" } -output "instance_self_link" { +output "instance" { description = "Self link to the master instance" - value = "${module.mysql.instance_self_link}" + value = "${module.mysql.instance}" } output "db_name" { @@ -22,8 +22,8 @@ output "proxy_connection" { value = "${module.mysql.proxy_connection}" } -output "db_self_link" { +output "db" { description = "Self link to the default database" - value = "${module.mysql.db_self_link}" + value = "${module.mysql.db}" } diff --git a/modules/mysql/outputs.tf b/modules/mysql/outputs.tf index 26971f0..23f64fc 100644 --- a/modules/mysql/outputs.tf +++ b/modules/mysql/outputs.tf @@ -8,7 +8,7 @@ output "public_ip" { value = "${var.publicly_accessible ? google_sql_database_instance.master.ip_address.0.ip_address : ""}" } -output "instance_self_link" { +output "instance" { description = "Self link to the master instance" value = "${google_sql_database_instance.master.self_link}" } @@ -18,11 +18,13 @@ output "db_name" { value = "${google_sql_database.default.name}" } + output "proxy_connection" { + description = "Instance path for connecting with Cloud SQL Proxy. Read more at https://cloud.google.com/sql/docs/mysql/sql-proxy" value = "${var.project}:${var.region}:${google_sql_database_instance.master.name}" } -output "db_self_link" { +output "db" { description = "Self link to the default database" value = "${google_sql_database.default.self_link}" } From 54f260786aa5655ca8a133fbd7bde82a76361bda Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 19:35:28 +0200 Subject: [PATCH 15/34] Naming changes based on PR review. --- examples/cloud-sql-mysql/main.tf | 2 +- examples/cloud-sql-mysql/variables.tf | 7 ++++--- modules/mysql/README.md | 4 ++-- modules/mysql/main.tf | 2 +- modules/mysql/outputs.tf | 2 +- modules/mysql/variables.tf | 2 +- 6 files changed, 10 insertions(+), 9 deletions(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 504961a..27c2ab9 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -24,7 +24,7 @@ module "mysql" { master_username = "${var.master_username}" master_host = "%" - publicly_accessible = "${var.publicly_accessible}" + enable_public_internet_access = "${var.enable_public_internet_access}" # Never do this in production! # We're setting permissive network rules to make diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 054a089..4754e98 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -28,10 +28,11 @@ variable "master_password" { # OPTIONAL PARAMETERS # Generally, these values won't need to be changed. # --------------------------------------------------------------------------------------------------------------------- -variable "publicly_accessible" { - default = "true" +# In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to GCP, this needs to remain open to the internet. +variable "enable_public_internet_access" { + description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." + default = true } - variable "mysql_version" { description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`." default = "MYSQL_5_7" diff --git a/modules/mysql/README.md b/modules/mysql/README.md index eaad1fe..cd7df48 100644 --- a/modules/mysql/README.md +++ b/modules/mysql/README.md @@ -31,8 +31,8 @@ variables](https://www.terraform.io/intro/getting-started/outputs.html): 1. TODO: **Private IP** `private_ip`: The public endpoint for the cluster. 1. **Public IP** `public_ip`: The public endpoint for the cluster. -1. **Connection name** `connection_name`: The private endpoint for the cluster. -1. **Replica endpoints** `replica_endpoints`: A comma-separated list of all DB instance URLs in the cluster, including the primary and all +1. **Proxy connection** `proxy_connection`: "Instance path for connecting with Cloud SQL Proxy. Read more at https://cloud.google.com/sql/docs/mysql/sql-proxy. +1. TODO: **Replica endpoints** `replica_endpoints`: A comma-separated list of all DB instance URLs in the cluster, including the primary and all read replicas. Use these URLs for reads (see "How do you scale this DB?" below). diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index 4334b91..76df989 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -25,7 +25,7 @@ resource "google_sql_database_instance" "master" { ip_configuration { authorized_networks = ["${var.authorized_networks}"], - ipv4_enabled = "${var.publicly_accessible}" + ipv4_enabled = "${var.enable_public_internet_access}" } location_preference { diff --git a/modules/mysql/outputs.tf b/modules/mysql/outputs.tf index 23f64fc..024c945 100644 --- a/modules/mysql/outputs.tf +++ b/modules/mysql/outputs.tf @@ -5,7 +5,7 @@ output "instance_name" { output "public_ip" { description = "The IPv4 address of the master database instance" - value = "${var.publicly_accessible ? google_sql_database_instance.master.ip_address.0.ip_address : ""}" + value = "${var.enable_public_internet_access ? google_sql_database_instance.master.ip_address.0.ip_address : ""}" } output "instance" { diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index 9695e21..dc4be6c 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -137,7 +137,7 @@ variable "master_host" { } # In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to GCP, this needs to remain open to the internet. -variable "publicly_accessible" { +variable "enable_public_internet_access" { description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." default = false } \ No newline at end of file From 065249b9649b2b65908c0c41ac351ec45d2a66cc Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 19:36:09 +0200 Subject: [PATCH 16/34] Delete sandbox --- examples/sandbox/main.tf | 45 ---------------------------------------- 1 file changed, 45 deletions(-) delete mode 100644 examples/sandbox/main.tf diff --git a/examples/sandbox/main.tf b/examples/sandbox/main.tf deleted file mode 100644 index 2551ea2..0000000 --- a/examples/sandbox/main.tf +++ /dev/null @@ -1,45 +0,0 @@ -provider "google-beta" { - region = "${var.region}" -} - -variable "region" { - default = "europe-north1" -} - -variable "project" { - default = "petri-sandbox" -} - -variable "endpoints" { - type = "list" - default = ["192.168.11.1", "192.168.11.2"] -} - -data "template_file" "single_ip" { - count = "${length(var.endpoints)}" - template = < Date: Tue, 5 Feb 2019 19:37:29 +0200 Subject: [PATCH 17/34] Update modules/mysql/main.tf Add beta provider Co-Authored-By: autero1 --- modules/mysql/main.tf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index 76df989..b4e0060 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -12,6 +12,7 @@ # ------------------------------------------------------------------------------ resource "google_sql_database_instance" "master" { + provider = "google-beta" name = "${var.name}" project = "${var.project}" region = "${var.region}" @@ -58,4 +59,4 @@ resource "google_sql_user" "default" { instance = "${google_sql_database_instance.master.name}" host = "${var.master_host}" password = "${var.master_password}" -} \ No newline at end of file +} From ba6ec1296dd2b3ca349aee4de69d425c286c73e8 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 19:45:46 +0200 Subject: [PATCH 18/34] Doc updates --- modules/mysql/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/mysql/README.md b/modules/mysql/README.md index cd7df48..919b4e7 100644 --- a/modules/mysql/README.md +++ b/modules/mysql/README.md @@ -18,7 +18,7 @@ and encryption. For a list of all available variables and their descriptions, se ## How do you connect to the database? -**Cloud SQL instances are created in a producer network (a VPC network internal to Google). They are not created in your VPC network.** +**Cloud SQL instances are created in a producer network (a VPC network internal to Google). They are not created in your VPC network. See https://cloud.google.com/sql/docs/mysql/private-ip** You can use both [public IP](https://cloud.google.com/sql/docs/mysql/connect-admin-ip) and [private IP](https://cloud.google.com/sql/docs/mysql/private-ip) to connect to a Cloud SQL instance. Neither connection method affects the other; you must protect the public IP connection whether the instance is configured to use private IP or not. From 9d32dc7de123dfaa5580f33c2a412a64e2edffef Mon Sep 17 00:00:00 2001 From: Yoriyasu Yano <430092+yorinasub17@users.noreply.github.com> Date: Tue, 5 Feb 2019 20:01:12 +0200 Subject: [PATCH 19/34] Update examples/cloud-sql-mysql/main.tf Add github address Co-Authored-By: autero1 --- examples/cloud-sql-mysql/main.tf | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 27c2ab9..2d629b7 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -10,6 +10,9 @@ terraform { } module "mysql" { + # When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you + # to a specific version of the modules, such as the following example: + # source = "git::git@github.com:gruntwork-io/terraform-google-sql.git//modules/mysql?ref=v0.1.0" source = "../../modules/mysql" project = "${var.project}" From 39770cbbaaf8e54fdb07ab13eb6422f1b80b4212 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Tue, 5 Feb 2019 23:51:46 +0200 Subject: [PATCH 20/34] Updates based on PR feedback --- examples/cloud-sql-mysql/main.tf | 34 +++++++----- examples/cloud-sql-mysql/variables.tf | 23 +++++--- modules/mysql/main.tf | 6 +- modules/mysql/variables.tf | 6 +- test/example_cloud_sql_mysql_test.go | 79 +++++++++++++++------------ 5 files changed, 86 insertions(+), 62 deletions(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 27c2ab9..422c671 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -9,22 +9,30 @@ terraform { required_version = ">= 0.10.3" } +resource "random_id" "name" { + byte_length = 2 +} + +locals { + # If name_override is specified, use that - otherwise use the name_prefix with a random string + instance_name = "${length(var.name_override) == 0 ? format("%s-%s", var.name_prefix, random_id.name.hex) : var.name_override}" +} + module "mysql" { - source = "../../modules/mysql" + source = "../../modules/mysql" - project = "${var.project}" - region = "${var.region}" - name = "${var.name}" - db_name = "${var.db_name}" + project = "${var.project}" + region = "${var.region}" + name = "${local.instance_name}" + db_name = "${var.db_name}" - engine = "${var.mysql_version}" - machine_type = "${var.machine_type}" + engine = "${var.mysql_version}" + machine_type = "${var.machine_type}" - master_password = "${var.master_password}" - master_username = "${var.master_username}" - - master_host = "%" - enable_public_internet_access = "${var.enable_public_internet_access}" + master_user_password = "${var.master_user_password}" + master_user_name = "${var.master_user_name}" + master_user_host = "%" + enable_public_internet_access = "${var.enable_public_internet_access}" # Never do this in production! # We're setting permissive network rules to make @@ -41,7 +49,7 @@ module "mysql" { database_flags = [ { name = "auto_increment_increment" - value = "10" + value = "5" }, { name = "auto_increment_offset" diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 4754e98..cf14b7a 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -11,16 +11,16 @@ variable "region" { description = "The region to host the database in." } -# Note, after a name is used, it cannot be reused for up to one week. -variable "name" { - description = "The name of the database instance. Use lowercase letters, numbers, and hyphens. Start with a letter." +# Note, after a name db instance is used, it cannot be reused for up to one week. +variable "name_prefix" { + description = "The name prefix for the database instance. Will be appended with a random string. Use lowercase letters, numbers, and hyphens. Start with a letter." } -variable "master_username" { +variable "master_user_name" { description = "The username for the master user." } -variable "master_password" { +variable "master_user_password" { description = "The password for the master user." } @@ -30,11 +30,12 @@ variable "master_password" { # --------------------------------------------------------------------------------------------------------------------- # In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to GCP, this needs to remain open to the internet. variable "enable_public_internet_access" { - description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." + description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet" default = true } + variable "mysql_version" { - description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`." + description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`" default = "MYSQL_5_7" } @@ -43,5 +44,13 @@ variable "machine_type" { } variable "db_name" { + description = "Name for the db" default = "default" } + +variable "name_override" { + description = "You may optionally override the name_prefix + random string by specifying an override" + default = "" +} + + diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index b4e0060..33cf441 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -54,9 +54,9 @@ resource "google_sql_database" "default" { } resource "google_sql_user" "default" { - name = "${var.master_username}" + name = "${var.master_user_name}" project = "${var.project}" instance = "${google_sql_database_instance.master.name}" - host = "${var.master_host}" - password = "${var.master_password}" + host = "${var.master_user_host}" + password = "${var.master_user_password}" } diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index dc4be6c..bacadc1 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -34,11 +34,11 @@ variable "db_name" { default = "" } -variable "master_username" { +variable "master_user_name" { description = "The username for the master user." } -variable "master_password" { +variable "master_user_password" { description = "The password for the master user." } @@ -131,7 +131,7 @@ variable "zone" { default = "" } -variable "master_host" { +variable "master_user_host" { description = "The host for the default user" default = "%" } diff --git a/test/example_cloud_sql_mysql_test.go b/test/example_cloud_sql_mysql_test.go index b1da0f9..e08630d 100644 --- a/test/example_cloud_sql_mysql_test.go +++ b/test/example_cloud_sql_mysql_test.go @@ -6,11 +6,10 @@ import ( _ "github.com/go-sql-driver/mysql" "github.com/gruntwork-io/terratest/modules/gcp" "github.com/gruntwork-io/terratest/modules/logger" - "github.com/gruntwork-io/terratest/modules/random" "github.com/gruntwork-io/terratest/modules/terraform" "github.com/gruntwork-io/terratest/modules/test-structure" "github.com/stretchr/testify/assert" - "log" + "os" "path/filepath" "strings" "testing" @@ -19,30 +18,36 @@ import ( const DB_NAME = "testdb" const DB_USER = "testuser" const DB_PASS = "testpassword" +const NAME_PREFIX = "mysql-test" +const MYSQL_VERSION = "MYSQL_5_7" +const EXAMPLE_NAME = "cloud-sql-mysql" + +const KEY_REGION = "region" +const KEY_PROJECT = "project" + +const OUTPUT_INSTANCE_NAME = "instance_name" +const OUTPUT_PROXY_CONNECTION = "proxy_connection" +const OUTPUT_DB_NAME = "db_name" +const OUTPUT_PUBLIC_IP = "public_ip" -// TODO: try to actually connect to the RDS DBs and check they are working func TestCloudSQLMySql(t *testing.T) { t.Parallel() - //os.Setenv("SKIP_bootstrap", "true") + os.Setenv("SKIP_bootstrap", "true") //os.Setenv("SKIP_deploy", "true") - //os.Setenv("SKIP_validate_outputs", "true") - //os.Setenv("SKIP_sql_tests", "true") - //os.Setenv("SKIP_teardown", "true") + os.Setenv("SKIP_validate_outputs", "true") + os.Setenv("SKIP_sql_tests", "true") + os.Setenv("SKIP_teardown", "true") _examplesDir := test_structure.CopyTerraformFolderToTemp(t, "../", "examples") - exampleDir := filepath.Join(_examplesDir, "cloud-sql-mysql") - rootDir := filepath.Join(_examplesDir, "../") + exampleDir := filepath.Join(_examplesDir, EXAMPLE_NAME) test_structure.RunTestStage(t, "bootstrap", func() { - uniqueId := strings.ToLower(random.UniqueId()) - instanceName := fmt.Sprintf("mysql-test-%s", uniqueId) projectId := gcp.GetGoogleProjectIDFromEnvVar(t) region := getRandomRegion(t, projectId) - test_structure.SaveString(t, exampleDir, "instance-name", instanceName) - test_structure.SaveString(t, exampleDir, "region", region) - test_structure.SaveString(t, exampleDir, "project-id", projectId) + test_structure.SaveString(t, exampleDir, KEY_REGION, region) + test_structure.SaveString(t, exampleDir, KEY_PROJECT, projectId) }) // At the end of the test, run `terraform destroy` to clean up any resources that were created @@ -52,10 +57,9 @@ func TestCloudSQLMySql(t *testing.T) { }) test_structure.RunTestStage(t, "deploy", func() { - region := test_structure.LoadString(t, exampleDir, "region") - projectId := test_structure.LoadString(t, exampleDir, "project-id") - instanceName := test_structure.LoadString(t, exampleDir, "instance-name") - terraformOptions := createTerratestOptionsForMySql(projectId, region, instanceName, rootDir) + region := test_structure.LoadString(t, exampleDir, KEY_REGION) + projectId := test_structure.LoadString(t, exampleDir, KEY_PROJECT) + terraformOptions := createTerratestOptionsForMySql(projectId, region, exampleDir) test_structure.SaveTerraformOptions(t, exampleDir, terraformOptions) terraform.InitAndApply(t, terraformOptions) @@ -63,22 +67,25 @@ func TestCloudSQLMySql(t *testing.T) { test_structure.RunTestStage(t, "validate_outputs", func() { terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir) - instanceName := test_structure.LoadString(t, exampleDir, "instance-name") - region := test_structure.LoadString(t, exampleDir, "region") - projectId := test_structure.LoadString(t, exampleDir, "project-id") + region := test_structure.LoadString(t, exampleDir, KEY_REGION) + projectId := test_structure.LoadString(t, exampleDir, KEY_PROJECT) - expectedDBConn := fmt.Sprintf("%s:%s:%s", projectId, region, instanceName) + instanceNameFromOutput := terraform.Output(t, terraformOptions, OUTPUT_INSTANCE_NAME) + dbNameFromOutput := terraform.Output(t, terraformOptions, OUTPUT_DB_NAME) + proxyConnectionFromOutput := terraform.Output(t, terraformOptions, OUTPUT_PROXY_CONNECTION) - assert.Equal(t, instanceName, terraform.Output(t, terraformOptions, "instance_name")) - assert.Equal(t, "testdb", terraform.Output(t, terraformOptions, "db_name")) - assert.Equal(t, expectedDBConn, terraform.Output(t, terraformOptions, "proxy_connection")) + expectedDBConn := fmt.Sprintf("%s:%s:%s", projectId, region, instanceNameFromOutput) + + assert.True(t, strings.HasPrefix(instanceNameFromOutput, NAME_PREFIX)) + assert.Equal(t, DB_NAME, dbNameFromOutput) + assert.Equal(t, expectedDBConn, proxyConnectionFromOutput) }) test_structure.RunTestStage(t, "sql_tests", func() { terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir) - publicIp := terraform.Output(t, terraformOptions, "public_ip") + publicIp := terraform.Output(t, terraformOptions, OUTPUT_PUBLIC_IP) connectionString := fmt.Sprintf("%s:%s@tcp(%s:3306)/%s", DB_USER, DB_PASS, publicIp, DB_NAME) @@ -128,26 +135,26 @@ func TestCloudSQLMySql(t *testing.T) { // Get the last insert id lastId, err := res.LastInsertId() if err != nil { - log.Fatal(err) + t.Fatalf("Failed to get last insert id: %v", err) } // Since we set the auto increment to 5, modulus should always be 0 assert.Equal(t, int64(0), int64(lastId%5)) }) } -func createTerratestOptionsForMySql(projectId string, region string, instanceName string, repoPath string) *terraform.Options { +func createTerratestOptionsForMySql(projectId string, region string, exampleDir string) *terraform.Options { terratestOptions := &terraform.Options{ // The path to where your Terraform code is located - TerraformDir: filepath.Join(repoPath, "examples", "cloud-sql-mysql"), + TerraformDir: exampleDir, Vars: map[string]interface{}{ - "region": region, - "project": projectId, - "name": instanceName, - "mysql_version": "MYSQL_5_7", - "db_name": DB_NAME, - "master_username": DB_USER, - "master_password": DB_PASS, + "region": region, + "project": projectId, + "name_prefix": NAME_PREFIX, + "mysql_version": MYSQL_VERSION, + "db_name": DB_NAME, + "master_user_name": DB_USER, + "master_user_password": DB_PASS, }, } From 2980fdf97c99b485d6aa48cfd05a11ae92a1034b Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 00:14:38 +0200 Subject: [PATCH 21/34] Beautify error handling in tests --- test/example_cloud_sql_mysql_test.go | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/test/example_cloud_sql_mysql_test.go b/test/example_cloud_sql_mysql_test.go index b59db0c..6fa6cbe 100644 --- a/test/example_cloud_sql_mysql_test.go +++ b/test/example_cloud_sql_mysql_test.go @@ -9,6 +9,7 @@ import ( "github.com/gruntwork-io/terratest/modules/terraform" "github.com/gruntwork-io/terratest/modules/test-structure" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" "path/filepath" "strings" "testing" @@ -92,10 +93,7 @@ func TestCloudSQLMySql(t *testing.T) { logger.Logf(t, "Connecting to: %s", publicIp) db, err := sql.Open("mysql", connectionString) - - if err != nil { - t.Fatalf("Failed to open DB connection: %v", err) - } + require.NoError(t, err, "Failed to open DB connection") // Make sure we clean up properly defer db.Close() @@ -121,21 +119,16 @@ func TestCloudSQLMySql(t *testing.T) { // Insert data to check that our auto-increment flags worked logger.Logf(t, "Insert data: %s", MYSQL_INSERT_TEST_ROW) stmt, err := db.Prepare(MYSQL_INSERT_TEST_ROW) - if err != nil { - t.Fatalf("Failed to prepare statement: %v", err) - } + require.NoError(t, err, "Failed to prepare statement") // Execute the statement res, err := stmt.Exec("Grunt") - if err != nil { - t.Fatalf("Failed to execute statement: %v", err) - } + require.NoError(t, err, "Failed to execute statement") // Get the last insert id lastId, err := res.LastInsertId() - if err != nil { - t.Fatalf("Failed to get last insert id: %v", err) - } + require.NoError(t, err, "Failed to get last insert id") + // Since we set the auto increment to 5, modulus should always be 0 assert.Equal(t, int64(0), int64(lastId%5)) }) From de6517da8313ab03f5347179fdb98c93727a87aa Mon Sep 17 00:00:00 2001 From: Yevgeniy Brikman Date: Wed, 6 Feb 2019 02:53:13 +0200 Subject: [PATCH 22/34] Update examples/cloud-sql-mysql/main.tf Fix typo Co-Authored-By: autero1 --- examples/cloud-sql-mysql/main.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index c52b836..30852e8 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -48,7 +48,7 @@ module "mysql" { ] # Set auto-increment flags to test the - # feature in during automated testing + # feature during automated testing database_flags = [ { name = "auto_increment_increment" From 45fd95b50397902869aa7e80f588c488e9fad402 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 03:51:31 +0200 Subject: [PATCH 23/34] terraform fmt --- examples/cloud-sql-mysql/main.tf | 31 +++++++++++++-------------- examples/cloud-sql-mysql/outputs.tf | 3 +-- examples/cloud-sql-mysql/variables.tf | 6 ++---- modules/mysql/main.tf | 24 ++++++++++----------- modules/mysql/outputs.tf | 6 ++---- modules/mysql/variables.tf | 11 +++++----- 6 files changed, 38 insertions(+), 43 deletions(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 30852e8..c79b500 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -1,5 +1,5 @@ provider "google-beta" { - region = "${var.region}" + region = "${var.region}" project = "${var.project}" } @@ -22,29 +22,29 @@ module "mysql" { # When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you # to a specific version of the modules, such as the following example: # source = "git::git@github.com:gruntwork-io/terraform-google-sql.git//modules/mysql?ref=v0.1.0" - source = "../../modules/mysql" + source = "../../modules/mysql" - project = "${var.project}" - region = "${var.region}" - name = "${local.instance_name}" - db_name = "${var.db_name}" + project = "${var.project}" + region = "${var.region}" + name = "${local.instance_name}" + db_name = "${var.db_name}" - engine = "${var.mysql_version}" - machine_type = "${var.machine_type}" + engine = "${var.mysql_version}" + machine_type = "${var.machine_type}" - master_user_password = "${var.master_user_password}" - master_user_name = "${var.master_user_name}" - master_user_host = "%" - enable_public_internet_access = "${var.enable_public_internet_access}" + master_user_password = "${var.master_user_password}" + master_user_name = "${var.master_user_name}" + master_user_host = "%" + enable_public_internet_access = "${var.enable_public_internet_access}" # Never do this in production! # We're setting permissive network rules to make # it easier to test the instance authorized_networks = [ { - name = "allow-all-inbound", + name = "allow-all-inbound" value = "0.0.0.0/0" - } + }, ] # Set auto-increment flags to test the @@ -57,7 +57,6 @@ module "mysql" { { name = "auto_increment_offset" value = "5" - } + }, ] } - diff --git a/examples/cloud-sql-mysql/outputs.tf b/examples/cloud-sql-mysql/outputs.tf index e5a70af..c7e71d4 100644 --- a/examples/cloud-sql-mysql/outputs.tf +++ b/examples/cloud-sql-mysql/outputs.tf @@ -15,7 +15,7 @@ output "instance" { output "db_name" { description = "Name of the default database" - value = "${module.mysql.db_name}" + value = "${module.mysql.db_name}" } output "proxy_connection" { @@ -26,4 +26,3 @@ output "db" { description = "Self link to the default database" value = "${module.mysql.db}" } - diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index cf14b7a..2705fb4 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -36,7 +36,7 @@ variable "enable_public_internet_access" { variable "mysql_version" { description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`" - default = "MYSQL_5_7" + default = "MYSQL_5_7" } variable "machine_type" { @@ -50,7 +50,5 @@ variable "db_name" { variable "name_override" { description = "You may optionally override the name_prefix + random string by specifying an override" - default = "" + default = "" } - - diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index 33cf441..0e4707a 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -12,11 +12,11 @@ # ------------------------------------------------------------------------------ resource "google_sql_database_instance" "master" { - provider = "google-beta" - name = "${var.name}" - project = "${var.project}" - region = "${var.region}" - database_version = "${var.engine}" + provider = "google-beta" + name = "${var.name}" + project = "${var.project}" + region = "${var.region}" + database_version = "${var.engine}" settings { tier = "${var.machine_type}" @@ -25,19 +25,19 @@ resource "google_sql_database_instance" "master" { disk_autoresize = "${var.disk_autoresize}" ip_configuration { - authorized_networks = ["${var.authorized_networks}"], - ipv4_enabled = "${var.enable_public_internet_access}" + authorized_networks = ["${var.authorized_networks}"] + ipv4_enabled = "${var.enable_public_internet_access}" } location_preference { follow_gae_application = "${var.follow_gae_application}" - zone = "${var.zone}" + zone = "${var.zone}" } - disk_size = "${var.disk_size}" - disk_type = "${var.disk_type}" - database_flags = ["${var.database_flags}"] - availability_type = "${var.availability_type}" + disk_size = "${var.disk_size}" + disk_type = "${var.disk_type}" + database_flags = ["${var.database_flags}"] + availability_type = "${var.availability_type}" } } diff --git a/modules/mysql/outputs.tf b/modules/mysql/outputs.tf index 024c945..ffd22c1 100644 --- a/modules/mysql/outputs.tf +++ b/modules/mysql/outputs.tf @@ -15,17 +15,15 @@ output "instance" { output "db_name" { description = "Name of the default database" - value = "${google_sql_database.default.name}" + value = "${google_sql_database.default.name}" } - output "proxy_connection" { description = "Instance path for connecting with Cloud SQL Proxy. Read more at https://cloud.google.com/sql/docs/mysql/sql-proxy" - value = "${var.project}:${var.region}:${google_sql_database_instance.master.name}" + value = "${var.project}:${var.region}:${google_sql_database_instance.master.name}" } output "db" { description = "Self link to the default database" value = "${google_sql_database.default.self_link}" } - diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index 94dcb7b..69757b7 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -54,7 +54,8 @@ variable "activation_policy" { variable "authorized_networks" { description = "A list of authorized CIDR-formatted IP address ranges that can connect to this DB." type = "list" - default = [] + default = [] + # Example: # # authorized_networks = [ @@ -88,7 +89,7 @@ variable "db_collation" { variable "database_flags" { description = "List of Cloud SQL flags that are applied to the database server" - type = "list" + type = "list" default = [] # Example: @@ -122,12 +123,12 @@ variable "disk_type" { variable "follow_gae_application" { description = "A GAE application whose zone to remain in. Must be in the same region as this instance." - default = "" + default = "" } variable "zone" { description = "Preferred zone for the instance." - default = "" + default = "" } variable "master_user_host" { @@ -139,4 +140,4 @@ variable "master_user_host" { variable "enable_public_internet_access" { description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." default = false -} \ No newline at end of file +} From 0b5efeebf220d1460b2567fa46021560a6725273 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 10:23:40 +0200 Subject: [PATCH 24/34] Better documentation for db user --- examples/cloud-sql-mysql/variables.tf | 4 ++-- modules/mysql/variables.tf | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 2705fb4..24ff014 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -17,11 +17,11 @@ variable "name_prefix" { } variable "master_user_name" { - description = "The username for the master user." + description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " } variable "master_user_password" { - description = "The password for the master user." + description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " } # --------------------------------------------------------------------------------------------------------------------- diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index 69757b7..b3cf79a 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -34,11 +34,11 @@ variable "db_name" { } variable "master_user_name" { - description = "The username for the master user." + description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " } variable "master_user_password" { - description = "The password for the master user." + description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " } # --------------------------------------------------------------------------------------------------------------------- @@ -118,7 +118,7 @@ variable "disk_size" { variable "disk_type" { description = "The type of storage to use. Must be one of `PD_SSD` or `PD_HDD`." - default = "PD_HDD" + default = "PD_SSD" } variable "follow_gae_application" { @@ -132,7 +132,7 @@ variable "zone" { } variable "master_user_host" { - description = "The host for the default user" + description = "The host part for the default user, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " default = "%" } From f602f6e0de8129d5006ca1974f5db4b979d99470 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 10:35:44 +0200 Subject: [PATCH 25/34] Better documentation. --- examples/cloud-sql-mysql/main.tf | 12 +++++++++--- examples/cloud-sql-mysql/variables.tf | 4 ++-- modules/mysql/variables.tf | 4 ++-- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index c79b500..efdf003 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -32,9 +32,15 @@ module "mysql" { engine = "${var.mysql_version}" machine_type = "${var.machine_type}" - master_user_password = "${var.master_user_password}" - master_user_name = "${var.master_user_name}" - master_user_host = "%" + # These together will construct the master_user privileges, i.e. + # 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password'. + # These should typically be set as the environment variable TF_VAR_master_user_password, etc. + # so you don't check these into source control." + master_user_password = "${var.master_user_password}" + + master_user_name = "${var.master_user_name}" + master_user_host = "%" + enable_public_internet_access = "${var.enable_public_internet_access}" # Never do this in production! diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 24ff014..3bc6c87 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -17,11 +17,11 @@ variable "name_prefix" { } variable "master_user_name" { - description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " + description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password'. This should typically be set as the environment variable TF_VAR_master_user_name so you don't check it into source control." } variable "master_user_password" { - description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " + description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password'. This should typically be set as the environment variable TF_VAR_master_user_password so you don't check it into source control." } # --------------------------------------------------------------------------------------------------------------------- diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index b3cf79a..f83a41b 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -34,11 +34,11 @@ variable "db_name" { } variable "master_user_name" { - description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " + description = "The username part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password'. This should typically be set as the environment variable TF_VAR_master_user_name so you don't check it into source control." } variable "master_user_password" { - description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password' " + description = "The password part for the default user credentials, i.e. 'master_user_name'@'master_user_host' IDENTIFIED BY 'master_user_password'. This should typically be set as the environment variable TF_VAR_master_user_password so you don't check it into source control." } # --------------------------------------------------------------------------------------------------------------------- From 71251fc6b4db73ad1834b5bf051f39a5535de906 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 10:45:20 +0200 Subject: [PATCH 26/34] Better documentation for config options. --- examples/cloud-sql-mysql/variables.tf | 3 ++- modules/mysql/variables.tf | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 3bc6c87..32ea0fa 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -35,11 +35,12 @@ variable "enable_public_internet_access" { } variable "mysql_version" { - description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`" + description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`. See https://cloud.google.com/sql/docs/features for supported versions." default = "MYSQL_5_7" } variable "machine_type" { + description = "The machine type to use, see https://cloud.google.com/sql/pricing for more details" default = "db-f1-micro" } diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index f83a41b..cbf7adc 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -16,7 +16,7 @@ variable "name" { } variable "engine" { - description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`." + description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`. See https://cloud.google.com/sql/docs/features for supported versions." } # TODO: Depending on how the replicas are set up, tweak this. From 4ff672a22133efdd693b3eafd634c764c3a08bd1 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 10:47:06 +0200 Subject: [PATCH 27/34] Remove Aurora ref --- modules/mysql/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/mysql/README.md b/modules/mysql/README.md index 919b4e7..def56cb 100644 --- a/modules/mysql/README.md +++ b/modules/mysql/README.md @@ -48,7 +48,7 @@ For full connectivity options and detailed documentation, see [Connecting to Clo * **Vertical scaling**: To scale vertically (i.e. bigger DB instances with more CPU and RAM), use the `machine_type` input variable. For a list of Cloud SQL Machine Types, see [Cloud SQL Pricing](https://cloud.google.com/sql/pricing#2nd-gen-pricing). * **Horizontal scaling**: To scale horizontally, you can add more replicas using the `instance_count` input variable, - and Aurora will automatically deploy the new instances, sync them to the master, and make them available as read + and the module will automatically deploy the new instances, sync them to the master, and make them available as read replicas. ## Known Issues From 785375f5f01b915734d9eda5fc7a3170b55cae66 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 11:03:10 +0200 Subject: [PATCH 28/34] Add ability to add custom labels --- examples/cloud-sql-mysql/main.tf | 4 ++++ examples/cloud-sql-mysql/variables.tf | 2 +- modules/mysql/main.tf | 2 ++ modules/mysql/variables.tf | 6 ++++++ 4 files changed, 13 insertions(+), 1 deletion(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index efdf003..61839da 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -65,4 +65,8 @@ module "mysql" { value = "5" }, ] + + custom_labels = { + project = "mysql-example" + } } diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index 32ea0fa..c543996 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -41,7 +41,7 @@ variable "mysql_version" { variable "machine_type" { description = "The machine type to use, see https://cloud.google.com/sql/pricing for more details" - default = "db-f1-micro" + default = "db-f1-micro" } variable "db_name" { diff --git a/modules/mysql/main.tf b/modules/mysql/main.tf index 0e4707a..828901e 100644 --- a/modules/mysql/main.tf +++ b/modules/mysql/main.tf @@ -38,6 +38,8 @@ resource "google_sql_database_instance" "master" { disk_type = "${var.disk_type}" database_flags = ["${var.database_flags}"] availability_type = "${var.availability_type}" + + user_labels = "${var.custom_labels}" } } diff --git a/modules/mysql/variables.tf b/modules/mysql/variables.tf index cbf7adc..eab8a9a 100644 --- a/modules/mysql/variables.tf +++ b/modules/mysql/variables.tf @@ -141,3 +141,9 @@ variable "enable_public_internet_access" { description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet." default = false } + +variable "custom_labels" { + description = "A map of custom labels to apply to the instance. The key is the label name and the value is the label value." + type = "map" + default = {} +} From 02c33adcbfa1f17985546b0c0bfe1bb51163a04a Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 13:05:51 +0200 Subject: [PATCH 29/34] Clarify the public ip / cidr --- examples/cloud-sql-mysql/main.tf | 8 ++++---- examples/cloud-sql-mysql/variables.tf | 6 ------ 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/examples/cloud-sql-mysql/main.tf b/examples/cloud-sql-mysql/main.tf index 61839da..60b844a 100644 --- a/examples/cloud-sql-mysql/main.tf +++ b/examples/cloud-sql-mysql/main.tf @@ -41,11 +41,11 @@ module "mysql" { master_user_name = "${var.master_user_name}" master_user_host = "%" - enable_public_internet_access = "${var.enable_public_internet_access}" + # To make it easier to test this example, we are giving the servers public IP addresses and allowing inbound + # connections from anywhere. In real-world usage, your servers should live in private subnets, only have private IP + # addresses, and only allow access from specific trusted networks, servers or applications in your VPC. + enable_public_internet_access = true - # Never do this in production! - # We're setting permissive network rules to make - # it easier to test the instance authorized_networks = [ { name = "allow-all-inbound" diff --git a/examples/cloud-sql-mysql/variables.tf b/examples/cloud-sql-mysql/variables.tf index c543996..0eecc43 100644 --- a/examples/cloud-sql-mysql/variables.tf +++ b/examples/cloud-sql-mysql/variables.tf @@ -28,12 +28,6 @@ variable "master_user_password" { # OPTIONAL PARAMETERS # Generally, these values won't need to be changed. # --------------------------------------------------------------------------------------------------------------------- -# In nearly all cases, databases should NOT be publicly accessible, however if you're migrating from a PAAS provider like Heroku to GCP, this needs to remain open to the internet. -variable "enable_public_internet_access" { - description = "WARNING: - In nearly all cases a database should NOT be publicly accessible. Only set this to true if you want the database open to the internet" - default = true -} - variable "mysql_version" { description = "The engine version of the database, e.g. `MYSQL_5_6` or `MYSQL_5_7`. See https://cloud.google.com/sql/docs/features for supported versions." default = "MYSQL_5_7" From 9ceebf9d99c42c601b76f476117a2eadbc11a532 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 14:38:38 +0200 Subject: [PATCH 30/34] Add CI config and pre-commit hooks --- .circleci/config.yml | 103 +++++++++++++++++++++++++++++++++++++++- .pre-commit-config.yaml | 5 ++ 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 .pre-commit-config.yaml diff --git a/.circleci/config.yml b/.circleci/config.yml index 199337b..cfa1109 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -1 +1,102 @@ -# NO-OP \ No newline at end of file +defaults: &defaults + machine: true + environment: + GRUNTWORK_INSTALLER_VERSION: v0.0.21 + TERRATEST_LOG_PARSER_VERSION: v0.13.24 + MODULE_CI_VERSION: v0.13.3 + TERRAFORM_VERSION: 0.11.8 + TERRAGRUNT_VERSION: NONE + PACKER_VERSION: NONE + GOLANG_VERSION: 1.11.2 + +install_gruntwork_utils: &install_gruntwork_utils + name: install gruntwork utils + command: | + curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash /dev/stdin --version "${GRUNTWORK_INSTALLER_VERSION}" + gruntwork-install --module-name "gruntwork-module-circleci-helpers" --repo "https://github.com/gruntwork-io/module-ci" --tag "${MODULE_CI_VERSION}" + gruntwork-install --binary-name "terratest_log_parser" --repo "https://github.com/gruntwork-io/terratest" --tag "${TERRATEST_LOG_PARSER_VERSION}" + configure-environment-for-gruntwork-module \ + --circle-ci-2-machine-executor \ + --terraform-version ${TERRAFORM_VERSION} \ + --terragrunt-version ${TERRAGRUNT_VERSION} \ + --packer-version ${PACKER_VERSION} \ + --use-go-dep \ + --go-version ${GOLANG_VERSION} \ + --go-src-path test +version: 2 +jobs: + build: + <<: *defaults + steps: + - checkout + - restore_cache: + keys: + - dep-v1-{{ checksum "test/Gopkg.lock" }} + + # Install gruntwork utilities + - run: + <<: *install_gruntwork_utils + + - save_cache: + key: dep-v1-{{ checksum "test/Gopkg.lock" }} + paths: + - ./test/vendor + + # Fail the build if the pre-commit hooks don't pass. Note: if you run pre-commit install locally, these hooks will + # execute automatically every time before you commit, ensuring the build never fails at this step! + - run: pip install pre-commit==1.11.2 + - run: pre-commit install + - run: pre-commit run --all-files + + - persist_to_workspace: + root: /home/circleci + paths: + - project + - terraform + - packer + + test: + <<: *defaults + steps: + - attach_workspace: + at: /home/circleci + - checkout + - run: echo 'export PATH=$HOME/terraform:$HOME/packer:$PATH' >> $BASH_ENV + - run: + <<: *install_gruntwork_utils + - run: + name: update gcloud + command: | + sudo apt-get remove -y google-cloud-sdk + sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update + sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update beta + - run: + name: run tests + command: | + mkdir -p /tmp/logs + # required for gcloud to authenticate correctly + echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=- + gcloud --quiet config set project ${GOOGLE_PROJECT_ID} + gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE} + # required for terraform and terratest to authenticate correctly + echo $GCLOUD_SERVICE_KEY > /tmp/gcloud.json + export GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcloud.json" + # run the tests + run-go-tests --path test --timeout 60m | tee /tmp/logs/all.log + no_output_timeout: 3600s + - run: + command: terratest_log_parser --testlog /tmp/logs/all.log --outputdir /tmp/logs + when: always + - store_artifacts: + path: /tmp/logs + - store_test_results: + path: /tmp/logs + +workflows: + version: 2 + build-and-test: + jobs: + - build + - test: + requires: + - build \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..d6ca6c5 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,5 @@ +repos: + - repo: https://github.com/gruntwork-io/pre-commit + rev: v0.0.4 + hooks: + - id: terraform-fmt From ec9dc9b01e2580ea4ac06f4c8146df4bc45ba7e6 Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 15:13:14 +0200 Subject: [PATCH 31/34] Dummy commit --- .pre-commit-config.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d6ca6c5..1752b28 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -3,3 +3,4 @@ repos: rev: v0.0.4 hooks: - id: terraform-fmt +# \ No newline at end of file From 8ba64b8e8eb1049d20a1eaf78aed9411b9fe598e Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Wed, 6 Feb 2019 16:39:41 +0200 Subject: [PATCH 32/34] Remove unnecessary comment --- test/example_cloud_sql_postgres_test.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/test/example_cloud_sql_postgres_test.go b/test/example_cloud_sql_postgres_test.go index c51466c..603cb9b 100644 --- a/test/example_cloud_sql_postgres_test.go +++ b/test/example_cloud_sql_postgres_test.go @@ -6,9 +6,6 @@ import ( "github.com/stretchr/testify/assert" ) -// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in -// the templates -// TODO: try to actually connect to the RDS DBs and check they are working func TestCloudSQLPostgres(t *testing.T) { t.Parallel() From 0a71803bc66b2245ec8514b6b99e2c5fc49f9a4b Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Thu, 7 Feb 2019 09:05:53 +0200 Subject: [PATCH 33/34] [skip-ci] Improved module documentation. --- modules/mysql/README.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/modules/mysql/README.md b/modules/mysql/README.md index def56cb..6b3d2cd 100644 --- a/modules/mysql/README.md +++ b/modules/mysql/README.md @@ -3,9 +3,7 @@ This module creates a [Google Cloud SQL](https://cloud.google.com/sql/) [MySQL](https://cloud.google.com/sql/docs/mysql/) cluster. The cluster is managed by Google, automating backups, replication, patches, and updates. -TODO: Figure out documentation format for separate modules. - -You can use Cloud SQL with either [MySQL](https://cloud.google.com/sql/docs/mysql/) or [PostgreSQL](https://cloud.google.com/sql/docs/postgres/). +This module helps you run [MySQL](https://cloud.google.com/sql/docs/mysql/), see [postgres](../postgresql) for running [PostgreSQL](https://cloud.google.com/sql/docs/postgres/). ## How do you use this module? @@ -30,8 +28,8 @@ variables](https://www.terraform.io/intro/getting-started/outputs.html): 1. TODO: **Private IP** `private_ip`: The public endpoint for the cluster. -1. **Public IP** `public_ip`: The public endpoint for the cluster. -1. **Proxy connection** `proxy_connection`: "Instance path for connecting with Cloud SQL Proxy. Read more at https://cloud.google.com/sql/docs/mysql/sql-proxy. +1. **Public IP** `public_ip`: The public endpoint for the cluster; see [Connecting mysql Client Using Public IP](https://cloud.google.com/sql/docs/mysql/connect-admin-ip). +1. **Proxy connection** `proxy_connection`: Instance path for connecting with Cloud SQL Proxy; see [Connecting mysql Client Using the Cloud SQL Proxy](https://cloud.google.com/sql/docs/mysql/connect-admin-proxy). 1. TODO: **Replica endpoints** `replica_endpoints`: A comma-separated list of all DB instance URLs in the cluster, including the primary and all read replicas. Use these URLs for reads (see "How do you scale this DB?" below). From 9649aa05c20e6a448409b67ba8249a49c6a298de Mon Sep 17 00:00:00 2001 From: Petri Autero Date: Thu, 7 Feb 2019 09:07:33 +0200 Subject: [PATCH 34/34] [skip-ci] Fix module link. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cf41fe7..99ac946 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ This repo contains modules for running relational databases such as MySQL and Po ## Code included in this Module * [mysql](/modules/mysql): Deploy a Cloud SQL MySQL cluster. -* [postgresql](/modules/mysql): Deploy a Cloud SQL PostgreSQL cluster. +* [postgresql](/modules/postgresql): Deploy a Cloud SQL PostgreSQL cluster. ## What is Cloud SQL?