diff --git a/.circleci/config.yml b/.circleci/config.yml new file mode 100644 index 0000000..199337b --- /dev/null +++ b/.circleci/config.yml @@ -0,0 +1 @@ +# NO-OP \ No newline at end of file diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..190168e --- /dev/null +++ b/.gitignore @@ -0,0 +1,26 @@ +# Terraform files +.terraform +terraform.tfstate +terraform.tfvars +*.tfstate* +*.zip + +# OS X files +.history +.DS_Store + +# IntelliJ files +.idea_modules +*.iml +*.iws +*.ipr +.idea/ +build/ +*/build/ +out/ + +# Go best practices dictate that libraries should not include the vendor directory +vendor + +#VIM swap files +*.swp diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..20b9986 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,99 @@ +# Contribution Guidelines + +Contributions to this Module are very welcome! We follow a fairly standard [pull request process]( +https://help.github.com/articles/about-pull-requests/) for contributions, subject to the following guidelines: + +1. [File a GitHub issue](#file-a-github-issue) +1. [Update the documentation](#update-the-documentation) +1. [Update the tests](#update-the-tests) +1. [Update the code](#update-the-code) +1. [Create a pull request](#create-a-pull-request) +1. [Merge and release](#merge-and-release) + +## File a GitHub issue + +Before starting any work, we recommend filing a GitHub issue in this repo. This is your chance to ask questions and +get feedback from the maintainers and the community before you sink a lot of time into writing (possibly the wrong) +code. If there is anything you're unsure about, just ask! + +## Update the documentation + +We recommend updating the documentation *before* updating any code (see [Readme Driven +Development](http://tom.preston-werner.com/2010/08/23/readme-driven-development.html)). This ensures the documentation +stays up to date and allows you to think through the problem at a high level before you get lost in the weeds of +coding. + +## Update the tests + +We also recommend updating the automated tests *before* updating any code (see [Test Driven +Development](https://en.wikipedia.org/wiki/Test-driven_development)). That means you add or update a test case, +verify that it's failing with a clear error message, and *then* make the code changes to get that test to pass. This +ensures the tests stay up to date and verify all the functionality in this Module, including whatever new +functionality you're adding in your contribution. Check out the [tests](https://github.com/gruntwork-io/terraform-google-sql/tree/master/test) folder for instructions on running the +automated tests. + +## Update the code + +At this point, make your code changes and use your new test case to verify that everything is working. As you work, +keep in mind two things: + +1. Backwards compatibility +1. Downtime + +### Backwards compatibility + +Please make every effort to avoid unnecessary backwards incompatible changes. With Terraform code, this means: + +1. Do not delete, rename, or change the type of input variables. +1. If you add an input variable, it should have a `default`. +1. Do not delete, rename, or change the type of output variables. +1. Do not delete or rename a module in the `modules` folder. + +If a backwards incompatible change cannot be avoided, please make sure to call that out when you submit a pull request, +explaining why the change is absolutely necessary. + +### Downtime + +Bear in mind that the Terraform code in this Module is used by real companies to run real infrastructure in +production, and certain types of changes could cause downtime. For example, consider the following: + +1. If you rename a resource (e.g. `google_sql_database_instance "foo"` -> `google_sql_database_instance "bar"`), Terraform will see that as deleting + the old resource and creating a new one. +1. If you change certain attributes of a resource (e.g. the `name` of an `google_compute_instance`), the cloud provider (e.g. Google) may + treat that as an instruction to delete the old resource and a create a new one. + +Deleting certain types of resources (e.g. virtual servers, load balancers) can cause downtime, so when making code +changes, think carefully about how to avoid that. For example, can you avoid downtime by using +[create_before_destroy](https://www.terraform.io/docs/configuration/resources.html#create_before_destroy)? Or via +the `terraform state` command? If so, make sure to note this in our pull request. If downtime cannot be avoided, +please make sure to call that out when you submit a pull request. + + +### Formatting and pre-commit hooks + +You must run `terraform fmt` on the code before committing. You can configure your computer to do this automatically +using pre-commit hooks managed using [pre-commit](http://pre-commit.com/): + +1. [Install pre-commit](http://pre-commit.com/#install). E.g.: `brew install pre-commit`. +1. Install the hooks: `pre-commit install`. + +That's it! Now just write your code, and every time you commit, `terraform fmt` will be run on the files you're +committing. + + +## Create a pull request + +[Create a pull request](https://help.github.com/articles/creating-a-pull-request/) with your changes. Please make sure +to include the following: + +1. A description of the change, including a link to your GitHub issue. +1. The output of your automated test run, preferably in a [GitHub Gist](https://gist.github.com/). We cannot run + automated tests for pull requests automatically due to [security + concerns](https://circleci.com/docs/fork-pr-builds/#security-implications), so we need you to manually provide this + test output so we can verify that everything is working. +1. Any notes on backwards incompatibility or downtime. + +## Merge and release + +The maintainers for this repo will review your code and provide feedback. If everything looks good, they will merge the +code and release a new version, which you'll be able to find in the [releases page](../../releases). \ No newline at end of file diff --git a/GRUNTWORK_PHILOSOPHY.md b/GRUNTWORK_PHILOSOPHY.md new file mode 100644 index 0000000..22a28a8 --- /dev/null +++ b/GRUNTWORK_PHILOSOPHY.md @@ -0,0 +1,64 @@ +# Gruntwork Philosophy + +At Gruntwork, we strive to accelerate the deployment of production grade infrastructure by prodiving a library of +stable, reusable, and battle tested infrastructure as code organized into a series of [modules](#what-is-a-module) with +[submodules](#what-is-a-submodule). Each module represents a particular set of infrastructure that is componentized into +smaller pieces represented by the submodules within the module. By doing so, we have built a composable library that can +be combined into building out everything from simple single service deployments to complicated microservice setups so +that your infrastructure can grow with your business needs. Every module we provide is built with the [production grade +infrastruture checklist](#production-grade-infrastructure-checklist) in mind, ensuring that the services you deploy are +resilient, fault tolerant, and scalable. + + +## What is a Module? + +A Module is a reusable, tested, documented, configurable, best-practices definition of a single piece of Infrastructure +(e.g., Docker cluster, VPC, Jenkins, Consul), written using a combination of [Terraform](https://www.terraform.io/), Go, +and Bash. A module contains a set of automated tests, documentation, and examples that have been proven in production, +providing the underlying infrastructure for [Gruntwork's customers](https://www.gruntwork.io/customers). + +Instead of figuring out the details of how to run a piece of infrastructure from scratch, you can reuse existing code +that has been proven in production. And instead of maintaining all that infrastructure code yourself, you can leverage +the work of the community to pick up infrastructure improvements through a version number bump. + + +## What is a Submodule? + +Each Infrastructure Module consists of one or more orthogonal Submodules that handle some specific aspect of that +Infrastructure Module's functionality. Breaking the code up into multiple submodules makes it easier to reuse and +compose to handle many different use cases. Although Modules are designed to provide an end to end solution to manage +the relevant infrastructure by combining the Submodules defined in the Module, Submodules can be used independently for +specific functionality that you need in your infrastructure code. + + +## Production Grade Infrastructure Checklist + +At Gruntwork, we have learned over the years that it is not enough to just get the services up and running in a publicly +accessible space to call your application "production-ready." There are many more things to consider, and oftentimes +many of these considerations are missing in the deployment plan of applications. These topics come up as afterthoughts, +and are learned the hard way after the fact. That is why we codified all of them into a checklist that can be used as a +reference to help ensure that they are considered before your application goes to production, and conscious decisions +are made to neglect particular components if needed, as opposed to accidentally omitting them from consideration. + + + +| Task | Description | Example tools | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| +| Install | Install the software binaries and all dependencies. | Bash, Chef, Ansible, Puppet | +| Configure | Configure the software at runtime. Includes port settings, TLS certs, service discovery, leaders, followers, replication, etc. | Bash, Chef, Ansible, Puppet | +| Provision | Provision the infrastructure. Includes EC2 instances, load balancers, network topology, security gr oups, IAM permissions, etc. | Terraform, CloudFormation | +| Deploy | Deploy the service on top of the infrastructure. Roll out updates with no downtime. Includes blue-green, rolling, and canary deployments. | Scripts, Orchestration tools (ECS, k8s, Nomad) | +| High availability | Withstand outages of individual processes, EC2 instances, services, Availability Zones, and regions. | Multi AZ, multi-region, replication, ASGs, ELBs | +| Scalability | Scale up and down in response to load. Scale horizontally (more servers) and/or vertically (bigger servers). | ASGs, replication, sharding, caching, divide and conquer | +| Performance | Optimize CPU, memory, disk, network, GPU, and usage. Includes query tuning, benchmarking, load testing, and profiling. | Dynatrace, valgrind, VisualVM, ab, Jmeter | +| Networking | Configure static and dynamic IPs, ports, service discovery, firewalls, DNS, SSH access, and VPN access. | EIPs, ENIs, VPCs, NACLs, SGs, Route 53, OpenVPN | +| Security | Encryption in transit (TLS) and on disk, authentication, authorization, secrets management, server hardening. | ACM, EBS Volumes, Cognito, Vault, CIS | +| Metrics | Availability metrics, business metrics, app metrics, server metrics, events, observability, tracing, and alerting. | CloudWatch, DataDog, New Relic, Honeycomb | +| Logs | Rotate logs on disk. Aggregate log data to a central location. | CloudWatch logs, ELK, Sumo Logic, Papertrail | +| Backup and Restore | Make backups of DBs, caches, and other data on a scheduled basis. Replicate to separate region/account. | RDS, ElastiCache, ec2-snapper, Lambda | +| Cost optimization | Pick proper instance types, use spot and reserved instances, use auto scaling, and nuke unused resources. | ASGs, spot instances, reserved instances | +| Documentation | Document your code, architecture, and practices. Create playbooks to respond to incidents. | READMEs, wikis, Slack | +| Tests | Write automated tests for your infrastructure code. Run tests after every commit and nightly. | Terratest | \ No newline at end of file diff --git a/LICENSE b/LICENSE.txt similarity index 100% rename from LICENSE rename to LICENSE.txt diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..24a7708 --- /dev/null +++ b/NOTICE @@ -0,0 +1,4 @@ +terraform-google-sql +Copyright 2019 Gruntwork, Inc. + +This product includes software developed at Gruntwork (https://www.gruntwork.io/). diff --git a/README.md b/README.md new file mode 100644 index 0000000..6a92293 --- /dev/null +++ b/README.md @@ -0,0 +1,53 @@ +[![Maintained by Gruntwork.io](https://img.shields.io/badge/maintained%20by-gruntwork.io-%235849a6.svg)](https://gruntwork.io/?ref=repo_google_cloudsql) + +# Cloud SQL Modules + +This repo contains modules for running relational databases such as MySQL and PostgreSQL on Google's +[Cloud SQL](https://cloud.google.com/sql/) on [GCP](https://cloud.google.com/). + +## Code included in this Module + +* [cloud-sql](/modules/cloud-sql): Deploy a Cloud SQL cluster. + + +## What is Cloud SQL? + +Cloud SQL is Google's fully-managed database service that makes it easy to set up, maintain, manage, and administer +your relational databases on Google Cloud Platform. Cloud SQL automatically includes Data replication between multiple +zones with automatic failover, automated and on-demand backups, and point-in-time recovery. + +You can learn more Cloud SQL from [the official documentation](https://cloud.google.com/sql/docs/). + +## Who maintains this Module? + +This Module and its Submodules are maintained by [Gruntwork](http://www.gruntwork.io/). If you are looking for help or +commercial support, send an email to +[support@gruntwork.io](mailto:support@gruntwork.io?Subject=Google%20SQL%20Module). + +Gruntwork can help with: + +* Setup, customization, and support for this Module. +* Modules and submodules for other types of infrastructure, such as VPCs, Docker clusters, databases, and continuous + integration. +* Modules and Submodules that meet compliance requirements, such as HIPAA. +* Consulting & Training on GCP, AWS, Terraform, and DevOps. + + +## How do I contribute to this Module? + +Contributions are very welcome! Check out the [Contribution Guidelines](/CONTRIBUTING.md) for instructions. + + +## How is this Module versioned? + +This Module follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, along +with the changelog, in the [Releases Page](../../releases). + +During initial development, the major version will be 0 (e.g., `0.x.y`), which indicates the code does not yet have a +stable API. Once we hit `1.0.0`, we will make every effort to maintain a backwards compatible API and use the MAJOR, +MINOR, and PATCH versions on each release to indicate any incompatibilities. + + +## License + +Please see [LICENSE.txt](/LICENSE.txt) for details on how the code in this repo is licensed. diff --git a/examples/cloud-sql-mysql/.gitkeep b/examples/cloud-sql-mysql/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/examples/cloud-sql-postgres/.gitkeep b/examples/cloud-sql-postgres/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/modules/cloud-sql/.gitkeep b/modules/cloud-sql/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/test/Gopkg.lock b/test/Gopkg.lock new file mode 100644 index 0000000..3c7bff3 --- /dev/null +++ b/test/Gopkg.lock @@ -0,0 +1,33 @@ +# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'. + + +[[projects]] + digest = "1:0deddd908b6b4b768cfc272c16ee61e7088a60f7fe2f06c547bd3d8e1f8b8e77" + name = "github.com/davecgh/go-spew" + packages = ["spew"] + pruneopts = "" + revision = "8991bc29aa16c548c550c7ff78260e27b9ab7c73" + version = "v1.1.1" + +[[projects]] + digest = "1:256484dbbcd271f9ecebc6795b2df8cad4c458dd0f5fd82a8c2fa0c29f233411" + name = "github.com/pmezard/go-difflib" + packages = ["difflib"] + pruneopts = "" + revision = "792786c7400a136282c1664665ae0a8db921c6c2" + version = "v1.0.0" + +[[projects]] + digest = "1:381bcbeb112a51493d9d998bbba207a529c73dbb49b3fd789e48c63fac1f192c" + name = "github.com/stretchr/testify" + packages = ["assert"] + pruneopts = "" + revision = "ffdc059bfe9ce6a4e144ba849dbedead332c6053" + version = "v1.3.0" + +[solve-meta] + analyzer-name = "dep" + analyzer-version = 1 + input-imports = ["github.com/stretchr/testify/assert"] + solver-name = "gps-cdcl" + solver-version = 1 diff --git a/test/Gopkg.toml b/test/Gopkg.toml new file mode 100644 index 0000000..7d7581c --- /dev/null +++ b/test/Gopkg.toml @@ -0,0 +1,25 @@ +# Gopkg.toml example +# +# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md +# for detailed Gopkg.toml documentation. +# +# required = ["github.com/user/thing/cmd/thing"] +# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"] +# +# [[constraint]] +# name = "github.com/user/project" +# version = "1.0.0" +# +# [[constraint]] +# name = "github.com/user/project2" +# branch = "dev" +# source = "github.com/myfork/project2" +# +# [[override]] +# name = "github.com/x/y" +# version = "2.4.0" + +[[constraint]] + name = "github.com/gruntwork-io/terratest" + source = "git@github.com:gruntwork-io/terratest" + version = "0.13.22" diff --git a/test/README.md b/test/README.md new file mode 100644 index 0000000..3f0f0d9 --- /dev/null +++ b/test/README.md @@ -0,0 +1,59 @@ +# Tests + +This folder contains automated tests for this Module. All of the tests are written in [Go](https://golang.org/). +Most of these are "integration tests" that deploy real infrastructure using Terraform and verify that infrastructure +works as expected using a helper library called [Terratest](https://github.com/gruntwork-io/terratest). + + + +## WARNING WARNING WARNING + +**Note #1**: Many of these tests create real resources in an AWS account and then try to clean those resources up at +the end of a test run. That means these tests may cost you money to run! When adding tests, please be considerate of +the resources you create and take extra care to clean everything up when you're done! + +**Note #2**: Never forcefully shut the tests down (e.g. by hitting `CTRL + C`) or the cleanup tasks won't run! + +**Note #3**: We set `-timeout 60m` on all tests not because they necessarily take that long, but because Go has a +default test timeout of 10 minutes, after which it forcefully kills the tests with a `SIGQUIT`, preventing the cleanup +tasks from running. Therefore, we set an overlying long timeout to make sure all tests have enough time to finish and +clean up. + + + +## Running the tests + +### Prerequisites + +- Install the latest version of [Go](https://golang.org/). +- Install [dep](https://github.com/golang/dep) for Go dependency management. +- Install [Terraform](https://www.terraform.io/downloads.html). +- Configure your Google credentials using one of the [options supported by GCP](https://cloud.google.com/docs/authentication/getting-started). + + +### One-time setup + +Download Go dependencies using dep: + +``` +cd test +dep ensure +``` + + +### Run all the tests + +```bash +cd test +go test -v -timeout 60m +``` + + +### Run a specific test + +To run a specific test called `TestFoo`: + +```bash +cd test +go test -v -timeout 60m -run TestFoo +``` \ No newline at end of file diff --git a/test/example_cloud_sql_mysql_test.go b/test/example_cloud_sql_mysql_test.go new file mode 100644 index 0000000..5310c38 --- /dev/null +++ b/test/example_cloud_sql_mysql_test.go @@ -0,0 +1,16 @@ +package test + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in +// the templates +// TODO: try to actually connect to the RDS DBs and check they are working +func TestCloudSQLMySql(t *testing.T) { + t.Parallel() + + assert.Equal(t, "3306", "3306") +} diff --git a/test/example_cloud_sql_postgres_test.go b/test/example_cloud_sql_postgres_test.go new file mode 100644 index 0000000..c51466c --- /dev/null +++ b/test/example_cloud_sql_postgres_test.go @@ -0,0 +1,16 @@ +package test + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// A basic sanity check of the MySQL example that just deploys and undeploys it to make sure there are no errors in +// the templates +// TODO: try to actually connect to the RDS DBs and check they are working +func TestCloudSQLPostgres(t *testing.T) { + t.Parallel() + + assert.Equal(t, "5432", "5432") +}