## 2024-07-24

### Attendees

- Christan Mesh ([@cam72cam](https://github.com/cam72cam)) (OpenTofu Tech Lead)
- Igor Savchenko ([@DiscyDel](https://github.com/DicsyDel))
- Roger Simms ([@allofthesepeople](https://github.com/allofthesepeople))
- Wojciech Barczynski ([@wojciech12](https://github.com/wojciech12))
- Zach Goldberg ([@ZachGoldberg](https://github.com/ZachGoldberg))

### Agenda

As has been discussed in past TSC meetings, backends are an important and sensitive part of OpenTofu. We have been hesitant to make any significant changes to backends for a variety of reasons, compatibility being the major reason.

#### Backends

Problems we face:

- Backends use libraries which are out of date or that are heavily dependent on HC
    - Azure backend is a piece of work…
    - We inherit any S3 backend bugs from HC’s libraries, which we are hesitant to patch
- Backends would benefit from restructuring
    - HTTP backend should support workspaces (frequently requested)
    - Azure backend needs to support auth from this decade
- New Backends are requested often
    - Support for additional clouds (Oracle being the largest)
    - Workaround is http (no workspaces, custom service required) or s3 (no locking / buggy)
    - Adding backends into opentofu bloats our test matrix and increases our maintenance burden
- Remote and Cloud “backends” are not properly documented or maintained
    - Some TACOS support a subset of cloud/remote features
    - Is dependent on HC’s go-tfe library, built for their cloud offering
    - Code is a copy-paste nightmare
    - We don’t know how to maintain this properly from our end
- Backends are tied to specific tofu version
    - Any bugfixes or workarounds for a given cloud/service must be rolled into an opentofu release.

Potential Solution: Backend Plugins

Advantages:

- Existing backends could be moved into their own repo and be versioned as 1.0
    - Maps 1-1 with existing configurations, requiring no changes to users
    - Potentially simpler collaboration with AWS/Azure/GCP/etc…
- New backends can be authored by independent teams (Oracle, etc…) with minimal involvement of core team
- Users can specify when they want to upgrade their backend configurations/services separate from tofu
- Encapsulates backends behind a simple API for compliance testing
- Allows forks to be maintained by others without having to fork all of opentofu
- Reduces tofu’s primary code and binary size (less spidering dependencies)
- Prototyped by Marcin
- Potential to collaborate with HC on a set of shared backends?

Disadvantages:

- Requires significant developer time / testing
- How to handle protocol upgrades
- How to handle validation/verification of downloaded backend during `tofu init`

Potential Solution: Extend http backend to support workspaces (or make new version)

Advantages:

- Simple HTTP API to implement
- Simple Authentication mechanism
- Has some ecosystem adoption already

Disadvantages:

- Restricted to HTTP transport layer (potential problems with large state files)
- Requires dedicated service backed by some other storage layer / abstraction (leaky)
- Complex auth (cloud environment vars / service tokens) are difficult to support properly

Potential Solution: Accept new backends and start to make large changes to existing backends

Advantages:

- Works with code / process we already have
- Requires closer collaboration with other organizations (Oracle)

Disadvantages:

- Requires documenting backend changes heavily in every OpenTofu release and forcing migrations
- Adds direct dependencies on external teams / cloud infrastructure
- Dramatically increases testing matrix / testing load
- Contributes to code / binary bloat

At this point we need a decision from the TSC on which paths we should be investigating and creating detailed RFCs for.

###### Discussion

- Roni: What are the upvotes for fixes modifications vs new backends?

- Igor: Cautious about end user requests (tech debt / architecture concerns).  Prioritize healthy code base alongside new features.

- Zach: Wants an overview of our top feature requests in something like a ghant chart to help understand impact of large projects like this

- Wojciech: Likes idea of accepting new backends, with codeowners from top clouds, long term extract protocol?

- Igor: Opposite view, oracle specifically we can just say No with clear explanation.  Limited developer capacity.  Users won’t know if it’s a “tofu” or “oracle” problem.  Initial blame will be on OpenTofu. Potential for conflicts in what opentofu wants for state management vs what oracle wishes

- Christian: I think it’s not a significant technical challenge to define and lock down the interface

- Roni: Many not bring a lot of value to our customers and users.  Potentially not worth the investment for backends as plugins?

- Christian: Potentially invest in http backend further to make that our initial “backend interface” with examples.

- Wojciech: Likes the idea of http backend being the interface

- Roni: Env0 built on top of remote backend protocol, was not terribly difficult.  Kuba suggested introducing “one new backend” which is or “official” solution (OCI perhaps).  Strongly prefers remote interface over pluggable backends.

- Christian: How does auth work for remote and cloud / our OCI solution.

- Igor: This is a reasonably solved problem that solves most scenarios

- Roni: remote backend is quite popular and very functional at Env0.

- Igor: Scalr forces remote/cloud backend. Migration from tfe is currently tricky, needs a taco.  TACOS + tofu could together build migration path away from tfe.

- Roni: Migration from cloud/remote tfe block to whatever tofu’s preferred solution.  Could cloud/remote be the preferred solution?  Focus on backend migration to tofu as a smooth transition.

- Igor: Scalr has significant test harness against cloud/remote backend.  Offers help from devs in Scalr.

- Roni: At least one core team member has experience on Env0’s remote backend

- Christian: Core team can take this discussion and produce comparisons between options discussed here and breakdown of issue voting. Try to have this prepared for a week or two from now.

- James: Compare to features we already done for comparison

##### Recommendation

- Preferable one officially supported backend with all necessary functionality, for example, http backend.
- Continue discussion in the core team and TSC.

