Previously we interpreted a "required_version" argument in a "terraform"
block as if it were specifying an OpenTofu version constraint, when in
reality most modules use this to represent a version constraint for
OpenTofu's predecessor instead.
The primary effect of this commit is to introduce a new top-level block
type called "language" which describes language and implementation
compatibility metadata in a way that intentionally differs from what's used
by OpenTofu's predecessor.
This also causes OpenTofu to ignore the required_version argument unless
it appears in an OpenTofu-specific file with a ".tofu" suffix, and makes
OpenTofu completely ignore the language edition and experimental feature
opt-in options from OpenTofu's predecessor on the assumption that those
could continue to evolve independently of changes in OpenTofu.
We retain support for using required_versions in .tofu files as a bridge
solution for modules that need to remain compatible with OpenTofu versions
prior to v1.12. Module authors should keep following the strategy of
having both a versions.tf and a versions.tofu file for now, and wait until
the OpenTofu v1.11 series is end-of-life before adopting the new "language"
block type.
I also took this opportunity to simplify how we handle these parts of the
configuration, since the OpenTofu project has no immediate plans to use
either multiple language editions or language experiments and so for now
we can reduce our handling of those language features to just enough that
we'd return reasonable error messages if today's OpenTofu is exposed to
a module that was written for a newer version of OpenTofu that extends
these language features. The cross-cutting plumbing for representing the
active experiments for a module is still present so that we can reactivate
it later if we need to, but for now that set will always be empty.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We don't typically just broadly run automatic rewriting tools like "go fix"
across our codebase because that tends to cause annoying and unnecessary
merge conflicts when we're backporting to earlier release branches.
But all of the files in this commit were changed in some non-trivial way
already during the OpenTofu v1.11 development period anyway, and so the
likelyhood we'd be able to successfully backport from them is reduced and
therefore this seems like a good opportunity to do some focused
modernization using "go fix".
My rules for what to include or not are admittedly quite "vibes-based", but
the general idea was:
- Focusing on files under the "command" directory only, because that's
already been an area of intentional refactoring during this development
period.
- If the existing diff in a file is already significantly larger than
the changes the fixer proposed to make, or if the fixer is proposing
to change a line that was already changed in this development period.
- More willing to include "_test.go" files than non-test files, even if
they hadn't changed as much already, just because backports from test
files for bug fixes tend to be entirely new test cases more than they
are modifications to existing test cases, and so the risk of conflicts
is lower there.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This introduces the concept of "backend aliases", which are alternative
names that can be used to refer to a given backend.
Each backend type has one canonical name and zero or more alias names. The
"backend" block in the root module can specify either a canonical backend
type or an alias, but internally OpenTofu will always track the backend
type using its canonical name.
In particular, the following are all true when the configuration specifies
an alias instead of a canonical backend type:
- The "tofu init" output includes a brief extra message saying which
backend type OpenTofu actually used, because that is the name that we'd
prioritize in our documentation and so an operator can use the canonical
type to find the relevant docs when needed.
- The .terraform/terraform.tfstate file that tracks the working directory's
currently-initialized backend settings always uses the canonical backend
type, and so it's possible to freely switch between aliases and canonical
without "tofu init" thinking that a state migration might be needed.
- Plan files similarly use the canonical backend type to track which
backend was active when the plan was created, which doesn't have any
significant user-facing purpose, but is consistent with the previous
point since the settings in the plan file effectively substitute for
the .terraform/terraform.tfstate file when applying a saved plan.
- The terraform_remote_state data source in the provider
terraform.io/builtin/terraform accepts both canonical and alias in its
backend type argument, treating both as equivalent for the purpose of
fetching the state snapshot for the configured workspace.
The primary motivation for this new facility is to allow the planned
"oracle_oci" backend to have an alias "oci" to allow writing configurations
that are cross-compatible with HashiCorp Terraform, since that software
has chosen to have unqualified OCI mean Oracle's system, whereas OpenTofu
has previously established that unqualified OCI means "Open Container
Initiative" in our ecosystem.
In particular, this design makes it possible in principle to bring an
existing Terraform configuration specifying backend "oci" over to OpenTofu
without modifications, and then to optionally switch it to specifying
backend "oracle-oci" at a later time without a spurious prompt to migrate
state snapshots to the same physical location where they are already
stored.
This commit doesn't actually introduce any aliases and therefore doesn't
have any tests for the new mechanism because our backend system uses a
global table that isn't friendly to mocking for testing purposes. I've
tested this manually using a placeholder alias to have confidence that it
works, and I expect that a subsequent commit introducing the new
"oracle_oci" backend will also introduce its "oci" alias and will include
tests that cover use of the alias and migration from the alias to the
canonical name and vice-versa.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This extends statemgr.Persistent, statemgr.Locker and remote.Client to
all expect context.Context parameters, and then updates all of the existing
implementations of those interfaces to support them.
All of the calls to statemgr.Persistent and statemgr.Locker methods outside
of tests are consistently context.TODO() for now, because the caller
landscape of these interfaces has some complications:
1. statemgr.Locker is also used by the clistate package for its state
implementation that was derived from statemgr.Filesystem's predecessor,
even though what clistate manages is not actually "state" in the sense
of package statemgr. The callers of that are not yet ready to provide
real contexts.
In a future commit we'll either need to plumb context through to all of
the clistate callers, or continue the effort to separate statemgr from
clistate by introducing a clistate-specific "locker" API for it
to use instead.
2. We call statemgr.Persistent and statemgr.Locker methods in situations
where the active context might have already been cancelled, and so we'll
need to make sure to ignore cancellation when calling those.
This is mainly limited to PersistState and Unlock, since both need to
be able to complete after a cancellation, but there are various
codepaths that perform a Lock, Refresh, Persist, Unlock sequence and so
it isn't yet clear where is the best place to enforce the invariant that
Persist and Unlock must not be called with a cancelable context. We'll
deal with that more in subsequent commits.
Within the various state manager and remote client implementations the
contexts _are_ wired together as best as possible with how these subsystems
are already laid out, and so once we deal with the problems above and make
callers provide suitable contexts they should be able to reach all of the
leaf API clients that might want to generate OpenTelemetry traces.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Recently we added a call to Locks.UpgradeFromPredecessorProject to try to
preserve dependency selections made for providers under
registry.terraform.io/hashicorp/* when switching to OpenTofu for the first
time.
However, this behavior did not properly cater for the situation where the
configuration intentionally specifies registry.terraform.io explicitly in
a source address: that would then cause OpenTofu to incorrectly try to make
a factory function for the shimmed provider version when working in
command.Meta.providerFactories, which would then fail because no such
provider appears in the cache directory.
Instead then, we'll limit the shimming only to installation-related actions
while only using the dependency locks exactly as written when preparing to
actually _run_ the provider plugins.
This is bothersome to test because our tests are not allowed to directly
access registry.terraform.io; the test case here mimicks one case in which
it could be valid for an OpenTofu user to explicitly use
registry.terraform.io: if they've used the CLI configuration to arrange for
that hostname to be handled only via a mirror rather than by direct access
to the origin registry. The terms of service for registry.terraform.io
currently prohibit using it for anything other than Terraform, so we ensure
that this test cannot make requests to any real services at that hostname.
Note that telling OpenTofu to use registry.terraform.io is not officially
supported and may cause other problems beyond what was addressed by this
PR, since OpenTofu tends to assume that this hostname would appear only
during the process of migrating from Terraform and might make unexpected
decisions based on that assumption. Despite us making this fix, those who
are explicitly specifying registry.terraform.io in their configuration
should make plans to stop doing that and to set things up some other way
instead.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.StateMgr method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As discussed in opentofu/opentofu#2656, this consolidates the two concerns
of the PackageAuthentication interface into a single function that deals
both with package authentication _and_ with reporting all of the package
hashes that were used to make the authentication decision.
This means that any .zip archive that OpenTofu directly verifies during
installation can now have its hash recorded in the dependency lock file
even if that package didn't come from the provider's origin registry, which
is beneficial when the first installation of a provider comes from a
secondary ("mirror") source because it creates an additional hook by which
that dependency lock file entry can be "upgraded" to be complete in a
future "tofu init" run against the origin registry, or by the
"tofu providers lock" command.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
