The human-oriented plan output includes a short summary of the meaning of
all of the different "icons" used to describe different kinds of change,
but nothing was previously describing how OpenTofu uses "->" to describe
the transition between current and planned values for update in-place, "~".
We'll now include a concise note about that as part of the icon summary,
keeping things still relatively compact but nonetheless giving something
to refer to if a reader is unsure about the meaning of this notation.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The JSON object describing an input variable can now include two additional
properties:
- "type" provides a JSON representation of the variable's type constraint,
if one is set. Omitted if either there is no constraint declared at all
or if it's set to "any", which are equivalent and both mean that the
type is completely unconstrained.
This uses the standard cty representation of a type constraint, which
matches how OpenTofu already describes types in the provider protocol,
in state snapshots, and in saved plan files.
- "required" directly represents whether callers are required to provide
a value for the variable. This is technically redundant since it is
set to true unless "default" is also set, but this avoids the need for
consuming software to reimplement this rule and potentially allows us to
make this rule more complicated/subtle in future if needed.
For some reason the documentation about the JSON configuration
representation did not previously mention the "variables" property at all,
so this adds documentation for both the new properties and the pre-existing
properties.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Since Go 1.18, "any" is the preferred spelling of the empty interface type.
This quiets some complaints from linters that the OpenTofu project does
not enforce but nonetheless may be enabled in the Go support for certain
text editors.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously added the -config mode for showing the entire assembled
configuration tree, including the content of any descendent modules, but
that mode requires first running "tofu init" to install all of the
provider and module dependencies of the configuration.
This new -module=DIR mode returns a subset of the same JSON representation
for only a single module that can be generated without first installing
any dependencies, making this mode more appropriate for situations like
generating documentation for a single module when importing it into the
OpenTofu Registry. The registry generation process does not want to endure
the overhead of installing other providers and modules when all it actually
needs is metadata about the top-level declarations in the module.
To minimize the risk to the already-working full-config JSON representation
while still reusing most of its code, the implementation details of package
jsonconfig are a little awkward here. Since this code changes relatively
infrequently and is implementing an external interface subject to
compatibility constraints, and since this new behavior is relatively
marginal and intended primarily for our own OpenTofu Registry purposes,
this is a pragmatic tradeoff that is hopefully compensated for well enough
by the code comments that aim to explain what's going on for the benefit
of future maintainers. If we _do_ find ourselves making substantial changes
to this code at a later date then we can consider a more significant
restructure of the code at that point; the weird stuff is intentionally
encapsulated inside package jsonconfig so it can change later without
changing any callers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Due to some past confusion about the purpose of this package, it has grown
to include a confusing mix of currently-viable code and legacy support
code from the move to HCL 2. This has in turn caused confusion about which
parts of this package _should_ be used for new code.
To help clarify that distinction we'll move the legacy support code into
a package under the "legacy" directory, which is also where most of its
callers live.
There are unfortunately still some callers to these outside of the legacy
tree, but the vast majority are either old tests written before HCL 2
adoption or helper code used only by those tests. The one dubious exception
is the use in ResourceInstanceObjectSrc.Decode, which makes a best effort
to shim flatmap as a concession to the fact that not all state-loading
codepaths are able to run the provider state upgrade function that would
normally be responsible for the flatmap-to-JSON conversion, which is
explained in a new comment inline.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This extends statemgr.Persistent, statemgr.Locker and remote.Client to
all expect context.Context parameters, and then updates all of the existing
implementations of those interfaces to support them.
All of the calls to statemgr.Persistent and statemgr.Locker methods outside
of tests are consistently context.TODO() for now, because the caller
landscape of these interfaces has some complications:
1. statemgr.Locker is also used by the clistate package for its state
implementation that was derived from statemgr.Filesystem's predecessor,
even though what clistate manages is not actually "state" in the sense
of package statemgr. The callers of that are not yet ready to provide
real contexts.
In a future commit we'll either need to plumb context through to all of
the clistate callers, or continue the effort to separate statemgr from
clistate by introducing a clistate-specific "locker" API for it
to use instead.
2. We call statemgr.Persistent and statemgr.Locker methods in situations
where the active context might have already been cancelled, and so we'll
need to make sure to ignore cancellation when calling those.
This is mainly limited to PersistState and Unlock, since both need to
be able to complete after a cancellation, but there are various
codepaths that perform a Lock, Refresh, Persist, Unlock sequence and so
it isn't yet clear where is the best place to enforce the invariant that
Persist and Unlock must not be called with a cancelable context. We'll
deal with that more in subsequent commits.
Within the various state manager and remote client implementations the
contexts _are_ wired together as best as possible with how these subsystems
are already laid out, and so once we deal with the problems above and make
callers provide suitable contexts they should be able to reach all of the
leaf API clients that might want to generate OpenTelemetry traces.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The code for matching relevant_attributes against resource_drift entries
(a part of the heuristic for deciding whether to show "changes outside of
OpenTofu" in the human-oriented plan UI) was previously assuming that paths
in resource_drift would always be valid for the associated resource
instance object values because in most cases the language runtime will
detect invalid references and so fail to generate a plan at all.
However, when the reference is to something within a dynamically-typed
argument (such as the manifest in kubernetes_manifest) and when it appears
only as an argument to either the "try" or "can" functions (so the dynamic
error is intentionally suppressed) the language runtime can't catch it
and so the incorrect reference will leak out into relevant_attributes,
thereby violating assumptions made by the path matcher.
Instead then, we'll continue the existing precedent that this "relevant
attributes" mechanism is a best-effort heuristic that prefers to succeed
with an incomplete result rather than to fail, extending that to the
traversals in the plan renderer which will now treat incorrectly-typed
steps as not matching rather than causing OpenTofu to crash completely.
Since a reference to something that doesn't exist cannot succeed it also
cannot possibly _actually_ contribute directly to the final result of the
expression it appeared in, so in practice it should be okay to disregard
these invalid references for the purposes of deciding which changes outside
of OpenTofu seem likely to have caused the actions that OpenTofu is
proposing to make during the apply phase.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Recently we added a call to Locks.UpgradeFromPredecessorProject to try to
preserve dependency selections made for providers under
registry.terraform.io/hashicorp/* when switching to OpenTofu for the first
time.
However, this behavior did not properly cater for the situation where the
configuration intentionally specifies registry.terraform.io explicitly in
a source address: that would then cause OpenTofu to incorrectly try to make
a factory function for the shimmed provider version when working in
command.Meta.providerFactories, which would then fail because no such
provider appears in the cache directory.
Instead then, we'll limit the shimming only to installation-related actions
while only using the dependency locks exactly as written when preparing to
actually _run_ the provider plugins.
This is bothersome to test because our tests are not allowed to directly
access registry.terraform.io; the test case here mimicks one case in which
it could be valid for an OpenTofu user to explicitly use
registry.terraform.io: if they've used the CLI configuration to arrange for
that hostname to be handled only via a mirror rather than by direct access
to the origin registry. The terms of service for registry.terraform.io
currently prohibit using it for anything other than Terraform, so we ensure
that this test cannot make requests to any real services at that hostname.
Note that telling OpenTofu to use registry.terraform.io is not officially
supported and may cause other problems beyond what was addressed by this
PR, since OpenTofu tends to assume that this hostname would appear only
during the process of migrating from Terraform and might make unexpected
decisions based on that assumption. Despite us making this fix, those who
are explicitly specifying registry.terraform.io in their configuration
should make plans to stop doing that and to set things up some other way
instead.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Continuing our work to gradually plumb context.Context to everywhere that
we want to generate OpenTelemetry traces, this completes the call path
for most (but not all) of the gRPC requests to provider plugins, so that
we can add OpenTelemetry trace instrumentation in a future commit.
Unfortunately there are still a few providers.Interface callers left in
functions that don't have context.Context plumbed to them yet, and so
those are temporarily stubbed as context.TODO() here so we can more easily
find and complete them later.
The two gRPC implementations of providers.Interface were previously making
provider requests using a single context.Context established at the time
the provider process was started, but that isn't an appropriate context
to use for per-request concerns like tracing, so that context is now
unused and could potentially be removed in a future commit, but this change
already got pretty large and so I intend to deal with that separately
later.
This now exposes the gRPC provider calls to potential context cancellation
that they would previously observe only indirectly though the Stop method.
Since Stop is primarily used for graceful shutdown of ApplyResourceChange,
the changes here explicitly disconnect the cancellation signal for
ApplyResourceChange in particular, while letting the others get canceled
in the normal way since they are expected to be free of significant
side-effects. In future work we could consider removing Stop from the
internal API entirely and keeping it only as an implementation detail of
the gRPC implementation of this interface, with ApplyResourceChange
directly reacting to context cancellation and sending the gRPC Stop call
itself, but again that's too much change for this already-large commit.
The internal/legacy package currently contains some legacy code preserved
for the benefit of the backends, and unfortunately contains more than is
strictly necessary to support those callers, and so there was some dead
code there that also needed updating. provider_mock.go is removed entirely
because it's just an older copy of the similar file in package tofu. The
few calls to providers in schemas.go are updated to use
context.Background() rather than context.TODO() because we have no
intention of plumbing context.Context into that legacy code, and will
hopefully just delete it wholesale one day.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As part of our ongoing work to plumb cross-cutting concerns like tracing
spans into the core language runtime, here we change the exported API
of the context.Schemas method to take a context.Context, and trivially
update all of the callers to pass in a suitable context.
Earlier work on this means that we don't have fix up too many call stack
levels before we already have a suitable context.Context value to use.
The Schemas method doesn't yet make any use of its new context.Context, but
that will follow in subsequent PRs.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.DeleteWorkspace
method, updates all of the implementations to match, and then updates all
of the callers to pass in a context.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.StateMgr method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.Workspaces method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
