The Go runtime provides a number of configuration knobs that subtly change
its behavior, and we cannot control which of these is available over time
as we change which version of Go we're building with.
It's therefore possible that an OpenTofu user might be intentionally or
unintentionally relying on one of these settings for OpenTofu to work on
their system, in which case they would be broken if they upgraded to a
newer version of OpenTofu which uses a different Go version that no longer
supports that setting.
These log lines are intended to help us more quickly notice that
possibility if someone opens a bug report describing an unexpected behavior
change after upgrading to a new OpenTofu minor release series. We can ask
the reporter to share "TF_LOG=debug" output from both the previous and new
releases to compare, and then these log lines should appear in the older
version's output so we can search the Go codebase and issue tracker for
each of the mentioned names to learn if the handling of that setting has
changed between Go versions.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
InterestingDependencies is used to include version information for a small
subset of our dependencies whose behavior is directly exposed in the
OpenTofu UX, such as including a parser for some syntax that OpenTofu
relies on and that is covered by our compatibility promises.
Unfortunately we recently switched to using our own forks of certain
libraries and hadn't updated this to match, so the corresponding logs had
become less complete. This updates those to the dependencies we actually
use, and also adds a few new ones that have similar characteristics.
We'll also now skip calling InterestingDependencies altogether if the log
level isn't high enough for the results to be visible, since that avoids
us wasting time generating a data structure that will not be used.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
To facilitate early development and testing of the new language runtime
we're introducing a temporary mechanism to opt in to using the new codepaths
based on an environment variable. This environment variable is effective
only for experiment-enabled builds of OpenTofu, and so it will be
completely ignored by official releases of OpenTofu.
This commit just deals with the "wiring" of this new mechanism, without
actually connecting it with the new language runtime yet. The goal here
is to disturb existing codepaths as little as possible to minimize both
the risk of making this change and the burden this causes for ongoing
maintenance unrelated to work on the new language runtime.
This strategy of switching at the local backend layer means that we will
have some duplicated logic in the experimental functions compared to the
non-experimental functions, which is an intentional tradeoff to allow us
to isolate what we're doing so we don't churn existing code while we're
still in this early exploration phase. In a later phase of the language
runtime project we may pivot to a different approach which switches at
a deeper point in the call stack, but for now we're keeping this broad
to give us flexibility.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The GODEBUG environment variable is a side-channel that allows dynamically
opting in to a million billion different variations of Go runtime and
standard library behavior, and we can obviously not routinely test
OpenTofu's behavior across all of those different variations.
Just in case someone encounters a problem caused by running OpenTofu with
a combination of settings that are not enabled in our default build
configuration, we'll include an explicit note in the logs so that we can
tell when we're investigating a bug report that it might only be
reproducible when the Go runtime behavior has been overridden in this way,
and so that someone debugging their own problem can notice that they are
using OpenTofu in an unsupported way and could potentially solve their own
problem by unsetting that environment variable.
This is a generalization of the previous addition of a log message when
running in FIPS 140-3 mode, which is also updated here to use [WARN]
instead of [WARNING] as the prefix because our logging system does not
actually understand "WARNING" as a valid prefix. Keeping the separate
specialized message for FIPS 140-3 mode is warranted because we _know_
that OpenTofu does not behave as intended when that mode is enabled, while
we've not tested with any other combination of settings so we cannot
predict whether they will cause misbehavior or not.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Unfortunately the Go team has unilaterally decided that all programs built
with Go 1.24 and later always allow enabling FIPS-140-3 mode -- both in
its "on" and "only" configurations -- regardless of whether the authors
of that software intend to support running in that restricted mode, or
whether they are even testing their application in that configuration.
We have not yet made a final decision on how and whether we intend to
support this mode in our official builds, but we _do_ know that OpenTofu
cannot currently function correctly with this mode enabled because it
relies on standard library features and external libraries that are not
available in that case.
Therefore in the meantime we'll mention explicitly in both the internal
logs and in the "tofu version" output if we appear to be running in that
mode, meaning that if someone tries to use it and finds that it doesn't
work properly then if they open a GitHub issue and share those two
artifacts (as requested by our bug report template) then we can know that
we might need to turn on the special mode in order to reproduce the
reported problem, rather than wasting time trying to reproduce it in the
standard mode.
We do still need to make a final decision about what we want to do with
this in the long run, but this is intended as an short-term compromise
that allows folks to experiment with this unsupported mode if they wish
while hopefully making it clearer that in the mean time we may
deprioritize fixing problems that only occur when this unusual mode is
enabled.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We have a number of trace attributes that we use across all of our OCI
Distribution-based functionality, so this centralizes their definitions
in package traceattrs.
This intentionally ignores a few additional attribute names that are used
only in the code that interacts with Docker-style credential helpers,
because all of those are used only in a single function and so adding
indirection for those doesn't have enough benefit to offset the cost of
additional indirection.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Prevously OpenTofu delegated browser launching entirely to the third-party
module github.com/cli/browser, which consists of a number of
platform-specific lists of executable commands to try to run to launch
a web browser.
On Unix systems there is also a de-facto convention of using an environment
variable called BROWSER to explicitly specify what to launch. That variable
can either point directly to a browser, or can point to a script which
implements some more complex policy for choosing a browser, such as
detecting whether the command is running in a GUI context and launching
either a GUI or textmode browser.
The BROWSER variable has been most commonly implemented with similar
treatment to earlier variables like EDITOR and PAGER where it's expected
to be set to just a single command to run, with the URL given as the first
and only argument. There was also an attempt to define a more complex
interpretation of this variable at http://www.catb.org/~esr/BROWSER/ , but
that extended treatment was only implemented in a small amount of software,
and those which implemented it did so slightly inconsistently due to the
specification being ambiguous.
OpenTofu's implementation therefore follows the common simpler convention,
but will silently ignore variable values it cannot use so that OpenTofu
won't fail when run in an environment that has that variable set in a way
that's intended for use by some other software. In that case OpenTofu
will continue to perform the default behavior as implemented in the
third-party library.
Because this convention is Unix-specific, OpenTofu will check for and use
this environment variable only on operating systems that the Go toolchain
considers to be "unix". This means that in particular on Windows systems
OpenTofu will continue to follow the Windows convention of specifying
the default browser via an entry in the Windows Registry.
As usual with this sort of system-integration mechanism it isn't really
viable to test this end-to-end in a portable way, but the main logic is
separated out into testable functions, and I manually tested this on my
own Linux system to verify that it works in a real OpenTofu executable.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Go 1.17 and earlier used a different syntax for build constraint comments,
starting with "+build". Go 1.18 changed this to the modern "go:build" form
as part of standardizing the structure of toolchain directive comments,
and so for a while it was convention to include comments in both styles
to allow building with both old and new Go compilers.
However, Go 1.17 is no longer supported, and regardless of that we only
expect OpenTofu to be built with the specific version we have selected
in "go.mod" and ".go-version" anyway, so we no longer need the legacy form
of these comments: the all supported Go toolchains now support the new
form, which this commit retains.
golangci-lint v2.6.0 includes a check for this legacy form, so removing
this will also allow us to upgrade to a newer version of that linter
aggregator in a future commit.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These were previously settable only via environment variables. These are
now handled as part of CLI Configuration and so also settable in a new
"registry_protocols" block in a CLI configuration file, with the
environment variables now treated as if they are an additional virtual
configuration file containing the corresponding settings.
This handles our settings in our modern style where package cliconfig is
responsible for deciding the configuration and then package main reacts
to that configuration without being aware of how it is decided.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we generated some logs during the discovery process indicating
which locations OpenTofu was probing for ambient credentials, but we didn't
explicitly report the overall result of the discovery process.
These new log lines will now report the final effective set of credential
configuration locations just before we try to use them in either the
provider installation or module installation codepaths. The strings
returned by CredentialsConfigLocationForUI are intended for just this sort
of feedback: the exact format varies for each kind of location, but it's
always a concise string identifying a location that OpenTofu will consider
when attempting to decide credentials.
Logging this here does unfortunately mean that the log output will be
repeated for each separate OCI registry request. There not being a great
single location to generate these logs was the main reason we didn't
include something like this in the first implementation, but the set of
config locations is small on any reasonable system and we've already had
a few folks struggle to understand why OpenTofu is making a certain
decision about credential sources so this is a pragmatic small step to give
us some extra diagnostic information in bug reports without affecting the
normal UI output for now.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The disco.Disco API isn't yet set up to pass through context.Context, and
so if we give it a HTTP client that has the OpenTelemetry instrumentation
on it then any HTTP request causes an orphan trace span disconnected from
the main trace, which causes annoying noise in some trace viewers.
As a temporary solution so we can ship v1.10 soon without making large
changes to the svchost library, we'll prevent the HTTP client constructor
function from detecting that tracing is enabled by passing it
context.TODO() instead of the actual context. This would not be acceptable
in the long run but is safe for this temporary workaround because currently
httpclient.New doesn't use the given context for anything except detecting
whether tracing is enabled.
We will address this in a more complete way during the v1.11 development
period by modernizing svchost to take context.Context arguments on all
functions that can potentially make external requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds more detailed OTel trace spans to our various different
interactions with OCI repositories, which is helpful to understand the
time spent in each of the various sequential steps involved in resolving
an OCI artifact.
OTel's centrally-maintained conventions for attribute names currently only
have a standard for reporting a manifest digest, so we'll use that where
it's appropriate but use our own "opentofu.oci."-prefixed attribute names
for everything else for now. Hopefully the upstream standard will be
broadened later to include some additional concepts, at which point we
can switch over to the standardized attribute names.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The ORAS-Go library's remote Repository implementation hides from us the
implementation detail that some registries require trading credentials
for a bearer token before using the bearer token to authenticate all
subsequent requests. In that case, the bearer token gets cached inside
the repository object to allow reusing it for subsequent requests until
the token expires, at which point it automatically issues itself a new
token.
Our provider installation process ends up calling getOCIRepositoryStore
multiple times, since we have three independent steps that all need to
interact with the repository: find available versions, get the metadata
for a selected version, and then get the actual package for a selected
version on a specific platform. Therefore it's beneficial to keep a cache
of previously-returned repository objects and reuse them when we get asked
for the same repository again.
This is not so beneficial for modules since we do all of the module package
installation steps in one pass with a single client, but these objects are
relatively cheap so not a big deal to retain them and a client will only
get created by commands that install modules, like "tofu init", anyway.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Our OCI credentials policy layer expects to be provided with an
implementation of the Docker credential helper protocol as part of its
"credentials lookup environment".
Since we're already using ORAS-Go for everything else we'll just wrap their
implementation of this protocol here too, and then translate the result
into our own type since we've been intentionally avoiding making ORAS-Go
types part of any of our exported package APIs.
Because this is the concrete implementation of an interface we introduced
so that unit tests elsewhere could fake it, it's pretty awkward to fully
test this implementation without the overhead of having a test build its
own credential helper executable dynamically to run on the platform where
the test program is running. ORAS-Go already has its own tests for this
functionality, so as a pragmatic compromise here we just focus on testing
that we're attempting to run the executable that the protocol expects us
to execute, but detecting that through an error result rather than through
a success result.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These completes the wiring of the OCI credentials policy into the "package
fetcher" component of the module installer. The module installer does not
yet make any use of this, but a future commit will introduce a new "oci"
source address scheme that will make use of this.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These functions and types are about to become cross-cutting concerns shared
between both the provider and module installers in a future commit, so
we'll move them out to a separate file to hopefully communicate the
relationships between these parts a little more clearly.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This continues our work to follow the dependency inversion style for the
"package fetcher" component of the module installer.
Mimicking the existing pattern for providers, package main is now
responsible for instantiating the PackageFetcher and providing it to
the "command" package as a field of command.Meta.
We could potentially go further here and follow dependency inversion style
for _all_ of the special clients needed by the various go-getter getters,
but our primary concern for now is preparing to add a new "getter" for
installation from an OCI Distribution repository, and so we'll leave the
other already-working code unchanged to reduce the risk of this initial
work.
Future commits will actually wire in the implementation details for OCI
Repository access. This commit focuses only on plumbing the necessary
objects through the API layers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In a future commit we'll introduce a new provider source that can use an
OCI registry as a new kind of provider mirror, but this commit is just to
illustrate how we'd get the needed credentials settings to that point using
our typical dependency inversion style, ending in a TODO comment that we'll
resolve later.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We seem to have inherited an incomplete implementation of something from
the predecessor project here: a "tofu cloud" command that just tries to
immediately delegate any invocation to another executable called
"terraform-cloudplugin" in the current working directory, used as a
go-plugin style plugin.
This has some TODO comments suggesting that it was intended to change to
download a plugin from some remote place before executing it, but our
stubby version doesn't do that. I was also hidden behind an experimental
feature guard and so has never been accessible in any released version of
OpenTofu; we don't currently produce any releases with experimental
features enabled.
Therefore this commit just deletes it so that we don't have this dead code
to potentially worry about. Perhaps one day we'll offer some extension
point for adding extra subcommands through plugins, but if we do that then
we'll presumably design our own mechanism for doing it rather than
extending this dead code that was added for reasons unknown to us.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
