Previously we generated some logs during the discovery process indicating
which locations OpenTofu was probing for ambient credentials, but we didn't
explicitly report the overall result of the discovery process.
These new log lines will now report the final effective set of credential
configuration locations just before we try to use them in either the
provider installation or module installation codepaths. The strings
returned by CredentialsConfigLocationForUI are intended for just this sort
of feedback: the exact format varies for each kind of location, but it's
always a concise string identifying a location that OpenTofu will consider
when attempting to decide credentials.
Logging this here does unfortunately mean that the log output will be
repeated for each separate OCI registry request. There not being a great
single location to generate these logs was the main reason we didn't
include something like this in the first implementation, but the set of
config locations is small on any reasonable system and we've already had
a few folks struggle to understand why OpenTofu is making a certain
decision about credential sources so this is a pragmatic small step to give
us some extra diagnostic information in bug reports without affecting the
normal UI output for now.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The disco.Disco API isn't yet set up to pass through context.Context, and
so if we give it a HTTP client that has the OpenTelemetry instrumentation
on it then any HTTP request causes an orphan trace span disconnected from
the main trace, which causes annoying noise in some trace viewers.
As a temporary solution so we can ship v1.10 soon without making large
changes to the svchost library, we'll prevent the HTTP client constructor
function from detecting that tracing is enabled by passing it
context.TODO() instead of the actual context. This would not be acceptable
in the long run but is safe for this temporary workaround because currently
httpclient.New doesn't use the given context for anything except detecting
whether tracing is enabled.
We will address this in a more complete way during the v1.11 development
period by modernizing svchost to take context.Context arguments on all
functions that can potentially make external requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds more detailed OTel trace spans to our various different
interactions with OCI repositories, which is helpful to understand the
time spent in each of the various sequential steps involved in resolving
an OCI artifact.
OTel's centrally-maintained conventions for attribute names currently only
have a standard for reporting a manifest digest, so we'll use that where
it's appropriate but use our own "opentofu.oci."-prefixed attribute names
for everything else for now. Hopefully the upstream standard will be
broadened later to include some additional concepts, at which point we
can switch over to the standardized attribute names.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The ORAS-Go library's remote Repository implementation hides from us the
implementation detail that some registries require trading credentials
for a bearer token before using the bearer token to authenticate all
subsequent requests. In that case, the bearer token gets cached inside
the repository object to allow reusing it for subsequent requests until
the token expires, at which point it automatically issues itself a new
token.
Our provider installation process ends up calling getOCIRepositoryStore
multiple times, since we have three independent steps that all need to
interact with the repository: find available versions, get the metadata
for a selected version, and then get the actual package for a selected
version on a specific platform. Therefore it's beneficial to keep a cache
of previously-returned repository objects and reuse them when we get asked
for the same repository again.
This is not so beneficial for modules since we do all of the module package
installation steps in one pass with a single client, but these objects are
relatively cheap so not a big deal to retain them and a client will only
get created by commands that install modules, like "tofu init", anyway.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Our OCI credentials policy layer expects to be provided with an
implementation of the Docker credential helper protocol as part of its
"credentials lookup environment".
Since we're already using ORAS-Go for everything else we'll just wrap their
implementation of this protocol here too, and then translate the result
into our own type since we've been intentionally avoiding making ORAS-Go
types part of any of our exported package APIs.
Because this is the concrete implementation of an interface we introduced
so that unit tests elsewhere could fake it, it's pretty awkward to fully
test this implementation without the overhead of having a test build its
own credential helper executable dynamically to run on the platform where
the test program is running. ORAS-Go already has its own tests for this
functionality, so as a pragmatic compromise here we just focus on testing
that we're attempting to run the executable that the protocol expects us
to execute, but detecting that through an error result rather than through
a success result.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These completes the wiring of the OCI credentials policy into the "package
fetcher" component of the module installer. The module installer does not
yet make any use of this, but a future commit will introduce a new "oci"
source address scheme that will make use of this.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These functions and types are about to become cross-cutting concerns shared
between both the provider and module installers in a future commit, so
we'll move them out to a separate file to hopefully communicate the
relationships between these parts a little more clearly.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This continues our work to follow the dependency inversion style for the
"package fetcher" component of the module installer.
Mimicking the existing pattern for providers, package main is now
responsible for instantiating the PackageFetcher and providing it to
the "command" package as a field of command.Meta.
We could potentially go further here and follow dependency inversion style
for _all_ of the special clients needed by the various go-getter getters,
but our primary concern for now is preparing to add a new "getter" for
installation from an OCI Distribution repository, and so we'll leave the
other already-working code unchanged to reduce the risk of this initial
work.
Future commits will actually wire in the implementation details for OCI
Repository access. This commit focuses only on plumbing the necessary
objects through the API layers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In a future commit we'll introduce a new provider source that can use an
OCI registry as a new kind of provider mirror, but this commit is just to
illustrate how we'd get the needed credentials settings to that point using
our typical dependency inversion style, ending in a TODO comment that we'll
resolve later.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We seem to have inherited an incomplete implementation of something from
the predecessor project here: a "tofu cloud" command that just tries to
immediately delegate any invocation to another executable called
"terraform-cloudplugin" in the current working directory, used as a
go-plugin style plugin.
This has some TODO comments suggesting that it was intended to change to
download a plugin from some remote place before executing it, but our
stubby version doesn't do that. I was also hidden behind an experimental
feature guard and so has never been accessible in any released version of
OpenTofu; we don't currently produce any releases with experimental
features enabled.
Therefore this commit just deletes it so that we don't have this dead code
to potentially worry about. Perhaps one day we'll offer some extension
point for adding extra subcommands through plugins, but if we do that then
we'll presumably design our own mechanism for doing it rather than
extending this dead code that was added for reasons unknown to us.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
