Recently we added a call to Locks.UpgradeFromPredecessorProject to try to
preserve dependency selections made for providers under
registry.terraform.io/hashicorp/* when switching to OpenTofu for the first
time.
However, this behavior did not properly cater for the situation where the
configuration intentionally specifies registry.terraform.io explicitly in
a source address: that would then cause OpenTofu to incorrectly try to make
a factory function for the shimmed provider version when working in
command.Meta.providerFactories, which would then fail because no such
provider appears in the cache directory.
Instead then, we'll limit the shimming only to installation-related actions
while only using the dependency locks exactly as written when preparing to
actually _run_ the provider plugins.
This is bothersome to test because our tests are not allowed to directly
access registry.terraform.io; the test case here mimicks one case in which
it could be valid for an OpenTofu user to explicitly use
registry.terraform.io: if they've used the CLI configuration to arrange for
that hostname to be handled only via a mirror rather than by direct access
to the origin registry. The terms of service for registry.terraform.io
currently prohibit using it for anything other than Terraform, so we ensure
that this test cannot make requests to any real services at that hostname.
Note that telling OpenTofu to use registry.terraform.io is not officially
supported and may cause other problems beyond what was addressed by this
PR, since OpenTofu tends to assume that this hostname would appear only
during the process of migrating from Terraform and might make unexpected
decisions based on that assumption. Despite us making this fix, those who
are explicitly specifying registry.terraform.io in their configuration
should make plans to stop doing that and to set things up some other way
instead.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As part of our ongoing work to plumb cross-cutting concerns like tracing
spans into the core language runtime, here we change the exported API
of the context.Schemas method to take a context.Context, and trivially
update all of the callers to pass in a suitable context.
Earlier work on this means that we don't have fix up too many call stack
levels before we already have a suitable context.Context value to use.
The Schemas method doesn't yet make any use of its new context.Context, but
that will follow in subsequent PRs.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.DeleteWorkspace
method, updates all of the implementations to match, and then updates all
of the callers to pass in a context.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.StateMgr method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.Workspaces method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.Configure method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This test uses a temporary file as an overridden CLI configuration to
force using a specific plugin cache directory, but temporary file paths
contain backslashes on Windows and the CLI configuration syntax is HCL
so would require backslashes to be escaped.
Since Windows will accept forward-slash paths as a supported variation,
and we typically recommend that folks write paths that way in HCL for
portability anyway, this uses filepath.ToSlash to force consistent use
of slashes on all platforms. It would also have been reasonable to use
a %q format verb to use Go's string quoting syntax, but that's not the
way we typically recommend folks hand-write their CLI configurations and
so this way is just-so-slightly more "realistic".
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Our handling of Docker-style configuration files as an authentication
source is intentionally a bit of a compromise because various other
software reads and writes these files despite there being no single
standard for the format, and unfortunately different software makes
different tradeoffs when the configuration is ambiguous.
One oddity that we didn't notice originally is that the "login" command of
some programs will respect the "credsStore" property for storing new
credentials using a helper program instead of storing them in cleartext
in the config file BUT will still create an empty entry in the "auths"
property for whatever domain the operator logged into. Our logic wasn't
built to tolerate an "auths" entry without an "auth" property inside it
and so we would then fail to select credentials correctly for the affected
domain.
This commit makes our handling a little more resilient against
oddly-generated configuration files by silently ignoring all three of the
following oddities:
- An "auths" entry that has no "auth" property, as described above.
- An "auths" entry that is JSON null, which would previously cause OpenTofu
to crash with a null pointer dereference.
- An "auths" entry with "auth" set to an empty string, since generating
that instead of omitting the property entirely is a relatively common
mistake when using Go's encoding/json library and forgetting to add
the special "omitempty" tag to the corresponding struct field.
(An empty string is never a valid value for this property because it's
supposed to be the base64 encoding of a string like "username:password",
and so it should always at least contain a base64-encoded colon.)
Since there is no plausible valid meaning of any of these odd constructions
we prefer to just silently ignore them without any errors and without
generating any distracting log noise, which seems to match how other
software like Docker CLI and ORAS CLI handles them.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The "tofu show" command has historically been difficult to extend to meet
new use-cases, such as showing the current configuration without creating
a plan, because it was designed to take zero or one arguments and then try
to guess what the one specified argument was intended to mean.
This commit introduces a new style where the type of object to inspect is
specified using command line option syntax, using one of two
mutually-exclusive options:
-state Show the latest state snapshot.
-plan=FILE Show the plan from the given saved plan file.
We expect that a future commit will extend this with a new "-config" option
to inspect the configuration rooted in the current working directory, and
possibly with "-module=DIR" to shallowly inspect a single module without
necessarily having to fully initialize it with all of its dependencies
first. However, both of those use-cases (and any others) are not in scope
for this commit, which is focused only on refactoring to make those future
use-cases possible.
The old mode of specifying neither option and providing zero or one
positional arguments is still supported for backward compatibility.
Notably, the legacy style is the only way to access the legacy behavior of
inspecting a specific state snapshot file from the local filesystem, which
has not often been used since Terraform v0.9 as we've moved away
from manual management of state files to the structure of state backends.
Those who _do_ still need that old behavior can still access it in the
old way, but there will be no new-style equivalent of it unless we learn
of a compelling use case for it.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Following the lead of similar earlier work on testing the installation of
provider packages from OCI repositories, this new test exercises the new
OCI-based module source address syntax in an end-to-end fashion by directly
running "tofu init".
For the reasons described inline, this test uses a local test server as its
target OCI Registry and therefore needs to rely on a Go standard library
feature for overriding the trusted TLS certs which only works on Unix
systems other than macOS, and therefore this test will only run when the
e2etest suite is run on Linux systems. This matches the same compromise we
previously made for the provider installation flavor of this test, with
the same assumption that our module installer isn't doing anything
particularly platform-specific and that we're doing this in e2etest only
because that's an effective way to test that "package main" is wiring all
of the internal components together correctly.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously had two tests of how the module installer responds to
cancellation (e.g. SIGINT) which were flakey because they tried to rely
on the cancellation being detected at some arbitrary point before the
module installer attempted to make a request, which isn't guaranteed in
practice because our interrupt mechanism only aims to cause OpenTofu to
exit "soon", with no guarantee about how much ongoing progress it will
make before it does.
To make these tests more robust, we'll now instead tell the module
installer to install from a real HTTP server that is intentionally designed
to stall the client by accepting its request but then just leaving the
connection open without responding.
This means that we can now test the more realistic situation of the cancel
signal being triggered after a slow request is already in progress, and
be sure that we're definitely sending the cancel signal at a moment that
matches that intention.
This is similar to a strategy we previously took to improve the reliability
of the tests for cancellation of the _provider_ installer, in
TestInit_cancelProviders. However, our provider installer version of this
used an intentionally-stalling implementation of getproviders.Source
instead of running a real server because the provider installer is designed
to support configurable installation methods, while the module installer
is not: its policy about what module source types are accepted is
hard-coded in package getproviders, at least for now.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This continues our work to follow the dependency inversion style for the
"package fetcher" component of the module installer.
Mimicking the existing pattern for providers, package main is now
responsible for instantiating the PackageFetcher and providing it to
the "command" package as a field of command.Meta.
We could potentially go further here and follow dependency inversion style
for _all_ of the special clients needed by the various go-getter getters,
but our primary concern for now is preparing to add a new "getter" for
installation from an OCI Distribution repository, and so we'll leave the
other already-working code unchanged to reduce the risk of this initial
work.
Future commits will actually wire in the implementation details for OCI
Repository access. This commit focuses only on plumbing the necessary
objects through the API layers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Earlier work started to reshape this API to follow the dependency inversion
style, but didn't go so far as treating the package fetcher as an argument
because so far it hasn't offered any customizable policy anyway.
In future commits we will be introducing some policy arguments for the
package fetcher, and so this is some preparation work where we move the
responsibility for calling getmodules.NewPackageFetcher() out into the
caller of initwd.NewModuleInstaller().
This changes the API consumed by a bunch of unit testing helpers, so
splitting this out into its own commit should hopefully make future
commits more focused. The module installer now explicitly supports being
instantiated without a registry client or a remote package fetcher and
will in that case return an error if it's asked to install from anywhere
other than local relative directories. Most of our existing tests are
comfortable running under that constraint and so will not need any further
work in later commits that will change the signature of
getmodules.NewPackageFetcher.
However, a couple tests in package initwd _itself_ were making use of the
esoteric legacy support for treating an absolute filesystem path as a funny
sort of remote source, and so for now those will instantiate their own
package fetcher. Future commits that change the NewPackageFetcher signature
will need to offer a concession for those two tests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As discussed in opentofu/opentofu#2656, this consolidates the two concerns
of the PackageAuthentication interface into a single function that deals
both with package authentication _and_ with reporting all of the package
hashes that were used to make the authentication decision.
This means that any .zip archive that OpenTofu directly verifies during
installation can now have its hash recorded in the dependency lock file
even if that package didn't come from the provider's origin registry, which
is beneficial when the first installation of a provider comes from a
secondary ("mirror") source because it creates an additional hook by which
that dependency lock file entry can be "upgraded" to be complete in a
future "tofu init" run against the origin registry, or by the
"tofu providers lock" command.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Most of the OCI registry interactions are unit tested in the most relevant
packages, but the overall system will only work correctly if all of the
components are correctly wired together by "package main", and that's one
part of the system that needs to be tested concretely rather than via
test doubles.
Therefore this adds an end-to-end test in our existing e2etest package
that runs "tofu init" with a CLI configuration that forces using an OCI
mirror with a TLS server provided locally by our test program. It exercises
the main happy path of provider installation in the same way that an
end-user would interact with it, to help avoid accidentally regressing
the interactions between these packages in future versions.
Unfortunately the technique this test uses to force the OpenTofu CLI
binary to trust the test server doesn't work on macOS or Windows and so
for now this test is Linux-specific. That's certainly non-ideal, but
pragmatic since we'll be relying mainly on the platform-agnostic unit tests
to cover this behavior, and we're unlikely to ever stop running the
e2etests on Linux as part of our pull request checks so even those
developing on macOS or Windows can still notice if this test becomes
broken before merging a change.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a minimal implementation of the read-only parts of the OCI
Distribution spec that OpenTofu uses for dependency installation. It's not
intended as a complete registry implementation.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This new method collects all of the various different settings that
describe the operator's chosen OCI credentials policy and returns a single
object that encapsulates that policy.
This is the method that will, in future commits, be used by package main
to provide the credentials policy to any OCI-registry-related subsystems
using dependency-inversion style.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We're not actually doing anything with the result of this yet, but this
covers the decoding and validation of this new block type which we'll start
making use of in subsequent commits as we start to bring together the
overall OCI credentials selection policy handling.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This was previously an unexported function used only for implementing the
Docker/Podman/etc auth format, but we'll be using the same syntax in the
labels of our CLI configuration language's "oci_credentials" blocks and
so we'll export it to allow use from package cliconfig.
The name also changes to use the "ContainersAuth" prefix rather than the
"DockerCLIStyle" prefix because we are technically implementing the
expanded form of this configuration format that supports repository path
matching, rather than the original form that Docker CLI pioneered. This
new format doesn't really have a catchy brand name, but it's documented
as "containers auth.json" and so "ContainersAuth" seems like a reasonable
compromise at following that name while keeping this concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we just required the slash-separated segments to match without
imposing any further constraint, but if the Docker-style config syntax
evolves to allow other syntaxes here in future it'd be better for us to
just ignore what we don't recognize rather than get confused into trying
to match it in the current way.
ParseRepositoryAddressPrefix is an exported function because package
cliconfig will use it in a future commit to deal with our OpenTofu-specific
equivalent of the "auths" objects: oci_credentials blocks in the CLI
configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
