We have a number of trace attributes that we use across all of our OCI
Distribution-based functionality, so this centralizes their definitions
in package traceattrs.
This intentionally ignores a few additional attribute names that are used
only in the code that interacts with Docker-style credential helpers,
because all of those are used only in a single function and so adding
indirection for those doesn't have enough benefit to offset the cost of
additional indirection.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously added re-exports for some of the functions we'd previously
been importing directly from semconv elsewhere in this codebase. For this
one we'd previously just hard-coded the standardized attribute name, but
for consistency we'll also use a re-export of a semconv function for this
one too.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds more detailed OTel trace spans to our various different
interactions with OCI repositories, which is helpful to understand the
time spent in each of the various sequential steps involved in resolving
an OCI artifact.
OTel's centrally-maintained conventions for attribute names currently only
have a standard for reporting a manifest digest, so we'll use that where
it's appropriate but use our own "opentofu.oci."-prefixed attribute names
for everything else for now. Hopefully the upstream standard will be
broadened later to include some additional concepts, at which point we
can switch over to the standardized attribute names.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Because the CLI Configuration allows specifying multiple provider
installation methods, we use getproviders.MultiSource to aggregate together
all of their results for a particular provider and then select the newest
available version from across all of the methods.
Different sources might offer a different subset of providers, and so
MultiSource ignores getproviders.ErrProviderNotFound errors from
Source.AvailableVersions unless _all_ sources return that error, which
therefore avoids the need for the operator to laboriously configure
include/exclude rules to constrain which real sources get queried for
each provider.
The OCIRegistryMirrorSource implements AvailableVersions by asking the
repository to list the tags it has available, and therefore this
source's implementation of the "provider not found" signal is based on
the registry returning a "not found" error from that specific request.
However, we were previously only handling the error code that ORAS-Go uses
for its local repository implementations, like the in-memory store or the
OCI layout store. For remote registries using the OCI Distribution protocol
ORAS-Go uses a different error type that is able to give more detail about
the server's response, and so we also need to treat some instances of
_that_ error type as representing "provider not found".
The OCI Distribution protocol does not require that a registry report a
JSON representation of the details of an error, but defines an optional
means of doing so that can give more detail than just the HTTP error
code. Therefore we use the OCI Distribution-specific error details if
possible, but fall back on just using the HTTP status code for servers
that choose not to return machine-readable error details. This compromise
is intended to allow us to report errors _other than_ "provider not found"
explicitly when possible, but to err on the side of allowing other sources
a chance to succeed if the situation is ambiguous. The ORAS-Go library
generates log lines for each request in our output to help with detailed
debugging in case we misclassify an error, but we'd prefer to give the
operator direct feedback about a configuration error if possible.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As discussed in opentofu/opentofu#2656, this consolidates the two concerns
of the PackageAuthentication interface into a single function that deals
both with package authentication _and_ with reporting all of the package
hashes that were used to make the authentication decision.
This means that any .zip archive that OpenTofu directly verifies during
installation can now have its hash recorded in the dependency lock file
even if that package didn't come from the provider's origin registry, which
is beneficial when the first installation of a provider comes from a
secondary ("mirror") source because it creates an additional hook by which
that dependency lock file entry can be "upgraded" to be complete in a
future "tofu init" run against the origin registry, or by the
"tofu providers lock" command.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The initial design of this attempted to maximize our flexibility to make
backward-compatible changes to the manifest structure for providers in
future OpenTofu releases, by expecting the "layers" entry describing the
provider package to include a specific artifactType and ignoring any
layers with different artifactType values.
However, general-purpose tools for constructing image manifests don't tend
to allow setting the artifactType in a layer descriptor, so we'll drop
that requirement as a measure of pragmatism.
This implies that the only possible extension we could make for _layers
in particular_ in a future release is to support an additional archive
format that would presumably then be used in much the same way as our
current archive/zip usage: by extracting the archive into the target
directory. This ability to introduce new formats is the most likely future
evolution we identified while designing this, with all other evolutions
being more speculative/theoretical.
This does still retain various other extension points for future additions,
including but not limited to:
- Introducing an alternative artifactType for the image manifest _itself_,
rather than for the layers within it, and then having the later version
of OpenTofu prefer to choose a manifest with the newer artifactType while
still supporting the old one as a fallback.
- Using the "config" property of the manifest to introduce arbitrary
additional metadata that future versions might need, and using the
config descriptor's own mediaType to recognize when those new additions
are present.
Therefore this seems like a reasonable compromise to make it easier to
assemble OCI manifests for OpenTofu provider packages using general-purpose
tools like ORAS CLI, rather than requiring OpenTofu-specific tools.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In practical testing I've found that some OCI registry implementations
don't return the artifactType field in their response to a tag resolution
request.
Unfortunately that behavior is ambiguous with what happens when a server
that _does_ support the artifact type describes a manifest that lacks one,
as is true for traditional container images and Helm chart artifacts.
Therefore we must unfortunately sacrifice our specialized error message
that tried to hint about accidentally selecting a container image, and
in that case we'll now return a more generic error about the manifest
having the wrong media type because that's the best we can do with the
information that an OCI registry is _required_ to return from tag
resolution.
Perhaps we'll try to improve on this in a different way later if we find
that folks are often getting confused by this, but for now we'll accept the
slightly-worse error message in this hopefully-rare case so we can
prioritize getting the happy path finished and shipped.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a new implementation of getproviders.Source that has a similar
purpose to HTTPMirrorSource but uses the OCI Distribution protocol instead
of the OpenTofu-specific Provider Mirror Protocol.
It translates OpenTofu-style provider source addresses into OCI repository
addresses using an externally-defined rule and then discovers provider
package metadata using the tags and associated manifests in the selected
repository.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is essentially a variation on PackageHTTPArchive that uses the
OCI Distribution protocol to retrieve a package from an OCI repository,
rather than directly from an HTTP URL.
It uses a similar technique of first downloading the package to a temporary
file and then delegating to the PackageLocalArchive location for the
final extraction, but it additionally verifies that the given OCI blob
descriptor seems reasonable and that the response matches the constraints
given in that descriptor.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we treated PackageLocation only as pure data, describing a
location from which something could be retrieved. Unexported functions in
package providercache then treated PackageLocation values as a closed
union dispatched using a type switch.
That strategy has been okay for our relatively-generic package locations
so far, but in a future commit we intend to add a new kind of package
location referring to a manifest in an OCI distribution repository, and
installing from _that_ will require a nontrivial amount of
OCI-distribution-specific logic that will be more tightly coupled to the
getproviders.Source that will return such locations, and so we're switching
to a model where _the location object itself_ is responsible for knowing
how to install a provider package from its location, as a method of
PackageLocation.
The implementation of installing from each location type now moves from
package providercache to package getproviders, which is arguably a better
thematic home for that functionality anyway.
For now these remain tested only indirectly through the tests in package
providercache, since we didn't previously have any independent tests for
these unexported functions. We might want to add more tightly-scoped unit
tests for these to package getproviders in future, but for now this is not
materially worse than it was before.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
