mockgcp

package module
v0.0.0-...-c007ca4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 28, 2026 License: Apache-2.0 Imports: 117 Imported by: 0

README

MockGCP

MockGCP contains stub implementations of GCP APIs, for use with testing Config Connector. The implementations don't do anything (there is no "backend") - creating a mock VM does not launch a VM. Instead, the goal is to provide just-enough consistent CRUD operations to test the Config Connector providers. If the tests pass, we consider that to be a "good enough" mock for GCP. When we discover an issue that the mocks failed to catch, we should update mockgcp to more accurately simulate GCP, ideally triggering the bug and then showing that the fix addresses the issue.

At their core, most GCP APIs are CRUD-based, so implementing a basic implementation is fairly straightforward. We are establishing some patterns to make the implementation as simple as possible, such as a simple storage driver that works with the existing protos.

In general, we use the protos published by GCP at https://github.com/googleapis/googleapis.git. These protos make it relatively easy to implement the CRUD operations needed.

In addition, the protos have the annotations needed for GRPC <-> HTTP interop. We use grpc-gateway to create HTTP servers from GRPC servers.

It is then fairly straightforward to inject our own HTTP handlers directly into the serving paths for tests, so we don't even need to start a real webserver (i.e. we don't even need to listen on a port)

Adding a new service

To add a new service, the easiest way is to copy one of the existing PRs. Example PR: mockGCP for cloudfunctions

Broadly the steps are:

  1. Add the proto to the Makefile and run make gen-proto (or just make).

    All google services are located in googleapis GitHub repo, refer to your resource's API documentation to identify the service name, for example privateca. Once you identify the service, find the proper path to the proto files, for example: cloud/security/privateca/v1/*.proto. Then replace the prefix googleapis/google/ to ./third_party/googleapis/mockgcp/, and add into the Makefile.

    • Note: Ensure you pick the same version the controller uses to call the GCP API during GCP resource instantiation.

    • Note: If the Config Connector resource does not map to a GCP resource, please refer to the REST API of the relevant GCP resource and find the proto file that covers the resource's API endpoints.(You may not find a proto file that matches the name of that Config Connector resource)

    • (Optional) If you determine that the proto file is not up to date, or if it doesn't exist at all, refer to the Generating Proto section

  2. (Optional). If you're adding an API outside of googleapis/google/cloud, you may need to add commands to rename the API to mockgcp in fixup-third-party.sh. Example:

    ...
mv google/logging/ mockgcp/
...
find . -type f -print0 | xargs -0 sed -i -e "s@google/logging/@mockgcp/logging/@g"
find . -type f -print0 | xargs -0 sed -i -e "s@google\.logging@mockgcp.logging@g"
...

  3. Add a subdirectory called mock<servicename>.

    • Copying one of the existing ones. mockprivateca is a reasonable basic one. Note the files names.go and service.go, rename the third one to <resourcename>.go. We have started collapsing the contents of the names.go files into the relevant <resourcename>.go files.

  4. Adjust your mock to use your protos (change the import paths), rename the types, etc.

  5. Implement the core CRUD methods in <resourcename>.go.

    In the example file capool.go, we have GET, CREATE and DELETE implemented. More probably, you will need to implement some more CRUD methods. In the log, you should see some errors indicating which method is not supported. For those methods, go into the service definition and implement a basic implementation - we have examples of most of the CRUD operations at this point.

    Note that until you started using the mock package you defined, VSCode may highlight a warning that the package is not used. See later steps for where to use the new mock package.

  6. Register the mock service of your resource in the service.go file. Example.

  7. Add the service handler of your resource in the service.go file. Example.

  8. Add your service to mock_http_roundtrip.go, something like services = append(services, mockcloudbilling.New(env, storage)).

  9. Add the kind(s) you are adding support for to func MaybeSkip in harness.go.

  10. Run the tests.

    Example command:

    E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=mock go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures  2>&1 | tee log

    You can also specify a single test to run to save time, for example, TestAllInSeries/fixtures/privatecacapool, or reduce the timeout from 3600s to 600s or any other reasonable value depends on your test.

If you are lucky, everything will "just work" and you will see that your new tests are being invoked.

If something is not behaving as you would expect, you should be able to launch a debugger because it all runs in one process. You can also use ARTIFACTS=artifacts to get detailed HTTP logs of the traffic, which is useful if you want to see the json requests & responses. If you also use E2E_GCP_TARGET=real you can run against the real (non-mocked) GCP, and easily see what the actual behaviour should be. Usually however, this is not necessary; the most common failure mode is that terraform or Config Connector expects a field to be automatically populated, and it normally logs an error like "foo not set" (in this case, simply add that to your mock implementation.)

Capture golden object and HTTP golden logs

  1. Capture golden object and HTTP golden logs against real GCP.

    1. Run the following command to generate the golden object (_generated_object_\[testname\].golden.yaml file): E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=real GOLDEN_OBJECT_CHECKS=1 WRITE_GOLDEN_OUTPUT=1 go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures/[testname].

    2. Run the following command to generate the golden request (_http.log file): E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=real GOLDEN_REQUEST_CHECKS=1 WRITE_GOLDEN_OUTPUT=1 go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures/[testname].

    3. Ensure both GOLDEN_OBJECT_CHECKS and GOLDEN_REQUEST_CHECKS pass: E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=real GOLDEN_OBJECT_CHECKS=1 GOLDEN_REQUEST_CHECKS=1 go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures/[testname].

      1. If the test doesn't pass, check the output log and identify diffs. Normalize the values if needed in tests/e2e/normalize.go for GOLDEN_OBJECT_CHECKS and in tests/e2e/unified_test.go for GOLDEN_REQUEST_CHECKS.

  2. Ensure the test works when E2E_GCP_TARGET=mock: E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=mock GOLDEN_OBJECT_CHECKS=1 GOLDEN_REQUEST_CHECKS=1 go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures/[testname].

    1. If the test doesn't pass, update the CRUD methods in the mock to match the behavior needed by the golden files.

  3. In some edge cases, we can consider getting HTTP logs against mockGCP after implementing the mockGCP service. For certain resources that are difficult to run against real GCP (for example, AttachedCluster requires an external cluster and EdgeContainer requires a physical machine, and etc), it's acceptable to capture the golden HTTP logs from the mock service.

    Example command: E2E_KUBE_TARGET=envtest RUN_E2E=1 E2E_GCP_TARGET=mock GOLDEN_REQUEST_CHECKS=1 WRITE_GOLDEN_OUTPUT=1 go test -test.count=1 -timeout 3600s -v ./tests/e2e -run TestAllInSeries/fixtures/test-name.

Appendix

Generating proto

If the proto file on googleapis is out of date or non existent, you can generate a proto file from the google generated api as such:

$ 	wget -O ids-api-v1.json https://raw.githubusercontent.com/googleapis/google-api-go-client/b49e3b908a8ed562e068736f1c42e992538ba6e0/ids/v1/ids-api.json
	mkdir -p apis/mockgcp/cloud/ids/v1/
	cd tools/gapic; go run . --proto-version=2 ../../ids-api-v1.json > ../../apis/mockgcp/cloud/ids/v1/service.proto

Modify the snippet above for your own service and add it to the Makefile target generate-proto-from-openapi.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Interface 

type Interface interface {
	// We support HTTP requests
	http.RoundTripper

	// NewGRPCConnection returns a grpc connection to our mock implementation
	NewGRPCConnection(ctx context.Context) *grpc.ClientConn

	// Run starts the grpc service, until ctx is closed
	Run(ctx context.Context) error

	// We can dispatch test commands
	SupportsTestCommands
}

func NewMockRoundTripper 

func NewMockRoundTripper(ctx context.Context, k8sClient client.Client, storage storage.Storage) (Interface, error)

func NewMockRoundTripperForTest 

func NewMockRoundTripperForTest(t *testing.T, k8sClient client.Client, storage storage.Storage) Interface

type SupportsTestCommands 

type SupportsTestCommands interface {
	// RunTestCommand is a "backdoor" into our mock implementation that is useful for fault injection or faking scaling events etc
	// In our script-driven tests, we trigger this with a special `MockGCPBackdoor` object.
	RunTestCommand(ctx context.Context, service string, command string) error
}

Source Files

View all Source files

Directories

Path Synopsis
fields
httpmux
httptogrpc
operations
projects
regions
workflows
generated
google/analytics/admin/v1alpha
Package adminpb is a reverse proxy.
 Package adminpb is a reverse proxy.
google/apps/cloudidentity/v1beta1
Package cloudidentitypb is a reverse proxy.
 Package cloudidentitypb is a reverse proxy.
google/bigtable/admin/v2
Package adminpb is a reverse proxy.
 Package adminpb is a reverse proxy.
google/cloud/aiplatform/v1
Package aiplatformpb is a reverse proxy.
 Package aiplatformpb is a reverse proxy.
google/cloud/apigateway/v1
Package apigatewaypb is a reverse proxy.
 Package apigatewaypb is a reverse proxy.
google/cloud/batch/v1
Package batchpb is a reverse proxy.
 Package batchpb is a reverse proxy.
google/cloud/clouddms/v1
Package clouddmspb is a reverse proxy.
 Package clouddmspb is a reverse proxy.
google/cloud/dns/v1
Package dnspb is a reverse proxy.
 Package dnspb is a reverse proxy.
google/cloud/networksecurity/v1beta1
Package networksecuritypb is a reverse proxy.
 Package networksecuritypb is a reverse proxy.
google/cloud/networkservices/v1
Package networkservicespb is a reverse proxy.
 Package networkservicespb is a reverse proxy.
google/cloud/recaptchaenterprise/v1
Package recaptchaenterprisepb is a reverse proxy.
 Package recaptchaenterprisepb is a reverse proxy.
google/iam/v2
Package iampb is a reverse proxy.
 Package iampb is a reverse proxy.
google/storage/control/v2
Package controlpb is a reverse proxy.
 Package controlpb is a reverse proxy.
mockgcp/api/apikeys/v2
Package apikeyspb is a reverse proxy.
 Package apikeyspb is a reverse proxy.
mockgcp/api/cloudquotas/v1
Package cloudquotaspb is a reverse proxy.
 Package cloudquotaspb is a reverse proxy.
mockgcp/api/cloudquotas/v1beta
Package cloudquotaspb is a reverse proxy.
 Package cloudquotaspb is a reverse proxy.
mockgcp/api/serviceusage/v1
Package serviceusagepb is a reverse proxy.
 Package serviceusagepb is a reverse proxy.
mockgcp/api/serviceusage/v1beta1
Package serviceusage is a reverse proxy.
 Package serviceusage is a reverse proxy.
mockgcp/cloud/aiplatform/v1
Package aiplatformpb is a reverse proxy.
 Package aiplatformpb is a reverse proxy.
mockgcp/cloud/aiplatform/v1beta1
Package aiplatformpb is a reverse proxy.
 Package aiplatformpb is a reverse proxy.
mockgcp/cloud/apigateway/v1
Package apigatewaypb is a reverse proxy.
 Package apigatewaypb is a reverse proxy.
mockgcp/cloud/apigee/v1
Package apigeepb is a reverse proxy.
 Package apigeepb is a reverse proxy.
mockgcp/cloud/asset/v1
Package assetpb is a reverse proxy.
 Package assetpb is a reverse proxy.
mockgcp/cloud/backupdr/v1
Package backupdrpb is a reverse proxy.
 Package backupdrpb is a reverse proxy.
mockgcp/cloud/batch/v1
Package batchpb is a reverse proxy.
 Package batchpb is a reverse proxy.
mockgcp/cloud/bigquery/analyticshub/v1
Package analyticshubpb is a reverse proxy.
 Package analyticshubpb is a reverse proxy.
mockgcp/cloud/bigquery/biglake/v1
Package biglakepb is a reverse proxy.
 Package biglakepb is a reverse proxy.
mockgcp/cloud/bigquery/datatransfer/v1
Package datatransferpb is a reverse proxy.
 Package datatransferpb is a reverse proxy.
mockgcp/cloud/bigquery/reservation/v1
Package reservationpb is a reverse proxy.
 Package reservationpb is a reverse proxy.
mockgcp/cloud/bigquery/v2
Package bigquerypb is a reverse proxy.
 Package bigquerypb is a reverse proxy.
mockgcp/cloud/common
mockgcp/cloud/compute/v1
Package computepb is a reverse proxy.
 Package computepb is a reverse proxy.
mockgcp/cloud/datacatalog/v1
Package datacatalogpb is a reverse proxy.
 Package datacatalogpb is a reverse proxy.
mockgcp/cloud/dataform/v1beta1
Package dataformpb is a reverse proxy.
 Package dataformpb is a reverse proxy.
mockgcp/cloud/documentai/v1
Package documentaipb is a reverse proxy.
 Package documentaipb is a reverse proxy.
mockgcp/cloud/edgenetwork/v1
Package edgenetworkpb is a reverse proxy.
 Package edgenetworkpb is a reverse proxy.
mockgcp/cloud/filestore/v1
Package filestorepb is a reverse proxy.
 Package filestorepb is a reverse proxy.
mockgcp/cloud/functions/v1
Package functionspb is a reverse proxy.
 Package functionspb is a reverse proxy.
mockgcp/cloud/gkebackup/v1
Package gkebackuppb is a reverse proxy.
 Package gkebackuppb is a reverse proxy.
mockgcp/cloud/gkehub/policycontroller/v1beta
mockgcp/cloud/gkehub/servicemesh/v1beta
mockgcp/cloud/gkehub/v1beta
Package gkehubpb is a reverse proxy.
 Package gkehubpb is a reverse proxy.
mockgcp/cloud/gkehub/v1beta/configmanagement
mockgcp/cloud/gkehub/v1beta/metering
mockgcp/cloud/gkehub/v1beta/multiclusteringress
mockgcp/cloud/gkehub/v1beta1
Package gkehubpb is a reverse proxy.
 Package gkehubpb is a reverse proxy.
mockgcp/cloud/gkemulticloud/v1
Package gkemulticloudpb is a reverse proxy.
 Package gkemulticloudpb is a reverse proxy.
mockgcp/cloud/ids/v1
Package idspb is a reverse proxy.
 Package idspb is a reverse proxy.
mockgcp/cloud/metastore/v1
Package metastorepb is a reverse proxy.
 Package metastorepb is a reverse proxy.
mockgcp/cloud/networkconnectivity/v1
Package networkconnectivitypb is a reverse proxy.
 Package networkconnectivitypb is a reverse proxy.
mockgcp/cloud/orchestration/airflow/service/v1
Package servicepb is a reverse proxy.
 Package servicepb is a reverse proxy.
mockgcp/cloud/parametermanager/v1
Package parametermanagerpb is a reverse proxy.
 Package parametermanagerpb is a reverse proxy.
mockgcp/cloud/privilegedaccessmanager/v1
Package privilegedaccessmanagerpb is a reverse proxy.
 Package privilegedaccessmanagerpb is a reverse proxy.
mockgcp/cloud/recaptchaenterprise/v1
Package recaptchaenterprisepb is a reverse proxy.
 Package recaptchaenterprisepb is a reverse proxy.
mockgcp/cloud/resourcemanager/v1
Package cloudresourcemanagerpb is a reverse proxy.
 Package cloudresourcemanagerpb is a reverse proxy.
mockgcp/cloud/resourcemanager/v3
Package resourcemanagerpb is a reverse proxy.
 Package resourcemanagerpb is a reverse proxy.
mockgcp/cloud/run/v2
Package runpb is a reverse proxy.
 Package runpb is a reverse proxy.
mockgcp/cloud/security/privateca/v1
Package privatecapb is a reverse proxy.
 Package privatecapb is a reverse proxy.
mockgcp/cloud/servicedirectory/v1beta1
Package servicedirectorypb is a reverse proxy.
 Package servicedirectorypb is a reverse proxy.
mockgcp/cloud/servicenetworking/v1
Package servicenetworkingpb is a reverse proxy.
 Package servicenetworkingpb is a reverse proxy.
mockgcp/cloud/sql/v1beta4
Package sqlpb is a reverse proxy.
 Package sqlpb is a reverse proxy.
mockgcp/cloud/tasks/v2
Package cloudtaskspb is a reverse proxy.
 Package cloudtaskspb is a reverse proxy.
mockgcp/cloud/tpu/v2
Package tpupb is a reverse proxy.
 Package tpupb is a reverse proxy.
mockgcp/cloud/vmwareengine/v1
Package vmwareenginepb is a reverse proxy.
 Package vmwareenginepb is a reverse proxy.
mockgcp/cloud/vpcaccess/v1
Package vpcaccesspb is a reverse proxy.
 Package vpcaccesspb is a reverse proxy.
mockgcp/container/v1beta1
Package containerpb is a reverse proxy.
 Package containerpb is a reverse proxy.
mockgcp/dataflow/v1beta3
Package dataflowpb is a reverse proxy.
 Package dataflowpb is a reverse proxy.
mockgcp/devtools/artifactregistry/v1
Package artifactregistrypb is a reverse proxy.
 Package artifactregistrypb is a reverse proxy.
mockgcp/devtools/cloudbuild/v1
Package cloudbuildpb is a reverse proxy.
 Package cloudbuildpb is a reverse proxy.
mockgcp/devtools/sourcerepo/v1
Package sourcerepo is a reverse proxy.
 Package sourcerepo is a reverse proxy.
mockgcp/iam/admin/v1
Package adminpb is a reverse proxy.
 Package adminpb is a reverse proxy.
mockgcp/iam/v1
mockgcp/pubsub/v1
Package pubsubpb is a reverse proxy.
 Package pubsubpb is a reverse proxy.
mockgcp/spanner/admin/database/v1
Package databasepb is a reverse proxy.
 Package databasepb is a reverse proxy.
mockgcp/spanner/admin/instance/v1
Package instancepb is a reverse proxy.
 Package instancepb is a reverse proxy.
mockgcp/storage/v1
Package storagepb is a reverse proxy.
 Package storagepb is a reverse proxy.
mockgrafeas/v1
Package grafeas is a reverse proxy.
 Package grafeas is a reverse proxy.
mockspanner
admin
+tool:mockgcp-support-spanner proto.service: google.spanner.admin.database.v1.DatabaseAdmin proto.message: google.spanner.admin.database.v1.Database
 +tool:mockgcp-support-spanner proto.service: google.spanner.admin.database.v1.DatabaseAdmin proto.message: google.spanner.admin.database.v1.Database
pkg
storage
validation
tools
gapic command
gapic/pkg/openapi
gapic/pkg/protogen
gapic/pkg/sorted
patch-proto module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.