Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #108533

Open
cockroach-teamcity opened this issue Aug 10, 2023 · 1,548 comments
Open

roachtest: cluster_creation failed #108533

cockroach-teamcity opened this issue Aug 10, 2023 · 1,548 comments
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 10, 2023 •
edited by jlinder
Loading

roachtest.cluster_creation failed with artifacts on release-23.1 @ b1194729b7be664edf159ed76e574b0855879866:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-08-10t17_16_00z,roachprod=true,cluster=teamcity-11261553-1691646222-148-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script2578581384 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11261553-1691646222-148-n9cpu8-geo-0004 teamcity-11261553-1691646222-148-n9cpu8-geo-0005 teamcity-11261553-1691646222-148-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-30510
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 10, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Aug 10, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 10, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 94860a03416ef03c81b8cb31c827469354a35af8:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-jVORjC7Vks7PeQLkM6yIPeDMDBY= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-08-11T08:01:32Z},{Key=Name,Value=teamcity-11277566-1691732642-29-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11277566-1691732642-29-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-08-11T08:01:32Z},{Key=Name,Value=teamcity-11277566-1691732642-29-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11277566-1691732642-29-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script4078814971 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3061003271 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@smg260
Copy link
Contributor

smg260 commented Aug 11, 2023

This latest one seems to be introduced by #108427
cc: @pavelkalinnikov

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 94860a03416ef03c81b8cb31c827469354a35af8:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-11277565-1691732628-145-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-11t17_42_37z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script166572881 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11277565-1691732628-145-n9cpu8-geo-0001 teamcity-11277565-1691732628-145-n9cpu8-geo-0002 teamcity-11277565-1691732628-145-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 11, 2023
Closed
@knz knz added the T-testeng TestEng Team label Aug 11, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 11, 2023

cc @cockroachdb/test-eng

This was referenced Aug 12, 2023
Closed
Closed
@srosenberg srosenberg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 12, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 84e0190d613b96728ff4622c7573f0743a9d5bb3:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-vNQPieQMjvFJr5iQweG8YBjlPYI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-12T07:58:35Z},{Key=Name,Value=teamcity-11290593-1691818940-28-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11290593-1691818940-28-n10cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-12T07:58:35Z},{Key=Name,Value=teamcity-11290593-1691818940-28-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11290593-1691818940-28-n10cpu8}] --user-data file:///tmp/aws-startup-script652637822 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping197316976 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 84e0190d613b96728ff4622c7573f0743a9d5bb3:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-08-12t17_19_32z,roachprod=true,cluster=teamcity-11290592-1691819083-143-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3125883753 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11290592-1691819083-143-n9cpu8-geo-0001 teamcity-11290592-1691819083-143-n9cpu8-geo-0002 teamcity-11290592-1691819083-143-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 76ea9a2b98758fd056d6d5366b9734dce9a337fd:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-62qfZKCVE9u_M5MbUog1_BP5TT4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11300115-1691905309-34-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-13T08:02:26Z},{Key=Name,Value=teamcity-11300115-1691905309-34-n10cpu8-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11300115-1691905309-34-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-13T08:02:26Z},{Key=Name,Value=teamcity-11300115-1691905309-34-n10cpu8-0001}] --user-data file:///tmp/aws-startup-script4019317834 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3935907734 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 76ea9a2b98758fd056d6d5366b9734dce9a337fd:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11300114-1691905330-137-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-13t17_10_49z --metadata-from-file startup-script=/tmp/gce-startup-script3740837604 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11300114-1691905330-137-n9cpu8-geo-0004 teamcity-11300114-1691905330-137-n9cpu8-geo-0005 teamcity-11300114-1691905330-137-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 76ea9a2b98758fd056d6d5366b9734dce9a337fd:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-XhhfxMnDqLKZmZPgYYHPxHerfnU= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-14T08:03:39Z},{Key=Name,Value=teamcity-11308712-1691991998-29-n10cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11308712-1691991998-29-n10cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-14T08:03:39Z},{Key=Name,Value=teamcity-11308712-1691991998-29-n10cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11308712-1691991998-29-n10cpu8}] --user-data file:///tmp/aws-startup-script4088333315 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1059779786 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 76ea9a2b98758fd056d6d5366b9734dce9a337fd:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11308711-1691991932-150-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-14t17_31_35z --metadata-from-file startup-script=/tmp/gce-startup-script915985337 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11308711-1691991932-150-n9cpu8-geo-0004 teamcity-11308711-1691991932-150-n9cpu8-geo-0005 teamcity-11308711-1691991932-150-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6d20f93562a6a78c8a4c4dc84ceb4167376b8faf:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-KSBu_RUhe3pC1cD8ugpcRHonwsc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11324178-1692078125-28-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-15T07:58:22Z},{Key=Name,Value=teamcity-11324178-1692078125-28-n10cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11324178-1692078125-28-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-15T07:58:22Z},{Key=Name,Value=teamcity-11324178-1692078125-28-n10cpu8-0002}] --user-data file:///tmp/aws-startup-script1430701466 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1022840563 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6d20f93562a6a78c8a4c4dc84ceb4167376b8faf:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11324177-1692078142-143-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-15t17_05_10z --metadata-from-file startup-script=/tmp/gce-startup-script875288865 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11324177-1692078142-143-n9cpu8-geo-0004 teamcity-11324177-1692078142-143-n9cpu8-geo-0005 teamcity-11324177-1692078142-143-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@srosenberg srosenberg added X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue and removed C-test-failure Broken test (automatically or manually discovered). labels Aug 16, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 10cbcf7497b4e56d43b6c461a10787a4a4fb9ca4:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-08-16t17_13_06z,roachprod=true,cluster=teamcity-11339567-1692164629-141-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3229083532 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11339567-1692164629-141-n9cpu8-geo-0001 teamcity-11339567-1692164629-141-n9cpu8-geo-0002 teamcity-11339567-1692164629-141-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ e0e71af3fc0d357968e5dc46ca95d881bc8937e4:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-08-17t16_57_08z,roachprod=true,cluster=teamcity-11354773-1692250972-138-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2126132351 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11354773-1692250972-138-n9cpu8-geo-0004 teamcity-11354773-1692250972-138-n9cpu8-geo-0005 teamcity-11354773-1692250972-138-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test jepsen/sequential/majority-ring-start-kill-2 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18024859-1733294575-220-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_03_57z --metadata-from-file startup-script=/tmp/gce-startup-script1424273728 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024859-1733294575-220-n6cpu4-0001 teamcity-18024859-1733294575-220-n6cpu4-0002 teamcity-18024859-1733294575-220-n6cpu4-0003 teamcity-18024859-1733294575-220-n6cpu4-0004 teamcity-18024859-1733294575-220-n6cpu4-0005 teamcity-18024859-1733294575-220-n6cpu4-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test jepsen/sequential/start-stop-2 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,created=2024-12-04t09_05_23z,roachprod=true,cluster=teamcity-18024859-1733294575-221-n6cpu4,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script1022695132 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024859-1733294575-221-n6cpu4-0001 teamcity-18024859-1733294575-221-n6cpu4-0002 teamcity-18024859-1733294575-221-n6cpu4-0003 teamcity-18024859-1733294575-221-n6cpu4-0004 teamcity-18024859-1733294575-221-n6cpu4-0005 teamcity-18024859-1733294575-221-n6cpu4-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test jepsen/sets/strobe-skews failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-12-04t09_07_45z,roachprod=true,cluster=teamcity-18024859-1733294575-229-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2983756467 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024859-1733294575-229-n6cpu4-0001 teamcity-18024859-1733294575-229-n6cpu4-0002 teamcity-18024859-1733294575-229-n6cpu4-0003 teamcity-18024859-1733294575-229-n6cpu4-0004 teamcity-18024859-1733294575-229-n6cpu4-0005 teamcity-18024859-1733294575-229-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18024859-1733294575-229-n6cpu4-0001].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test jepsen/sets/subcritical-skews failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-12-04t09_07_49z,roachprod=true,cluster=teamcity-18024859-1733294575-230-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script98373890 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024859-1733294575-230-n6cpu4-0001 teamcity-18024859-1733294575-230-n6cpu4-0002 teamcity-18024859-1733294575-230-n6cpu4-0003 teamcity-18024859-1733294575-230-n6cpu4-0004 teamcity-18024859-1733294575-230-n6cpu4-0005 teamcity-18024859-1733294575-230-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-230-n6cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-230-n6cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,created=2024-12-04t09_11_21z,roachprod=true,cluster=teamcity-18024859-1733294575-242-n4cpu96,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script2338262294 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024859-1733294575-242-n4cpu96-0001 teamcity-18024859-1733294575-242-n4cpu96-0002 teamcity-18024859-1733294575-242-n4cpu96-0003 teamcity-18024859-1733294575-242-n4cpu96-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test kv95/enc=false/nodes=3 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-18024859-1733294575-244-n4cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_11_30z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2724073019 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024859-1733294575-244-n4cpu8-0001 teamcity-18024859-1733294575-244-n4cpu8-0002 teamcity-18024859-1733294575-244-n4cpu8-0003 teamcity-18024859-1733294575-244-n4cpu8-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test kv95/enc=false/nodes=3/tracing failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18024859-1733294575-246-n4cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_12_36z --metadata-from-file startup-script=/tmp/gce-startup-script2616281590 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024859-1733294575-246-n4cpu8-0001 teamcity-18024859-1733294575-246-n4cpu8-0002 teamcity-18024859-1733294575-246-n4cpu8-0003 teamcity-18024859-1733294575-246-n4cpu8-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test kv95/enc=false/nodes=3/batch=16 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18024859-1733294575-247-n4cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_12_46z --metadata-from-file startup-script=/tmp/gce-startup-script2831731854 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024859-1733294575-247-n4cpu8-0001 teamcity-18024859-1733294575-247-n4cpu8-0002 teamcity-18024859-1733294575-247-n4cpu8-0003 teamcity-18024859-1733294575-247-n4cpu8-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18024859-1733294575-247-n4cpu8-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test loqrecovery/workload=tpcc/rangeSize=16mb failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-18024859-1733294575-262-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_16_32z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1283362391 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024859-1733294575-262-n6cpu4-0001 teamcity-18024859-1733294575-262-n6cpu4-0002 teamcity-18024859-1733294575-262-n6cpu4-0003 teamcity-18024859-1733294575-262-n6cpu4-0004 teamcity-18024859-1733294575-262-n6cpu4-0005 teamcity-18024859-1733294575-262-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-262-n6cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-262-n6cpu4-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test rebalance/by-load/replicas/mixed-version failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18024859-1733294575-263-n7cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_17_08z --metadata-from-file startup-script=/tmp/gce-startup-script3700923678 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024859-1733294575-263-n7cpu4-0001 teamcity-18024859-1733294575-263-n7cpu4-0002 teamcity-18024859-1733294575-263-n7cpu4-0003 teamcity-18024859-1733294575-263-n7cpu4-0004 teamcity-18024859-1733294575-263-n7cpu4-0005 teamcity-18024859-1733294575-263-n7cpu4-0006 teamcity-18024859-1733294575-263-n7cpu4-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test replicagc-changed-peers/restart=true failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18024859-1733294575-264-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_17_10z --metadata-from-file startup-script=/tmp/gce-startup-script2830000361 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024859-1733294575-264-n6cpu4-0001 teamcity-18024859-1733294575-264-n6cpu4-0002 teamcity-18024859-1733294575-264-n6cpu4-0003 teamcity-18024859-1733294575-264-n6cpu4-0004 teamcity-18024859-1733294575-264-n6cpu4-0005 teamcity-18024859-1733294575-264-n6cpu4-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5990edc8fdb60e84bada9fb50d631f95b7283ea5:

test replicate/rebalance/3to5 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-18024859-1733294575-270-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t09_19_02z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2552057981 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024859-1733294575-270-n6cpu4-0001 teamcity-18024859-1733294575-270-n6cpu4-0002 teamcity-18024859-1733294575-270-n6cpu4-0003 teamcity-18024859-1733294575-270-n6cpu4-0004 teamcity-18024859-1733294575-270-n6cpu4-0005 teamcity-18024859-1733294575-270-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-270-n6cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-270-n6cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-270-n6cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-270-n6cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024859-1733294575-270-n6cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test decommissionBench/nodes=4/warehouses=1000/while-down failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-18075258-1733553763-39-n5cpu16,lifetime=12h0m0s,arch=arm64,created=2024-12-07t08_47_02z --metadata-from-file startup-script=/tmp/gce-startup-script1397958269 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18075258-1733553763-39-n5cpu16-0001 teamcity-18075258-1733553763-39-n5cpu16-0002 teamcity-18075258-1733553763-39-n5cpu16-0003 teamcity-18075258-1733553763-39-n5cpu16-0004 teamcity-18075258-1733553763-39-n5cpu16-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test failover/chaos/read-write/lease=expiration failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-2 --labels usage=roachtest,roachprod=true,cluster=teamcity-18075258-1733553763-46-n10cpu2,lifetime=12h0m0s,arch=arm64,created=2024-12-07t09_00_03z --metadata-from-file startup-script=/tmp/gce-startup-script3007456016 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18075258-1733553763-46-n10cpu2-0001 teamcity-18075258-1733553763-46-n10cpu2-0002 teamcity-18075258-1733553763-46-n10cpu2-0003 teamcity-18075258-1733553763-46-n10cpu2-0004 teamcity-18075258-1733553763-46-n10cpu2-0005 teamcity-18075258-1733553763-46-n10cpu2-0006 teamcity-18075258-1733553763-46-n10cpu2-0007 teamcity-18075258-1733553763-46-n10cpu2-0008 teamcity-18075258-1733553763-46-n10cpu2-0009 teamcity-18075258-1733553763-46-n10cpu2-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-46-n10cpu2-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-46-n10cpu2-0010].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test failover/non-system/blackhole-send/lease=expiration failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-2 --labels usage=roachtest,created=2024-12-07t09_10_33z,roachprod=true,cluster=teamcity-18075258-1733553763-50-n7cpu2,lifetime=12h0m0s,arch=arm64 --metadata-from-file startup-script=/tmp/gce-startup-script988813445 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18075258-1733553763-50-n7cpu2-0001 teamcity-18075258-1733553763-50-n7cpu2-0002 teamcity-18075258-1733553763-50-n7cpu2-0003 teamcity-18075258-1733553763-50-n7cpu2-0004 teamcity-18075258-1733553763-50-n7cpu2-0005 teamcity-18075258-1733553763-50-n7cpu2-0006 teamcity-18075258-1733553763-50-n7cpu2-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-50-n7cpu2-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-50-n7cpu2-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-50-n7cpu2-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18075258-1733553763-50-n7cpu2-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test indexes/2/nodes=6/multi-region failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18085029-1733639967-77-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2024-12-08t10_26_39z --metadata-from-file startup-script=/tmp/gce-startup-script748105721 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-18085029-1733639967-77-n7cpu16-geo-0003 teamcity-18085029-1733639967-77-n7cpu16-geo-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-west1-b/instances/teamcity-18085029-1733639967-77-n7cpu16-geo-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 30000.0 in region us-west1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 30000.0
	dimensions = region: us-west1
vm_family: N2
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test tpcc/multiregion/survive=region/chaos=true failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18085029-1733639967-98-n10cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-12-08t11_13_31z --metadata-from-file startup-script=/tmp/gce-startup-script1889883856 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-18085029-1733639967-98-n10cpu4-geo-0004 teamcity-18085029-1733639967-98-n10cpu4-geo-0005 teamcity-18085029-1733639967-98-n10cpu4-geo-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-west1-b/instances/teamcity-18085029-1733639967-98-n10cpu4-geo-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-west1-b/instances/teamcity-18085029-1733639967-98-n10cpu4-geo-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 30000.0 in region us-west1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 30000.0
	dimensions = region: us-west1
vm_family: N2
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 38ca5384cab47ed2bacd71f49ad00471d82ab7ac:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,roachprod=true,cluster=teamcity-18085029-1733639967-105-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-12-08t12_00_22z --metadata-from-file startup-script=/tmp/gce-startup-script37709191 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18085029-1733639967-105-n4cpu96-0001 teamcity-18085029-1733639967-105-n4cpu96-0002 teamcity-18085029-1733639967-105-n4cpu96-0003 teamcity-18085029-1733639967-105-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18085029-1733639967-105-n4cpu96-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 80000.0
	dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
	metric name = compute.googleapis.com/n2d_cpus
	limit name = N2D-CPUS-per-project-region
	limit = 1000.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test decommission/slow failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18179428-1734158389-47-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_27_34z --metadata-from-file startup-script=/tmp/gce-startup-script3748256227 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18179428-1734158389-47-n6cpu4-0001 teamcity-18179428-1734158389-47-n6cpu4-0002 teamcity-18179428-1734158389-47-n6cpu4-0003 teamcity-18179428-1734158389-47-n6cpu4-0004 teamcity-18179428-1734158389-47-n6cpu4-0005 teamcity-18179428-1734158389-47-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-47-n6cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-47-n6cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-47-n6cpu4-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-47-n6cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test drain-and-decommission/nodes=9 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18179428-1734158389-56-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_39_01z --metadata-from-file startup-script=/tmp/gce-startup-script3967962493 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18179428-1734158389-56-n9cpu4-0001 teamcity-18179428-1734158389-56-n9cpu4-0002 teamcity-18179428-1734158389-56-n9cpu4-0003 teamcity-18179428-1734158389-56-n9cpu4-0004 teamcity-18179428-1734158389-56-n9cpu4-0005 teamcity-18179428-1734158389-56-n9cpu4-0006 teamcity-18179428-1734158389-56-n9cpu4-0007 teamcity-18179428-1734158389-56-n9cpu4-0008 teamcity-18179428-1734158389-56-n9cpu4-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-56-n9cpu4-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test failover/chaos/read-only failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_39_54z,roachprod=true,cluster=teamcity-18179428-1734158389-58-n10cpu2 --metadata-from-file startup-script=/tmp/gce-startup-script3858670658 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18179428-1734158389-58-n10cpu2-0001 teamcity-18179428-1734158389-58-n10cpu2-0002 teamcity-18179428-1734158389-58-n10cpu2-0003 teamcity-18179428-1734158389-58-n10cpu2-0004 teamcity-18179428-1734158389-58-n10cpu2-0005 teamcity-18179428-1734158389-58-n10cpu2-0006 teamcity-18179428-1734158389-58-n10cpu2-0007 teamcity-18179428-1734158389-58-n10cpu2-0008 teamcity-18179428-1734158389-58-n10cpu2-0009 teamcity-18179428-1734158389-58-n10cpu2-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18179428-1734158389-58-n10cpu2-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18179428-1734158389-58-n10cpu2-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test failover/non-system/blackhole-send/lease=expiration failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18179428-1734158389-71-n7cpu2,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_47_28z --metadata-from-file startup-script=/tmp/gce-startup-script2121842813 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18179428-1734158389-71-n7cpu2-0001 teamcity-18179428-1734158389-71-n7cpu2-0002 teamcity-18179428-1734158389-71-n7cpu2-0003 teamcity-18179428-1734158389-71-n7cpu2-0004 teamcity-18179428-1734158389-71-n7cpu2-0005 teamcity-18179428-1734158389-71-n7cpu2-0006 teamcity-18179428-1734158389-71-n7cpu2-0007]
Output: WARNING: Some requests generated warnings:
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test failover/non-system/crash failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_47_28z,roachprod=true,cluster=teamcity-18179428-1734158389-72-n7cpu2 --metadata-from-file startup-script=/tmp/gce-startup-script1026330035 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18179428-1734158389-72-n7cpu2-0001 teamcity-18179428-1734158389-72-n7cpu2-0002 teamcity-18179428-1734158389-72-n7cpu2-0003 teamcity-18179428-1734158389-72-n7cpu2-0004 teamcity-18179428-1734158389-72-n7cpu2-0005 teamcity-18179428-1734158389-72-n7cpu2-0006 teamcity-18179428-1734158389-72-n7cpu2-0007]
Output: WARNING: Some requests generated warnings:
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test clearrange/checks=true/rangeTs=true failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-18179428-1734158389-73-n10cpu16,lifetime=12h0m0s,arch=amd64,created=2024-12-14t08_47_33z --metadata-from-file startup-script=/tmp/gce-startup-script2276936276 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18179428-1734158389-73-n10cpu16-0001 teamcity-18179428-1734158389-73-n10cpu16-0002 teamcity-18179428-1734158389-73-n10cpu16-0003 teamcity-18179428-1734158389-73-n10cpu16-0004 teamcity-18179428-1734158389-73-n10cpu16-0005 teamcity-18179428-1734158389-73-n10cpu16-0006 teamcity-18179428-1734158389-73-n10cpu16-0007 teamcity-18179428-1734158389-73-n10cpu16-0008 teamcity-18179428-1734158389-73-n10cpu16-0009 teamcity-18179428-1734158389-73-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-73-n10cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-73-n10cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-73-n10cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-73-n10cpu16-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18179428-1734158389-73-n10cpu16-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test import/tpcc/warehouses=1000/nodes=32 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,created=2024-12-14t09_27_28z,roachprod=true,cluster=teamcity-18179428-1734158389-94-n32cpu4,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3706613523 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18179428-1734158389-94-n32cpu4-0001 teamcity-18179428-1734158389-94-n32cpu4-0002 teamcity-18179428-1734158389-94-n32cpu4-0003 teamcity-18179428-1734158389-94-n32cpu4-0004 teamcity-18179428-1734158389-94-n32cpu4-0005 teamcity-18179428-1734158389-94-n32cpu4-0006 teamcity-18179428-1734158389-94-n32cpu4-0007 teamcity-18179428-1734158389-94-n32cpu4-0008 teamcity-18179428-1734158389-94-n32cpu4-0009 teamcity-18179428-1734158389-94-n32cpu4-0010 teamcity-18179428-1734158389-94-n32cpu4-0011 teamcity-18179428-1734158389-94-n32cpu4-0012 teamcity-18179428-1734158389-94-n32cpu4-0013 teamcity-18179428-1734158389-94-n32cpu4-0014 teamcity-18179428-1734158389-94-n32cpu4-0015 teamcity-18179428-1734158389-94-n32cpu4-0016 teamcity-18179428-1734158389-94-n32cpu4-0017 teamcity-18179428-1734158389-94-n32cpu4-0018 teamcity-18179428-1734158389-94-n32cpu4-0019 teamcity-18179428-1734158389-94-n32cpu4-0020 teamcity-18179428-1734158389-94-n32cpu4-0021 teamcity-18179428-1734158389-94-n32cpu4-0022 teamcity-18179428-1734158389-94-n32cpu4-0023 teamcity-18179428-1734158389-94-n32cpu4-0024 teamcity-18179428-1734158389-94-n32cpu4-0025 teamcity-18179428-1734158389-94-n32cpu4-0026 teamcity-18179428-1734158389-94-n32cpu4-0027 teamcity-18179428-1734158389-94-n32cpu4-0028 teamcity-18179428-1734158389-94-n32cpu4-0029 teamcity-18179428-1734158389-94-n32cpu4-0030 teamcity-18179428-1734158389-94-n32cpu4-0031 teamcity-18179428-1734158389-94-n32cpu4-0032]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0020].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0022].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0025].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18179428-1734158389-94-n32cpu4-0032].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-16t12_11_17z,roachprod=true,cluster=teamcity-18196744-1734331245-88-n5cpu96 --metadata-from-file startup-script=/tmp/gce-startup-script3040350070 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18196744-1734331245-88-n5cpu96-0001 teamcity-18196744-1734331245-88-n5cpu96-0002 teamcity-18196744-1734331245-88-n5cpu96-0003 teamcity-18196744-1734331245-88-n5cpu96-0004 teamcity-18196744-1734331245-88-n5cpu96-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18196744-1734331245-88-n5cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18196744-1734331245-88-n5cpu96-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18196744-1734331245-88-n5cpu96-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 80000.0
	dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6ba4ca05e288ef942bd6e49175f65ee64c4209e8:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,cluster=teamcity-18196744-1734331245-91-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-12-16t12_16_55z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script999201191 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18196744-1734331245-91-n4cpu96-0001 teamcity-18196744-1734331245-91-n4cpu96-0002 teamcity-18196744-1734331245-91-n4cpu96-0003 teamcity-18196744-1734331245-91-n4cpu96-0004]
Output: WARNING: Some requests generated warnings:
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 80000.0
	dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
	metric name = compute.googleapis.com/n2d_cpus
	limit name = N2D-CPUS-per-project-region
	limit = 1000.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 4be71b4caccaa453d45a89fc3b9c3b45ac04ca43:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,cluster=teamcity-18211468-1734417656-96-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-12-17t11_21_46z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2951159494 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18211468-1734417656-96-n4cpu96-0001 teamcity-18211468-1734417656-96-n4cpu96-0002 teamcity-18211468-1734417656-96-n4cpu96-0003 teamcity-18211468-1734417656-96-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-96-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-96-n4cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-96-n4cpu96-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 80000.0
	dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 4be71b4caccaa453d45a89fc3b9c3b45ac04ca43:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-17t11_32_22z,roachprod=true,cluster=teamcity-18211468-1734417656-99-n5cpu96 --metadata-from-file startup-script=/tmp/gce-startup-script907861050 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18211468-1734417656-99-n5cpu96-0001 teamcity-18211468-1734417656-99-n5cpu96-0002 teamcity-18211468-1734417656-99-n5cpu96-0003 teamcity-18211468-1734417656-99-n5cpu96-0004 teamcity-18211468-1734417656-99-n5cpu96-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-99-n5cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-99-n5cpu96-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18211468-1734417656-99-n5cpu96-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
	metric name = compute.googleapis.com/n2d_cpus
	limit name = N2D-CPUS-per-project-region
	limit = 1000.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5c7fcd4cc40074b6c9781f987253d0c1a247f1e7:

test import/tpcc/warehouses=1000/nodes=32 failed: (test_runner.go:741).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --labels usage=roachtest,arch=arm64,created=2024-12-24t09_29_31z,roachprod=true,cluster=teamcity-18285279-1735022382-52-n32cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3372743318 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18285279-1735022382-52-n32cpu4-0001 teamcity-18285279-1735022382-52-n32cpu4-0002 teamcity-18285279-1735022382-52-n32cpu4-0003 teamcity-18285279-1735022382-52-n32cpu4-0004 teamcity-18285279-1735022382-52-n32cpu4-0005 teamcity-18285279-1735022382-52-n32cpu4-0006 teamcity-18285279-1735022382-52-n32cpu4-0007 teamcity-18285279-1735022382-52-n32cpu4-0008 teamcity-18285279-1735022382-52-n32cpu4-0009 teamcity-18285279-1735022382-52-n32cpu4-0010 teamcity-18285279-1735022382-52-n32cpu4-0011 teamcity-18285279-1735022382-52-n32cpu4-0012 teamcity-18285279-1735022382-52-n32cpu4-0013 teamcity-18285279-1735022382-52-n32cpu4-0014 teamcity-18285279-1735022382-52-n32cpu4-0015 teamcity-18285279-1735022382-52-n32cpu4-0016 teamcity-18285279-1735022382-52-n32cpu4-0017 teamcity-18285279-1735022382-52-n32cpu4-0018 teamcity-18285279-1735022382-52-n32cpu4-0019 teamcity-18285279-1735022382-52-n32cpu4-0020 teamcity-18285279-1735022382-52-n32cpu4-0021 teamcity-18285279-1735022382-52-n32cpu4-0022 teamcity-18285279-1735022382-52-n32cpu4-0023 teamcity-18285279-1735022382-52-n32cpu4-0024 teamcity-18285279-1735022382-52-n32cpu4-0025 teamcity-18285279-1735022382-52-n32cpu4-0026 teamcity-18285279-1735022382-52-n32cpu4-0027 teamcity-18285279-1735022382-52-n32cpu4-0028 teamcity-18285279-1735022382-52-n32cpu4-0029 teamcity-18285279-1735022382-52-n32cpu4-0030 teamcity-18285279-1735022382-52-n32cpu4-0031 teamcity-18285279-1735022382-52-n32cpu4-0032]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0009].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0010].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0011].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0012].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0013].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0014].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0017].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0018].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0020].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0021].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0024].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0026].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0027].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0029].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0030].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18285279-1735022382-52-n32cpu4-0031].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20241218'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
4 participants
@knz @srosenberg @cockroach-teamcity @smg260