-
Notifications
You must be signed in to change notification settings - Fork 554
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cephfs: Add CSI-Addons NetworkFence support to CephFS #4113
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An early review. Please add unit-tests for functions like fetchIP()
so that the parsing of the client IP-address is validated.
@@ -38,6 +39,27 @@ type NetworkFence struct { | |||
cr *util.Credentials | |||
} | |||
|
|||
// ActiveClient represents the structure of an active client. | |||
type ActiveClient struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this struct is not used outside of this package, you do not need to export it, so call it activeClient
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And also remove unwanted members from the struct
return ipCidr.Contains(ipAddress) | ||
} | ||
|
||
func fetchIP(clientInfo string) (string, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make this
func (ac *activeClient) fetchIP() (string, error) {
clientInfo := ac.Inst
...
and then call
clientIP, err := client.fetchIP()
further below.
@@ -132,6 +154,90 @@ func (nf *NetworkFence) AddNetworkFence(ctx context.Context) error { | |||
return nil | |||
} | |||
|
|||
func listActiveClients(ctx context.Context, mdsDaemonID string) ([]ActiveClient, error) { | |||
cmd := []string{"tell", fmt.Sprintf("mds.%s", mdsDaemonID), "client", "ls"} | |||
stdout, stdErr, err := util.ExecCommand(ctx, "ceph", cmd...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make sure to leave a FIXME:
comment to use go-ceph in the future
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this is much required
"google.golang.org/grpc/status" | ||
) | ||
|
||
// FenceControllerServer struct of cephfs CSI driver with supported methods |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cephfs
to cephFS
@@ -132,6 +154,90 @@ func (nf *NetworkFence) AddNetworkFence(ctx context.Context) error { | |||
return nil | |||
} | |||
|
|||
func listActiveClients(ctx context.Context, mdsDaemonID string) ([]ActiveClient, error) { | |||
cmd := []string{"tell", fmt.Sprintf("mds.%s", mdsDaemonID), "client", "ls"} | |||
stdout, stdErr, err := util.ExecCommand(ctx, "ceph", cmd...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this is much required
// Evict the client if it matches the CIDR block | ||
err := evictCephFSClient(ctx, clientIP, "0") | ||
if err != nil { | ||
return fmt.Errorf("Error evicting client %s: %s\n", clientIP, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
start error with small letters ad use %w
for wrapping error
// Check if the clientIP is in the CIDR block | ||
if isIPInCIDR(ctx, clientIP, cidr) { | ||
// Evict the client if it matches the CIDR block | ||
err := evictCephFSClient(ctx, clientIP, "0") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what will happen for already evicted clients? are we going to see any error or it will be skipped?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be no-op, that is skip. I can try to test it out.
// Parse the CIDR block | ||
_, ipCidr, err := net.ParseCIDR(cidr) | ||
if err != nil { | ||
log.DebugLog(ctx, "Error parsing CIDR block %s: %v\n", cidr, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use error logging.
cmd := []string{"tell", fmt.Sprintf("mds.%s", mdsDaemonID), "client", "evict", clientIP} | ||
_, stdErr, err := util.ExecCommand(ctx, "ceph", cmd...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use with timeout as the command might get stuck?
for _, client := range activeClients { | ||
clientIP, err := fetchIP(client.Inst) | ||
if err != nil { | ||
return fmt.Errorf("Error fetching client IP: %s\n", err) | ||
} | ||
// Check if the clientIP is in the CIDR block | ||
if isIPInCIDR(ctx, clientIP, cidr) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if we dont find any activeClients but we still want to blocklist all the IP's in the CIDR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Madhu-1 , updated.
For the IPs in CIDR which doesn't have any corresponding active clients, blocklisting like rbd does -> i.e. blocklisting the entire IP by adding to ceph blocklist.
PTAL.
/* | ||
Copyright 2023 The Ceph-CSI Authors. | ||
Licensed under the Apache License, Version 2.0 (the "License"); | ||
you may not use this file except in compliance with the License. | ||
You may obtain a copy of the License at | ||
http://www.apache.org/licenses/LICENSE-2.0 | ||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it looks like its missing some new lines please check
@@ -132,6 +142,134 @@ func (nf *NetworkFence) AddNetworkFence(ctx context.Context) error { | |||
return nil | |||
} | |||
|
|||
func listActiveClients(ctx context.Context, mdsRank string) ([]activeClient, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why to pass mdsRank
as function argument its hardcoded value 0
always?
return parts[1], nil | ||
} | ||
|
||
return "", errors.New("failed to extract IP address") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return default from which client we failed to fetch IP?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just for clarity,
Did you mean to return the ID of client in place of empty string in that case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i mean to return from what we are not able to extra IP address or there should be atleast a log of clientInfo which helps of debugging
return clientID, nil | ||
} | ||
|
||
return 0, errors.New("unable to extract client ID") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any more details we can return in error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Madhu-1, Updated.
Returning the client info too which contains the instance field of client struct, to check the missing format.
Thanks @nixpanic, Updated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few minor things to improve. Generally looks pretty good to me.
} | ||
// if no active client found corresponding to the ip in cidr | ||
// blocklisting entire ip | ||
if !isFound { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is complete yet.
Consider a cidr
that contains multiple worker nodes. Maybe a single worker node has an active CephFS client, and was evicted. All other workernodes in that cidr
need to be blocklisted.
Instead of isFound
as bool, you could create a list of evicted clients, and skip those clients while blocklisting all IP-addresses in the CIDR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nixpanic for the review,
Updated, PTAL.
} | ||
|
||
for _, tt := range tests { | ||
tt := tt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not very fond of this. It would be clearer to use tc := tt
so that tc
is clearly an instance for the text-case. You can then use tc
in all calls below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just needs a few more smaller improvements
} | ||
log.DebugLog(ctx, "client %d has been evicted\n", clientID) | ||
// add the CIDR to the list of blocklisted IPs | ||
evictedIPs[cidr] = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to track the singe client IP-address that was evicted, not the whole CIDR block. There might be other IP-addresses in the CIDR block that are not evicted, and those will need to be blocklisted.
Instead of cidr
as key in the map, use client.fetchIP()
.
|
||
// Parse the stdout to extract blocklist entries | ||
// each entry is a separate line in the output | ||
entries := strings.Split(strings.TrimSpace(stdout), "\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Give an example of the output that is expected from the ceph command. It is difficult to review this without knowing what the command returns.
Even better: move the parsing to a helper function and add a unit-test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Niels, have added comments, explaining the workflow here and in main removeClientEviction too, PTAL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing extra is being done here for parsing, just storing them as new lines. So might not be a need for unit testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything that can be unit-tested, should ideally be unit-tested 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, updated. PTAL 😄
// iterate through CIDR blocks and remove matching entries | ||
for _, cidr := range nf.Cidr { | ||
for _, entry := range blocklistEntries { | ||
if strings.HasPrefix(entry, cidr) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is difficult to verify how this works. Make it easier for others (and for yourself in a few months time) to understand this by adding comments with examples of what cidr
contains, and how enty
is formatted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nixpanic , updated. PTAL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A cidr
in the format of 127.0.0.1:0/3710147553
is weird...
A CIDR could be 192.168.123.234/16
, a blocklisted IP in that range can be 192.168.1.2
(note that the prefix-comparing won't work here). You will need to check if the IP-address from the entry
is included in the range of IP-addresses described by the CIDR. At the moment, the current check is not correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nixpanic ,
Updated.
Testing: riyasinghal@rsinghal-mac ceph-csi % kubectl exec -c csi-addons csi-cephfsplugin-provisioner-676758bc76-7fxr4 -n rook-ceph -- csi-addons -operation NetworkUnFence -clusterid rook-ceph -secret default/csi-cephfs-secret -cidrs 1.1.1.1/32 -endpoint unix:///csi/csi-addons.sock PTAL, cc: @nixpanic , @Madhu-1 , @Rakshith-R |
// iterate through CIDR blocks and remove matching entries | ||
for _, cidr := range nf.Cidr { | ||
for _, entry := range blocklistEntries { | ||
if strings.HasPrefix(entry, cidr) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A cidr
in the format of 127.0.0.1:0/3710147553
is weird...
A CIDR could be 192.168.123.234/16
, a blocklisted IP in that range can be 192.168.1.2
(note that the prefix-comparing won't work here). You will need to check if the IP-address from the entry
is included in the range of IP-addresses described by the CIDR. At the moment, the current check is not correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks complete to me. Left a few minor comments that could improve the code a little more.
var entries []string | ||
for _, line := range lines { | ||
if strings.TrimSpace(line) != "" && !strings.HasPrefix(line, "listed") { | ||
entries = append(entries, line) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are complete entry lines used anywhere? If not, you could call fetchIPfromEntry()
here and add the IP-address to the []string
that is returned.
If that is an option, you could rename getBlocklistEntries()
to getBlocklistIPs()
and make the code a little simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @nixpanic
Updated, PTAL.
// fetch IP from entry | ||
entryIP, err := fetchIPfromEntry(entry) | ||
if err != nil { | ||
return fmt.Errorf("error fetching client IP: %w", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider including the entry
contents in the error message (with %q
). It'll make troubleshooting and debugging much easier.
Pull request has been modified.
Thanks @nixpanic , Updated! |
examples/cephfs/secret.yaml
Outdated
userID: <plaintext ID> | ||
userKey: <Ceph auth key corresponding to ID above> | ||
userID: admin | ||
userKey: AQDY7RNlYlbUGhAASH0UHq8Jn4fvVmGIzsbARQ== | ||
|
||
# Required for dynamically provisioned volumes | ||
adminID: <plaintext ID> | ||
adminKey: <Ceph auth key corresponding to ID above> | ||
adminID: admin | ||
adminKey: AQDY7RNlYlbUGhAASH0UHq8Jn4fvVmGIzsbARQ== |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please revert back this changes
// TestFenceClusterNetwork is a minimal test for the FenceClusterNetwork() | ||
// procedure. During unit-testing, there is no Ceph cluster available, so | ||
// actual operations can not be performed. | ||
func TestFenceClusterNetwork(t *testing.T) { | ||
t.Parallel() | ||
|
||
controller := NewFenceControllerServer() | ||
|
||
req := &fence.FenceClusterNetworkRequest{ | ||
Parameters: map[string]string{}, | ||
Secrets: nil, | ||
Cidrs: nil, | ||
} | ||
|
||
_, err := controller.FenceClusterNetwork(context.TODO(), req) | ||
assert.Error(t, err) | ||
} | ||
|
||
// TestUnfenceClusterNetwork is a minimal test for the UnfenceClusterNetwork() | ||
// procedure. During unit-testing, there is no Ceph cluster available, so actual | ||
// operations can not be performed. | ||
func TestUnfenceClusterNetwork(t *testing.T) { | ||
t.Parallel() | ||
controller := NewFenceControllerServer() | ||
|
||
req := &fence.UnfenceClusterNetworkRequest{ | ||
Parameters: map[string]string{}, | ||
Secrets: nil, | ||
Cidrs: nil, | ||
} | ||
_, err := controller.UnfenceClusterNetwork(context.TODO(), req) | ||
assert.Error(t, err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the benefit we are getting from this unit test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not much, bit it should not panic either.
func listActiveClients(ctx context.Context) ([]activeClient, error) { | ||
// FIXME: replace the ceph command with go-ceph API in future | ||
cmd := []string{"tell", fmt.Sprintf("mds.%s", mdsRank), "client", "ls"} | ||
stdout, stdErr, err := util.ExecCommandWithTimeout(ctx, 5*time.Minute, "ceph", cmd...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RPC timeout is 3 minutes, i would suggest keeping it for 2 minutes or even less.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, will update it.
Apologies for the delay @Madhu-1 , these reviews somehow were not showing up to me before and now its showing 2 weeks ago.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no worries, that's my mistake, i did review but forgot to submit it 🤦🏻
return parts[1], nil | ||
} | ||
|
||
return "", errors.New("failed to extract IP address") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i mean to return from what we are not able to extra IP address or there should be atleast a log of clientInfo which helps of debugging
"testing" | ||
|
||
"github.com/stretchr/testify/assert" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no new line is required
err = nf.addCephBlocklist(ctx, host, false) | ||
if err != nil { | ||
return err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think this can lead to a problem for below case
- Create RBD NetworkFence CR for one IP 10.10.10.10
- Created CephFS NetworkFence CR for IP range but above IP comes in the Range
- Delete the CephFS Network Fence CR to unblocklist the IP
Now the expectation is the IP that is used in RBD NetworkFence is also blocklisted but that's not the case anyone isn't it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that looks possible.
I do not know if it is possible to see if a blocklisted client is CephFS or RBD. If that is returned (or resolvable otherwise) from listing the blocklisted client output, a check for the type can be added. Otherwise, we'll need to com up with some other way of tracking/identifying.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The blocklisted output doesn't have anything to distinguish between two
ex- $ ceph osd blocklist ls
listed 1 entries
127.0.0.1:0/3710147553 2018-03-19 11:32:24.716146
So, one way could be to add a field in NetworkFence struct to track the type of network fence (RBD or CephFS). When creating a NetworkFence object, we can set this field based on the network fence type, and then while evicting we can check if this has only cephfs and not rbd then evict.
WDYT @Madhu-1 , @nixpanic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i dont think we can make any specific change related to cephcsi in the Spec/API. we can decide what to support and what not to support from the cephcsi point. if you are talking about faster recovery of cephfs volume and if that is the whole responsibility of this API you can just document here, use RBD network fence to fence the whole cluster IP and use cephfs network fence to fence any active client which helps to evict only active client, if there are no active client, just return the success or if there any active client in the range just blocklist them and return success. @nixpanic thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess the behavior can be documented as some kind of corner-case. Most workflows will NetweorkFence a node (or whole cluster) for both RBD and CephFS. If there is an issue with the node, neither RBD and CephFS is expected to function. Unfencing one of the protocols suggests the node is expected to be recovered, so both CephFS and RBD are expected to work again too.
We can improve this in a future version of the CSI-Addons Spec, I guess. A NetworkFence operation could pass, and update a NetworkFence.Context, similar to what a PersistentVolume.Context has. In a Kubernetes environment, a NetworkFenceClass might even be appropriate, it could link to secrets and contain driver specific options.
return blocklistIPs, nil | ||
} | ||
|
||
func (nf *NetworkFence) removeBlocklistEntry(ctx context.Context, entry string) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, is the function really different from removeCephBlocklist()
?
https://github.com/ceph/ceph-csi/blob/devel/internal/csi-addons/networkfence/fencing.go#L178
Maybe the same function can be used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right Niels, thanks!
It's the same computation, for the whole un-blocklisting the same logic can be used.
|
||
// RemoveClientEviction unblocks access for all the IPs in the IP range mentioned via the CIDR block | ||
// using a network fence. | ||
func (nf *NetworkFence) RemoveClientEviction(ctx context.Context) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be similar a RemoveNetworkFence()
from https://github.com/ceph/ceph-csi/blob/devel/internal/csi-addons/networkfence/fencing.go#L202
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, one small change is required.
|
||
cr, err := util.NewUserCredentials(req.GetSecrets()) | ||
if err != nil { | ||
return nil, status.Error(codes.Internal, err.Error()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Internal error or invalid? in UnfenceClusterNetwork its Invalid and here its Internal.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Madhu, should be InvalidArgument, updated. PTAL.
Pull request has been modified.
@riya-singhal31 instead of a new commit, please amend this change to the commit in which this was implemented |
Done! |
@Mergifyio rebase |
this commit adds client eviction to cephfs, based on the IPs in cidr block, it evicts those IPs from the network. Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
this commit un-blocklists the clients provided in cidr for unfencing operation. Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
Signed-off-by: Riya Singhal <[email protected]>
✅ Branch has been successfully rebased |
/test ci/centos/upgrade-tests-cephfs |
/test ci/centos/k8s-e2e-external-storage/1.28 |
/test ci/centos/k8s-e2e-external-storage/1.27 |
/test ci/centos/upgrade-tests-rbd |
/test ci/centos/k8s-e2e-external-storage/1.26 |
/test ci/centos/mini-e2e-helm/k8s-1.28 |
/test ci/centos/mini-e2e-helm/k8s-1.27 |
/test ci/centos/mini-e2e/k8s-1.28 |
/test ci/centos/mini-e2e-helm/k8s-1.26 |
/test ci/centos/mini-e2e/k8s-1.27 |
/test ci/centos/mini-e2e/k8s-1.26 |
on the IPs in cidr block, it evicts those IPs from
the network.
updates: #3969