Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrent Map Writes With CephFS Plugin #4162

Closed
dpeckett opened this issue Oct 4, 2023 · 1 comment · Fixed by #4163
Closed

Concurrent Map Writes With CephFS Plugin #4162

dpeckett opened this issue Oct 4, 2023 · 1 comment · Fixed by #4163
Labels
bug Something isn't working component/cephfs Issues related to CephFS good first issue Good for newcomers

Comments

@dpeckett
Copy link

dpeckett commented Oct 4, 2023

Describe the bug

The cephfsplugin sometimes crashes with a concurrent map write error.

Environment details

  • Image/version of Ceph CSI driver : v3.9.0
  • Kernel version : 5.15.49-linuxkit-pr
  • Mounter: cephfs kernel
  • Kubernetes cluster version : v1.28.2
  • Ceph cluster version : v18.2.0

Steps to reproduce

Create a new cephcluster, a cephfs storageclass, and several PVCs.

Actual results

When first starting the cephfs provisioner will sometimes crash with a concurrent map write error. After it crashes the provisioner pod restarts and the provisioning succeeds.

Expected behavior

The provisioner should start without erroring out.

Logs

E1004 12:09:55.634003       1 omap.go:79] ID: 17 Req-ID: pvc-611be305-bdd2-42b4-b496-4aa1d5fb7e2f omap not found (pool="koopt-metadata", namespace="csi", name="csi.volumes.default"): rados: ret=-2, No such file or directory
E1004 12:09:55.634011       1 omap.go:79] ID: 18 Req-ID: pvc-4494ed0a-1aa7-4788-b3cc-4c290ecf06d8 omap not found (pool="koopt-metadata", namespace="csi", name="csi.volumes.default"): rados: ret=-2, No such file or directory
E1004 12:09:55.634199       1 omap.go:79] ID: 19 Req-ID: pvc-3b0afecb-96fe-4590-bf0f-1acdd1cf06f2 omap not found (pool="koopt-metadata", namespace="csi", name="csi.volumes.default"): rados: ret=-2, No such file or directory
fatal error: concurrent map writes

goroutine 74 [running]:
github.com/ceph/ceph-csi/internal/cephfs/core.newLocalClusterState(...)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/core/volume.go:218
github.com/ceph/ceph-csi/internal/cephfs/core.(*subVolumeClient).CreateVolume(0xc00012d740, {0x26525e8, 0xc0007968a0})
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/core/volume.go:225 +0xaa
github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).createBackingVolume(0xc0000d5400, {0x26525e8, 0xc0007968a0}, 0xc000939e00, 0x0, 0xc0007672c0, 0x0, 0x0, 0x0?)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/controllerserver.go:96 +0x333
github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).CreateVolume(0xc0000d5400, {0x26525e8, 0xc0007968a0}, 0xc0003e0850)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/controllerserver.go:386 +0x15fd
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x26525e8, 0xc0007968a0}, {0x2069760?, 0xc0003e0850})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6420 +0x78
github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x26525e8?, 0xc0007968a0?}, {0x2069760?, 0xc0003e0850?}, 0xc00040e900?, 0x2660c01?)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:233 +0x82
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc0007968a0?}, {0x2069760?, 0xc0003e0850?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x26525e8, 0xc0007968a0}, {0x2069760?, 0xc0003e0850}, 0x1d75200?, 0xc00012d200)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:208 +0x372
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc0007968a0?}, {0x2069760?, 0xc0003e0850?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x26525e8, 0xc000796780}, {0x2069760, 0xc0003e0850}, 0xc0009362a0?, 0xc00012d2c0)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:186 +0x13f
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2({0x26525e8, 0xc000796780}, {0x2069760, 0xc0003e0850}, 0xc0000ca0e0, 0x1e750c0?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x14e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x20d5da0?, 0xc0000d5400}, {0x26525e8, 0xc000796780}, 0xc0003e0700, 0xc0000d9a10)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6422 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc0000dd7a0, 0xc0000d9bf0, 0x38242a0, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc0000dd7a0, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:957 +0x18c

goroutine 1 [semacquire, 2 minutes]:
sync.runtime_Semacquire(0x265e3c0?)
	/usr/local/go/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0x0?)
	/usr/local/go/src/sync/waitgroup.go:116 +0x4b
github.com/ceph/ceph-csi/internal/csi-common.(*nonBlockingGRPCServer).Wait(0x2643af0?)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/server.go:75 +0x1b
github.com/ceph/ceph-csi/internal/cephfs.(*Driver).Run(0xc00061fef8, 0x38d5f60)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/driver.go:170 +0xe36
main.main()
	/go/src/github.com/ceph/ceph-csi/cmd/cephcsi.go:241 +0x4b6

goroutine 54 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f86a689b0d8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0000ea400?, 0x20?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0000ea400)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc0000ea400)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x450a60?)
	/usr/local/go/src/net/unixsock_posix.go:172 +0x1c
net.(*UnixListener).Accept(0xc0000d99e0)
	/usr/local/go/src/net/unixsock.go:260 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc000848000, {0x2651140?, 0xc0000d99e0})
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:821 +0x475
github.com/ceph/ceph-csi/internal/csi-common.(*nonBlockingGRPCServer).serve(0xc0002e64c8, {0x7ffda2b59c7e?, 0x18ee3b5?}, {0x215b642, 0x7}, {{0x2643af0, 0xc000120480}, {0x265e3c0, 0xc0000d5400}, {0x26591b0, ...}, ...}, ...)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/server.go:150 +0x990
created by github.com/ceph/ceph-csi/internal/csi-common.(*nonBlockingGRPCServer).Start
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/server.go:70 +0x18b

goroutine 83 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cd520)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1155 +0x233
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:344 +0x1bf8

goroutine 43 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000c9270, 0x1)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000749ea0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x91
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0xda
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:338 +0x1bb3

goroutine 44 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cd040)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1155 +0x233
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:344 +0x1bf8

goroutine 45 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f86a689afe8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0000ea600?, 0xc000860000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0000ea600, {0xc000860000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0000ea600, {0xc000860000?, 0x60100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000120658, {0xc000860000?, 0x448600?, 0xc00084fcc8?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0000c6d80, {0xc0000e44a0, 0x9, 0xc00084fcf0?})
	/usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x26338c0, 0xc0000c6d80}, {0xc0000e44a0, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0000e44a0?, 0x9?, 0x26381e0?}, {0x26338c0?, 0xc0000c6d80?})
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000e4460)
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cd040, 0x0?, 0x0?)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:642 +0x167
google.golang.org/grpc.(*Server).serveStreams(0xc000848000, {0x2659fc0?, 0xc0000cd040})
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:946 +0x162
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:889 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:888 +0x185

goroutine 82 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000aaff0, 0x1)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003e0930)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x91
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0xda
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:338 +0x1bb3

goroutine 89 [runnable]:
github.com/ceph/go-ceph/rados._Cfunc_rados_write_op_operate2(0x5161760, 0x7f863c008a70, 0x51722b0, 0x0, 0x0)
	_cgo_gotypes.go:1880 +0x4c
github.com/ceph/go-ceph/rados.WriteOp.operate2.func2(0x51722b0?, 0x2f?, 0x3?, 0x3?, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/rados/write_op.go:59 +0x9d
github.com/ceph/go-ceph/rados.WriteOp.operate2({{{0x0, 0x0, 0x0}}, 0x5161760}, 0xc0007a31f0, {0xc0005f08d0?, 0x0?}, 0x0, 0x0?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/rados/write_op.go:59 +0xfb
github.com/ceph/go-ceph/rados.(*WriteOp).Operate(...)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/rados/write_op.go:65
github.com/ceph/go-ceph/rados.(*WriteOp).operateCompat(0x5161760?, 0xc0007acb48?, {0xc0005f08d0?, 0x420397?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/rados/write_op.go:77 +0x4f
github.com/ceph/go-ceph/rados.(*IOContext).SetOmap(0xc0007acc08?, {0xc0005f08d0, 0x2f}, 0x10?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/rados/omap.go:105 +0xee
github.com/ceph/ceph-csi/internal/journal.setOMapKeys({0x26525e8, 0xc00041b380}, 0x10?, {0xc00005d8e0, 0xe}, {0x2157b31, 0x3}, {0xc0005f08d0, 0x2f}, 0xc000797290)
	/go/src/github.com/ceph/ceph-csi/internal/journal/omap.go:151 +0x369
github.com/ceph/ceph-csi/internal/journal.(*Connection).ReserveName(0xc000797140, {0x26525e8, 0xc00041b380}, {0xc00005d8e0, 0xe}, 0xffffffffffffffff, {0xc00005d8e0, 0xe}, 0xffffffffffffffff, {0xc00059e150, ...}, ...)
	/go/src/github.com/ceph/ceph-csi/internal/journal/voljournal.go:672 +0xad9
github.com/ceph/ceph-csi/internal/cephfs/store.ReserveVol({0x26525e8, 0xc00041b380}, 0xc00017af00, 0xc00017af00?)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/store/fsjournal.go:291 +0x2b9
github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).CreateVolume(0xc0000d5400, {0x26525e8, 0xc00041b380}, 0xc00018e3f0)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/controllerserver.go:368 +0x1372
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x26525e8, 0xc00041b380}, {0x2069760?, 0xc00018e3f0})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6420 +0x78
github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x26525e8?, 0xc00041b380?}, {0x2069760?, 0xc00018e3f0?}, 0xc0006c66c0?, 0x2660c01?)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:233 +0x82
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc00041b380?}, {0x2069760?, 0xc00018e3f0?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x26525e8, 0xc00041b380}, {0x2069760?, 0xc00018e3f0}, 0x1d75200?, 0xc0000d5480)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:208 +0x372
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc00041b380?}, {0x2069760?, 0xc00018e3f0?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x26525e8, 0xc00041a330}, {0x2069760, 0xc00018e3f0}, 0xc000012bd0?, 0xc0000d54c0)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:186 +0x13f
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2({0x26525e8, 0xc00041a330}, {0x2069760, 0xc00018e3f0}, 0xc000766860, 0x1e750c0?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x14e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x20d5da0?, 0xc0000d5400}, {0x26525e8, 0xc00041a330}, 0xc00018e2a0, 0xc0000d9a10)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6422 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc000287b00, 0xc0000d9bf0, 0x38242a0, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc000287b00, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:957 +0x18c

goroutine 59 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000aa6e0, 0x1)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00018e230)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x91
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0xda
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:338 +0x1bb3

goroutine 60 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cc680)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1155 +0x233
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:344 +0x1bf8

goroutine 61 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f86a689aef8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000949c00?, 0xc000960000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000949c00, {0xc000960000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000949c00, {0xc000960000?, 0x60100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0002f2c30, {0xc000960000?, 0x448600?, 0xc00095fcc8?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00092f620, {0xc000186f20, 0x9, 0xc00095fcf0?})
	/usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x26338c0, 0xc00092f620}, {0xc000186f20, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000186f20?, 0x9?, 0x26381e0?}, {0x26338c0?, 0xc00092f620?})
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000186ee0)
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cc680, 0x0?, 0x0?)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:642 +0x167
google.golang.org/grpc.(*Server).serveStreams(0xc000848000, {0x2659fc0?, 0xc0000cc680})
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:946 +0x162
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:889 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:888 +0x185

goroutine 88 [runnable]:
github.com/ceph/go-ceph/internal/commands.MarshalMgrCommand({0x7f867c0a8b58, 0xc0004017f0}, {0x1cedc40?, 0xc00018f420?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/internal/commands/commands.go:28 +0x17e
github.com/ceph/go-ceph/cephfs/admin.(*FSAdmin).marshalMgrCommand(...)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/cephfs/admin/fsadmin.go:68
github.com/ceph/go-ceph/cephfs/admin.(*FSAdmin).CreateSubVolumeGroup(0xc000626f38, {0xc0005fe040?, 0xc0005fe040?}, {0x2157b31?, 0xc000626f28?}, 0x5056c5?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/ceph/go-ceph/cephfs/admin/subvolumegroup.go:49 +0xa5
github.com/ceph/ceph-csi/internal/cephfs/core.(*subVolumeClient).CreateVolume(0xc00077ae00, {0x26525e8, 0xc000796120})
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/core/volume.go:237 +0x2f6
github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).createBackingVolume(0xc0000d5400, {0x26525e8, 0xc000796120}, 0xc000939980, 0x0, 0xc0007671c0, 0x0, 0x0, 0x0?)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/controllerserver.go:96 +0x333
github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).CreateVolume(0xc0000d5400, {0x26525e8, 0xc000796120}, 0xc0003e01c0)
	/go/src/github.com/ceph/ceph-csi/internal/cephfs/controllerserver.go:386 +0x15fd
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x26525e8, 0xc000796120}, {0x2069760?, 0xc0003e01c0})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6420 +0x78
github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x26525e8?, 0xc000796120?}, {0x2069760?, 0xc0003e01c0?}, 0xc00040e480?, 0x2660c01?)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:233 +0x82
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc000796120?}, {0x2069760?, 0xc0003e01c0?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x26525e8, 0xc000796120}, {0x2069760?, 0xc0003e01c0}, 0x1d75200?, 0xc00012c640)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:208 +0x372
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0x26525e8?, 0xc000796120?}, {0x2069760?, 0xc0003e01c0?})
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x4f
github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x26525e8, 0xc000796000}, {0x2069760, 0xc0003e01c0}, 0xc000936018?, 0xc00012c6c0)
	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:186 +0x13f
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2({0x26525e8, 0xc000796000}, {0x2069760, 0xc0003e01c0}, 0xc0000ca000, 0x1e750c0?)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x14e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x20d5da0?, 0xc0000d5400}, {0x26525e8, 0xc000796000}, 0xc0003e0070, 0xc0000d9a10)
	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6422 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc0002879e0, 0xc0000d9bf0, 0x38242a0, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000848000, {0x2659fc0, 0xc0000cd520}, 0xc0002879e0, 0x0)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:957 +0x18c

goroutine 66 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000ab180, 0x1)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00018e850)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:552 +0x91
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0xda
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:338 +0x1bb3

goroutine 67 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cd860)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1155 +0x233
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:344 +0x1bf8

goroutine 68 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f86a689ad18, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000412200?, 0xc0005e6000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000412200, {0xc0005e6000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000412200, {0xc0005e6000?, 0x60100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0002f2f88, {0xc0005e6000?, 0x448600?, 0xc00008ecc8?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00092fc20, {0xc0001870e0, 0x9, 0xc00008ecf0?})
	/usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x26338c0, 0xc00092fc20}, {0xc0001870e0, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0001870e0?, 0x9?, 0x26381e0?}, {0x26338c0?, 0xc00092fc20?})
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001870a0)
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cd860, 0x0?, 0x0?)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:642 +0x167
google.golang.org/grpc.(*Server).serveStreams(0xc000848000, {0x2659fc0?, 0xc0000cd860})
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:946 +0x162
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:889 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:888 +0x185

goroutine 84 [IO wait]:
internal/poll.runtime_pollWait(0x7f86a689ae08, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000412080?, 0xc0005a0000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000412080, {0xc0005a0000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000412080, {0xc0005a0000?, 0x1060100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0002f2f70, {0xc0005a0000?, 0x18?, 0xc000078800?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00092fbc0, {0xc000187000, 0x9, 0x0?})
	/usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x26338c0, 0xc00092fbc0}, {0xc000187000, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000187000?, 0x9?, 0xc000058d38?}, {0x26338c0?, 0xc00092fbc0?})
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000186fc0)
	/go/src/github.com/ceph/ceph-csi/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cd520, 0x1ef8b80?, 0x1951ff1?)
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/internal/transport/http2_server.go:642 +0x167
google.golang.org/grpc.(*Server).serveStreams(0xc000848000, {0x2659fc0?, 0xc0000cd520})
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:946 +0x162
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:889 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:888 +0x185
@Madhu-1
Copy link
Collaborator

Madhu-1 commented Oct 4, 2023

ceph-csi/internal/cephfs/core/volume.go

Line 40 in cba5402

var clusterAdditionalInfo = make(map[string]*localClusterState)
map needs to be replaced with sync map.

@Madhu-1 Madhu-1 added bug Something isn't working good first issue Good for newcomers component/cephfs Issues related to CephFS labels Oct 4, 2023
@Madhu-1 Madhu-1 mentioned this issue Oct 5, 2023
Merged
@Rakshith-R Rakshith-R linked a pull request Oct 6, 2023 that will close this issue
cephfs: safeguard localClusterState struct from race conditions #4163
Merged
@mergify mergify bot closed this as completed in #4163 Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component/cephfs Issues related to CephFS good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cephfs: safeguard localClusterState struct from race conditions Rakshith-R/ceph-csi
2 participants
@dpeckett @Madhu-1