Hello,
i have VM on local SATA datastore. On a Windows 2008 R2 VM i've installed file services and configured NFS so i exported one VM drive to a ESXi host over NFS. The problem is when i start cloning some other machine or copying it to this datastore it fails every time.
I found in log that ESXi host looses network connectivity to the NFS datastore. Since everything is inside same host a on a same virtual switch it doesn't make much sense how can host loose connectivity to NFS.
This is from the log:
Jan 24 06:24:57 ND-12119 vmkernel: 416:22:56:48.661 cpu3:4119)CBT: 1029: Created device 40bb04a1-cbt for cbt driver with filehandle 1085998241
Jan 24 06:24:57 ND-12119 vmkernel: 416:22:56:48.748 cpu15:229927)CBT: 682: Disconnecting the cbt device 40bb04a1-cbt with filehandle 1085998241
Jan 24 06:24:58 ND-12119 vmkernel: 416:22:56:49.773 cpu3:4118)CBT: 1029: Created device 40c004a1-cbt for cbt driver with filehandle 1086325921
Jan 24 06:24:58 ND-12119 vmkernel: 416:22:56:49.779 cpu15:229926)CBT: 682: Disconnecting the cbt device 40c004a1-cbt with filehandle 1086325921
Jan 24 06:24:58 ND-12119 vmkernel: 416:22:56:49.990 cpu4:4120)CBT: 1029: Created device 40c004a1-cbt for cbt driver with filehandle 1086325921
Jan 24 06:24:58 ND-12119 vmkernel: 416:22:56:50.041 cpu5:229923)CBT: 682: Disconnecting the cbt device 40c004a1-cbt with filehandle 1086325921
Jan 24 06:28:40 ND-12119 vmkernel: 416:23:00:31.994 cpu6:262348)NetPort: 1157: disabled port 0x1000053
Jan 24 06:28:40 ND-12119 vmkernel: 416:23:00:31.995 cpu7:262347)VSCSI: 6495: handle 11642(vscsi0:0):Destroying Device for world 262348 (pendCom 0)
Jan 24 06:28:41 ND-12119 vmkernel: 416:23:00:33.383 cpu4:262348)NetPort: 982: enabled port 0x1000053 with mac 00:50:56:b2:01:2f
Jan 24 06:28:42 ND-12119 vmkernel: 416:23:00:33.475 cpu7:262347)VSCSI: 3482: handle 11653(vscsi0:0):Using sync mode due to sparse disks
Jan 24 06:28:42 ND-12119 vmkernel: 416:23:00:33.475 cpu7:262347)VSCSI: 3523: handle 11653(vscsi0:0):Creating Virtual Device for world 262348 (FSS handle 707134613)
Jan 24 06:28:43 ND-12119 vmkernel: 416:23:00:34.523 cpu4:262348)NetPort: 1157: disabled port 0x1000053
Jan 24 06:28:43 ND-12119 vmkernel: 416:23:00:34.524 cpu7:262347)VSCSI: 6495: handle 11653(vscsi0:0):Destroying Device for world 262348 (pendCom 0)
Jan 24 06:28:44 ND-12119 vmkernel: 416:23:00:35.634 cpu6:262348)NetPort: 982: enabled port 0x1000053 with mac 00:50:56:b2:01:2f
Jan 24 06:28:44 ND-12119 vmkernel: 416:23:00:35.645 cpu5:262347)VSCSI: 3523: handle 11654(vscsi0:0):Creating Virtual Device for world 262348 (FSS handle 708510867)
Jan 24 06:33:01 ND-12119 vmkernel: 416:23:04:52.861 cpu15:4119)CBT: 1029: Created device 3c948499-cbt for cbt driver with filehandle 1016366233
Jan 24 06:33:01 ND-12119 vmkernel: 416:23:04:52.912 cpu15:4119)CBT: 1029: Created device 2f40849d-cbt for cbt driver with filehandle 792757405
Jan 24 06:33:01 ND-12119 vmkernel: 416:23:04:52.917 cpu12:229920)CBT: 682: Disconnecting the cbt device 2f40849d-cbt with filehandle 792757405
Jan 24 06:33:01 ND-12119 vmkernel: 416:23:04:52.922 cpu3:229928)CBT: 682: Disconnecting the cbt device 3c948499-cbt with filehandle 1016366233
Jan 24 06:33:01 ND-12119 vmkernel: 416:23:04:52.975 cpu11:4118)CBT: 1029: Created device 2f41849d-cbt for cbt driver with filehandle 792822941
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.479 cpu5:229927)CBT: 682: Disconnecting the cbt device 2f41849d-cbt with filehandle 792822941
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.736 cpu9:4119)CBT: 1029: Created device 2401049c-cbt for cbt driver with filehandle 604046492
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.762 cpu9:4119)CBT: 1029: Created device 499c84a2-cbt for cbt driver with filehandle 1234994338
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.768 cpu7:229926)CBT: 682: Disconnecting the cbt device 499c84a2-cbt with filehandle 1234994338
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.774 cpu3:229931)CBT: 682: Disconnecting the cbt device 2401049c-cbt with filehandle 604046492
Jan 24 06:33:02 ND-12119 vmkernel: 416:23:04:53.805 cpu13:4118)CBT: 1029: Created device 499d84a2-cbt for cbt driver with filehandle 1235059874
Jan 24 06:33:03 ND-12119 vmkernel: 416:23:04:54.581 cpu7:229922)CBT: 682: Disconnecting the cbt device 499d84a2-cbt with filehandle 1235059874
Jan 24 08:37:25 ND-12119 vmkernel: 417:01:09:17.152 cpu2:4120)NFS: 149: Command: (mount) Server: (10.0.47.132) IP: (10.0.47.132) Path: (DR2) Label: (DR2) Options: (No ne)
Jan 24 08:37:26 ND-12119 vmkernel: 417:01:09:17.595 cpu1:4120)NFS: 287: Restored connection to the server 10.0.47.132 mount point DR2, mounted as d28785c6-d4ede1ed-00 00-000000000000 ("DR2")
Jan 24 08:37:26 ND-12119 vmkernel: 417:01:09:17.595 cpu1:4120)NFS: 171: NFS mount 10.0.47.132:DR2 status: Success
Jan 24 08:48:50 ND-12119 vmkernel: 417:01:20:42.154 cpu9:229931)CBT: 682: Disconnecting the cbt device 49e98497-cbt with filehandle 1240040599
Jan 24 08:48:58 ND-12119 vmkernel: 417:01:20:49.833 cpu13:4119)CBT: 1029: Created device 2f45849d-cbt for cbt driver with filehandle 793085085
Jan 24 08:48:58 ND-12119 vmkernel: 417:01:20:49.837 cpu11:229930)CBT: 682: Disconnecting the cbt device 2f45849d-cbt with filehandle 793085085
Jan 24 08:48:58 ND-12119 vmkernel: 417:01:20:50.032 cpu11:4120)CBT: 1029: Created device 2f45849d-cbt for cbt driver with filehandle 793085085
Jan 24 08:48:59 ND-12119 vmkernel: 417:01:20:50.886 cpu7:229932)CBT: 682: Disconnecting the cbt device 2f45849d-cbt with filehandle 793085085
Jan 24 08:49:20 ND-12119 vmkernel: 417:01:21:11.890 cpu12:4120)CBT: 1029: Created device 2bc1049b-cbt for cbt driver with filehandle 734069915
Jan 24 08:49:20 ND-12119 vmkernel: 417:01:21:11.949 cpu9:4120)CBT: 1029: Created device 5142849f-cbt for cbt driver with filehandle 1363313823
Jan 24 08:49:20 ND-12119 vmkernel: 417:01:21:11.953 cpu9:229926)CBT: 682: Disconnecting the cbt device 5142849f-cbt with filehandle 1363313823
Jan 24 08:49:20 ND-12119 vmkernel: 417:01:21:11.957 cpu9:229929)CBT: 682: Disconnecting the cbt device 2bc1049b-cbt with filehandle 734069915
Jan 24 08:49:20 ND-12119 vmkernel: 417:01:21:12.009 cpu15:4119)CBT: 1029: Created device 5143849f-cbt for cbt driver with filehandle 1363379359
Jan 24 08:49:21 ND-12119 vmkernel: 417:01:21:12.711 cpu6:229933)CBT: 682: Disconnecting the cbt device 5143849f-cbt with filehandle 1363379359
Jan 24 08:52:52 ND-12119 vmkernel: 417:01:24:43.609 cpu9:4114)WARNING: NFS: 278: Lost connection to the server 10.0.47.132 mount point DR2, mounted as d28785c6-d4ede1 ed-0000-000000000000 ("DR2")
Jan 24 08:53:03 ND-12119 vmkernel: 417:01:24:54.617 cpu10:4114)NFSLock: 579: Stop accessing fd 0x4100047443d8 4
Jan 24 08:55:11 ND-12119 vmkernel: 417:01:27:02.840 cpu8:4120)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:11 ND-12119 vmkernel: 417:01:27:02.840 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:11 ND-12119 vmkernel: 417:01:27:02.841 cpu8:4119)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:11 ND-12119 vmkernel: 417:01:27:02.841 cpu8:4119)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu14:4118)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu8:4120)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu14:4118)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.722 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu0:4096)VMNIX: VMKFS: 2561: status = -5
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4120)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu0:4096)VMNIX: VMKFS: 2561: status = -5
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4119)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4119)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4120)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4119)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu8:4119)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :No connection
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)WARNING: NFS: 1830: Failed to get attributes (No connection)
Jan 24 08:55:13 ND-12119 vmkernel: 417:01:27:04.723 cpu14:4118)FSS: 735: Failed to get object b00f 40 d28785c6 d4ede1ed 5 50000 5 50000 35314047 652 20d86ed5 0 0 0 :N o connection
Jan 24 08:57:38 ND-12119 vmkernel: 417:01:29:29.463 cpu4:4100)NFS: 287: Restored connection to the server 10.0.47.132 mount point DR2, mounted as d28785c6-d4ede1ed-00 00-000000000000 ("DR2")
Jan 24 08:58:59 ND-12119 vmkernel: 417:01:30:50.815 cpu9:4120)WARNING: NFS: 1744: oid 0x417f800c7d52 volumeHash ffffffa002ffffffffffffffd26133ffffff9b6a not found
Jan 24 08:58:59 ND-12119 vmkernel: 417:01:30:50.815 cpu9:4120)FSS: 735: Failed to get object b00f 12 d2ff02a0 6a9b3361 ffffffff 30000001 5 295e 0 800c7eef 5e288900000 417f 800c7ec000004100 2060000417f f4abab3000000000 :Not found
Jan 24 08:58:59 ND-12119 vmkernel: 417:01:30:50.815 cpu9:4120)WARNING: NFS: 1815: Failed to reserve volume (Not found) b00f 12 d2ff02a0 6a9b3361 ffffffff 30000001 5 2 95e 0 800c7eef 5e288900000417f 800c7ec000004100 2060000417f f4abab3000000000
Jan 24 08:58:59 ND-12119 vmkernel: 417:01:30:50.815 cpu9:4120)FSS: 735: Failed to get object b00f 40 d2ff02a0 6a9b3361 3d 90000 5 50000 35314047 652 6f91aa26 0 0 0 :N ot found
Jan 24 08:58:59 ND-12119 vmkernel: 417:01:30:50.815 cpu0:4096)VMNIX: VMKFS: 2561: status = -2