Description of problem: ====================== on an ec volume, stale entries of softlinks are not at all getting cleared even after healing is complete [root@dhcp35-192 ecv]# gluster v heal ecv full Launching heal operation to perform full self heal on volume ecv has been successful Use heal info commands to check status [root@dhcp35-192 ecv]# gluster v heal ecv info Brick dhcp35-192.lab.eng.blr.redhat.com:/rhs/brick2/ecv /var/run /var/lock /var/mail Status: Connected Number of entries: 3 Brick dhcp35-214.lab.eng.blr.redhat.com:/rhs/brick2/ecv /var/run /var/lock /var/mail Status: Connected Number of entries: 3 Brick dhcp35-215.lab.eng.blr.redhat.com:/rhs/brick2/ecv Status: Connected Number of entries: 0 root@dhcp35-214 ecv]# ls /rhs/brick2/ecv/var/ -lh total 8.0K drwxr-xr-x. 2 root root 6 Dec 19 12:45 adm drwxr-xr-x. 5 root root 44 Dec 19 12:46 cache drwxr-xr-x. 2 root root 6 Dec 19 12:46 crash drwxr-xr-x. 3 root root 34 Dec 19 12:46 db drwxr-xr-x. 3 root root 18 Dec 19 12:46 empty drwxr-xr-x. 2 root root 6 Dec 19 12:46 games drwxr-xr-x. 2 root root 6 Dec 19 12:46 gopher drwxr-xr-x. 3 root root 18 Dec 19 12:46 kerberos drwxr-xr-x. 26 root root 4.0K Dec 19 12:45 lib drwxr-xr-x. 2 root root 6 Dec 19 12:46 local lrwxrwxrwx. 2 root root 11 Dec 19 12:45 lock -> ../run/lock drwxr-xr-x. 9 root root 4.0K Dec 19 12:45 log lrwxrwxrwx. 2 root root 10 Dec 19 12:46 mail -> spool/mail drwxr-xr-x. 2 root root 6 Dec 19 12:46 nis drwxr-xr-x. 2 root root 6 Dec 19 12:46 opt drwxr-xr-x. 2 root root 6 Dec 19 12:46 preserve lrwxrwxrwx. 2 root root 6 Dec 19 12:45 run -> ../run drwxr-xr-x. 10 root root 114 Dec 19 12:46 spool drwxr-xr-t. 3 root root 85 Dec 19 12:45 tmp drwxr-xr-x. 2 root root 6 Dec 19 12:46 yp [root@dhcp35-214 ecv]# Version-Release number of selected component (if applicable): [root@dhcp35-78 ~]# rpm -qa|grep gluster glusterfs-rdma-3.12.2-1.el7rhgs.x86_64 glusterfs-server-3.12.2-1.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-3.12.2-1.el7rhgs.x86_64 glusterfs-libs-3.12.2-1.el7rhgs.x86_64 glusterfs-fuse-3.12.2-1.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-1.el7rhgs.x86_64 gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64 glusterfs-api-3.12.2-1.el7rhgs.x86_64 python2-gluster-3.12.2-1.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-1.el7rhgs.x86_64 vdsm-gluster-4.17.33-1.2.el7rhgs.noarch libvirt-daemon-driver-storage-gluster-3.9.0-1.el7.x86_64 glusterfs-cli-3.12.2-1.el7rhgs.x86_64 [root@dhcp35-78 ~]# How reproducible: ================ 2/2 Steps to Reproduce: 1.create a 4+2 ec volume 2.copied /var to mount point 3.from backend deleted var directory on one of the bricks 4. did an ls -lRt on mount 5. issued a heal command to heal files Actual results: ============= all files got healed except below 3 entries which were showing up in heal info , irrespective of number of time heal was triggered. all the files were softlinks [root@dhcp35-192 ecv]# gluster v heal ecv info Brick dhcp35-192.lab.eng.blr.redhat.com:/rhs/brick2/ecv /var/run /var/lock /var/mail Status: Connected Number of entries: 3 Brick dhcp35-214.lab.eng.blr.redhat.com:/rhs/brick2/ecv /var/run /var/lock /var/mail Status: Connected Number of entries: 3 Brick dhcp35-215.lab.eng.blr.redhat.com:/rhs/brick2/ecv Status: Connected Number of entries: 0 root@dhcp35-214 ecv]# ls /rhs/brick2/ecv/var/ -lh total 8.0K drwxr-xr-x. 2 root root 6 Dec 19 12:45 adm drwxr-xr-x. 5 root root 44 Dec 19 12:46 cache drwxr-xr-x. 2 root root 6 Dec 19 12:46 crash drwxr-xr-x. 3 root root 34 Dec 19 12:46 db drwxr-xr-x. 3 root root 18 Dec 19 12:46 empty drwxr-xr-x. 2 root root 6 Dec 19 12:46 games drwxr-xr-x. 2 root root 6 Dec 19 12:46 gopher drwxr-xr-x. 3 root root 18 Dec 19 12:46 kerberos drwxr-xr-x. 26 root root 4.0K Dec 19 12:45 lib drwxr-xr-x. 2 root root 6 Dec 19 12:46 local lrwxrwxrwx. 2 root root 11 Dec 19 12:45 lock -> ../run/lock drwxr-xr-x. 9 root root 4.0K Dec 19 12:45 log lrwxrwxrwx. 2 root root 10 Dec 19 12:46 mail -> spool/mail drwxr-xr-x. 2 root root 6 Dec 19 12:46 nis drwxr-xr-x. 2 root root 6 Dec 19 12:46 opt drwxr-xr-x. 2 root root 6 Dec 19 12:46 preserve lrwxrwxrwx. 2 root root 6 Dec 19 12:45 run -> ../run drwxr-xr-x. 10 root root 114 Dec 19 12:46 spool drwxr-xr-t. 3 root root 85 Dec 19 12:45 tmp drwxr-xr-x. 2 root root 6 Dec 19 12:46 yp
reran above case on 3.12.2-8 on rhel7.5 I now don't see the issue anymore hence marking as verified. all softlinks were healed properly.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607