• Resolving GFID mismatch problems in Gluster (RHGS) volumes

    Gluster is a distributed filesystem. I'm not a massive fan of it, but most of the alternatives (like Ceph) suffer with their own set of issues, so it's no better or worse than the competition of the most part.

    One issue that can sometimes occur is Gluster File ID (GFID) mismatch following a split-brain or similar failure.

    When this occurs, running ls -l in a directory will generally lead to I/O errors and/or question marks in the output

    ls -i
    ls: cannot access ban-gai.rss: Input/output error
    ? 2-nguoi-choi.rss ? game.rss
    

    If you look within the brick's log (normally under /var/log/glusterfs/bricks) you'll see lines reporting Gfid mismatch detected 

    [2019-12-12 12:28:28.100417] E [MSGID: 108008] [afr-self-heal-common.c:392:afr_gfid_split_brain_source] 0-shared-replicate-0: Gfid mismatch detected for <gfid:31bcb959-efb4-46bf-b858-7f964f0c699d>/ban-gai.rss>, 1c7a16fe-3c6c-40ee-8bb4-cb4197b5035d on shared-client-4 and fbf516fe-a67e-4fd3-b17d-fe4cfe6637c3 on shared-client-1.
    [2019-12-12 12:28:28.113998] W [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 31bcb959-efb4-46bf-b858-7f964f0c699d/ban-gai.rss: failed to resolve (Stale file handle)
    

    This documentation details how to resolve GFID mismatches