sysconf(_SC_GETGR_R_SIZE_MAX) is just a hint for getgrnam_r()
When there is a group that has a lot of members, TestProcess#test_execopts_gid fails. Following is a more simple example:
$ ruby -e 'system("true", gid: "largegroup")' -e:1:in `system': Numerical result out of range - getgrnam_r (Errno::ERANGE) from -e:1:in `<main>'
This wrong exception happens because obj2gid() in process.c calls getgrnam_r() with a buffer that was allocated based on the size returned by sysconf(_SC_GETGR_R_SIZE_MAX). In Linux, sysconf(_SC_GETGR_R_SIZE_MAX) returns 1024, but actually this value is just a hint, so obj2gid() should gradually extend the buffer until getgrnam_r() no longer throws ERANGE.
The same issue was reported in the following page: