Bug #3231: Digest Does Not Build

• Status: Closed
• Assignee: nahi (Hiroshi Nakamura)
• Target version: -
• ruby -v: HEAD
• Backport: 2.0.0: REQUIRED, 2.1: DONE, 2.2: DONE

[cfis (Charlie Savage)]

=begin
Revisiting this one from a few weeks back. MD5/rmd160/sha1/sha2 do not build using VC 2010. They do build with Mingw (mingw + msys). See compiler logs below.

Focusing on MD5 (they all have the same issue):

extconf.h is generated like this:

#ifndef EXTCONF_H
#define EXTCONF_H
#define HAVE_CONFIG_H 1
#define HAVE_OPENSSL_MD5_H 1
#endif

Then in md5init.c:

#include "digest.h"
#if defined(HAVE_OPENSSL_MD5_H)
#include "md5ossl.h"
#else
#include "md5.h"
#endif

Since extconf.h is never passed to the compiler, the error happens. Adding this to the top of the file:

#define HAVE_OPENSSL_MD5_H

Fixes the issue. Clearly that's a hack, but I don't understand how extconf.h is used in building the extension since the only place it comes into play is in the generated makefile:

$(OBJS): $(RUBY_EXTCONF_H)

So not sure the solution, but the problem is that having openssl installed correctly breaks the VC digest extension build.

---

1. Digest/MD5

   cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/md5.so md5init.obj md5ossl.obj msvcr100-ruby191.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:md5-i386-mswin32_100.lib -pdb:md5-i386-mswin32_100.pdb -def:md5-i386-mswin32_100.def
   Creating library md5-i386-mswin32_100.lib and object md5-i386-mswin32_100.exp
   md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Finish
   md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Update
   md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Init
   ../../../.ext/i386-mswin32_100/digest/md5.so : fatal error LNK1120: 3 unresolved externals
   NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
   Stop.

2. Digest/rmd160

    cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/rmd160.so rmd160init.obj rmd160ossl.obj msvcr100-ruby191.lib crypto.lib  unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:rmd160-i386-mswin32_100.lib -pdb:rmd160-i386-mswin32_100.pdb -def:rmd160-i386-mswin32_100.def
   

   Creating library rmd160-i386-mswin32_100.lib and object rmd160-i386-mswin32_100.exp
   rmd160init.obj : error LNK2001: unresolved external symbol _rb_Digest_RMD160_Finish
   rmd160init.obj : error LNK2001: unresolved external symbol _rb_Digest_RMD160_Update
   rmd160init.obj : error LNK2001: unresolved external symbol _rb_Digest_RMD160_Init
   ../../../.ext/i386-mswin32_100/digest/rmd160.so : fatal error LNK1120: 3 unresolved externals
   NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'

3. Digest/sha1

   cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/sha1.so sha1init.obj sha1ossl.obj msvcr100-ruby191.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:sha1-i386-mswin32_100.lib -pdb:sha1-i386-mswin32_100.pdb -def:sha1-i386-mswin32_100.def
   Creating library sha1-i386-mswin32_100.lib and object sha1-i386-mswin32_100.exp
   sha1init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA1_Finish
   sha1init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA1_Update
   sha1init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA1_Init
   ../../../.ext/i386-mswin32_100/digest/sha1.so : fatal error LNK1120: 3 unresolved externals
   NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
   Stop.

4. Digest/sha2
   cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/sha2.so sha2init.obj sha2ossl.obj msvcr100-ruby191.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:sha2-i386-mswin32_100.lib -pdb:sha2-i386-mswin32_100.pdb -def:sha2-i386-mswin32_100.def
   Creating library sha2-i386-mswin32_100.lib and object sha2-i386-mswin32_100.exp
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA512_Finish
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA512_Update
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA512_Init
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA384_Finish
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA384_Update
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA384_Init
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA256_Finish
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA256_Update
   sha2init.obj : error LNK2001: unresolved external symbol _rb_Digest_SHA256_Init
   ../../../.ext/i386-mswin32_100/digest/sha2.so : fatal error LNK1120: 9 unresolved externals
   NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
   Stop.

c:\Development\src\ruby\ext\digest\sha2>
=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Sat, 1 May 2010 15:12:35 +0900,
Charlie Savage wrote in [ruby-core:29911]:

Since extconf.h is never passed to the compiler, the error
happens. Adding this to the top of the file:

CPPFLAGS should have -DRUBY_EXTCONF_H="$(RUBY_EXTCONF_H)",
and include/ruby/ruby.h has

#ifdef RUBY_EXTCONF_H
#include RUBY_EXTCONF_H
#endif

How is CPPFLAGS defined in Makefile?

--
Nobu Nakada

=end

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Feedback

=begin

=end

[cfis (Charlie Savage)]

• File Makefile Makefile added

=begin
Ok, I see what is happening. The include is pulling in digest/extconf.h, not digest/md5/extconf.h. Thus the compile breaks.

The makefile has this:

CPPFLAGS = -DRUBY_EXTCONF_H="$(RUBY_EXTCONF_H)"

Include/ruby/ruby.h (in the source tree) has this:

#ifdef RUBY_EXTCONF_H
#include RUBY_EXTCONF_H
#endif

But if you do this:

cd /ext/digest/md5
nmake

Then for whatever reason the wrong extconf.h file is being used.

Attaching the generated makefile if it helps.

=end

[nobu (Nobuyoshi Nakada)]

• Category set to ext

=begin
Could you show an mkmf.log?
And from where have you installed OpenSSL?
=end

[cfis (Charlie Savage)]

• File mkmf.log mkmf.log added
• File Makefile Makefile added

=begin
Sure, attached.

OpenSSL (v1.0) is installed to c:\Development\msys\local\bin and c:\Development\msys\local\include.

This is where I have other 3rd party libraries installed (iconv, zlib, libxml, etc.) and those extensions build fine as does the openssl extension. So this isn't a path issue.
=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Sun, 9 May 2010 08:12:27 +0900,
Charlie Savage wrote in [ruby-core:30108]:

OpenSSL (v1.0) is installed to c:\Development\msys\local\bin
and c:\Development\msys\local\include.

How were those directory names given to the compiler? The
mkmf.log doesn't seem contain them.

have_library: checking for main() in crypto.lib... -------------------- yes
"cl -nologo -Feconftest -I../../../.ext/include/i386-mswin32_100 -I../../.././include -I../../.././ext/digest/md5 -I../../.././ext/digest/md5/.. -I../../.././include -MD -Zi -W2 -wd4996 -Od -Zm600 conftest.c msvcr100-ruby191-static.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib -link -incremental:no -debug -opt:ref -opt:icf -libpath:. -libpath:../../.. "

have_header: checking for openssl/md5.h... -------------------- yes
"cl -nologo -I../../../.ext/include/i386-mswin32_100 -I../../.././include -I../../.././ext/digest/md5 -I../../.././ext/digest/md5/.. -MD -Zi -W2 -wd4996 -Od -Zm600 -c conftest.c"

And from where have you downloaded and installed that OpenSSL?

--
Nobu Nakada

=end

[cfis (Charlie Savage)]

=begin

1. Open VC command prompt.
2. set INCLUDE=c:\Development\msys\local\include;%INCLUDE%
3. set LIB=c:\Development\msys\local\lib;%LIB%
4. cd c:\Development\src\ruby
5. nmake

(Note I have previously run win32/configure.bat)

Note the openssl includes and libs are being found just fine as you can see in mkmf.log
=end

[cfis (Charlie Savage)]

=begin
Missed the openssl question. Its the official release, from http://www.openssl.org/source. Builds fine, and works fine with the openssl extension as previously noted. And works fine with digest too with including extconf.h from the digest directory (versus the parent directory).
=end

[usa (Usaku NAKAMURA)]

=begin
Does this patch help you?
(only ext/digest/md5)

Index: ext/digest/md5/extconf.rb

--- ext/digest/md5/extconf.rb (revision 28034)
+++ ext/digest/md5/extconf.rb (working copy)
@@ -4,7 +4,6 @@
require "mkmf"

$defs << "-DHAVE_CONFIG_H"
-$INCFLAGS << " -I$(srcdir)/.."

$objs = [ "md5init.#{$OBJEXT}" ]

Index: ext/digest/md5/md5init.c

--- ext/digest/md5/md5init.c (revision 28034)
+++ ext/digest/md5/md5init.c (working copy)
@@ -1,7 +1,7 @@
/* $RoughId: md5init.c,v 1.2 2001/07/13 19:49:10 knu Exp $ /
/ $Id$ */

-#include "digest.h"
+#include "../digest.h"
#if defined(HAVE_OPENSSL_MD5_H)
#include "md5ossl.h"
#else
Index: ext/digest/md5/md5.h

--- ext/digest/md5/md5.h (revision 28034)
+++ ext/digest/md5/md5.h (working copy)
@@ -46,7 +46,7 @@
#ifndef MD5_INCLUDED

define MD5_INCLUDED

-#include "defs.h"
+#include "../defs.h"

/*

• This code has some adaptations for the Ghostscript environment, but it

=end

[cfis (Charlie Savage)]

=begin
Yes - it does fix the issue for md5. Can this also be done for the other digest subdirectories?

Thanks for the help.

=end

[mame (Yusuke Endoh)]

• Status changed from Feedback to Open
• Assignee set to usa (Usaku NAKAMURA)
• Target version set to 1.9.2

=begin
Hi, Usaku

Could you please commit your patch?

--
Yusuke Endoh [e-mail]
=end

[usa (Usaku NAKAMURA)]

• Status changed from Open to Assigned
• Assignee changed from usa (Usaku NAKAMURA) to knu (Akinori MUSHA)

=begin
This patch shows only concept to fix.
The maintainer should think how to do now. It's not my task.
=end

[knu (Akinori MUSHA)]

=begin
I'll handle this.

This fix is okay for me because defs.h is intended to be for internal use inside ext/digest/.
=end

[knu (Akinori MUSHA)]

=begin
I think I misread the patch.
Digest submodules should be able to include digest.h without having to use a relative path form, while that's not the case for defs.h.

Let me think awhile about what the real problem is here.
I won't object to putting the proposed fix into ruby_1_9_2 for the forthcoming release, but for trunk as a permanent fix I'm not quite convinced.
=end

[knu (Akinori MUSHA)]

• File dotdotslash.patch dotdotslash.patch added

=begin

=end

[usa (Usaku NAKAMURA)]

=begin
In building process, both ext/digest/extconf.h and ext/digest/md5/extconf.h exist.
If you specify -I.. in ext/digest/md5 and #include <extconf.h>, compiler cannot
decide to include whether ext/digest/extconf.h or ext/digest/md5/extconf.h.

Gcc gives priority to the place specified by the rear -I, but of course this
behavior is not the spec of any standards. It's merely implementation dependence.
And, VC10 actually shows different behavior.
This is the real problem.

My conclusion is that you should specify to complier what do you actually want
to include.
=end

[knu (Akinori MUSHA)]

=begin
Thanks so much for the detailed explanation. Can I ask you something? Does VC++ search directories in random order, or does it work under any specific rules that changed in VC++ 2010?

In Unix, there is a spec in SUS that the order of specifying -I/-L options is significant for the compiler commands, and they shall search directories named in those options in the order specified.
=end

[mame (Yusuke Endoh)]

• Target version changed from 1.9.2 to 2.0.0

=begin
Hi,

2010/6/15 Akinori MUSHA [e-mail]:

Thanks so much for the detailed explanation. ?Can I ask you something? ?Does VC++ search directories in random order, or does it work under any specific rules that changed in VC++ 2010?

In Unix, there is a spec in SUS that the order of specifying -I/-L options is significant for the compiler commands, and they shall search directories named in those options in the order specified.

http://msdn.microsoft.com/ja-jp/library/36k2cdd4.aspx

The preprocessor searches for include files in the following order:

 1. In the same directory as the file that contains the #include statement.
 2. In the directories of any previously opened include files in the reverse order in which they were opened. The search starts from the directory of the include file that was opened last and continues through the directory of the include file that was opened first.
 3. Along the path specified by each /I compiler option.
 4. Along the paths specified by the INCLUDE environment variable.

The 2 seems to cause this issue.

BTW, knu seemed to apply Usaku's patch at r28341 to ruby_1_9_2 branch,
so this issue would not occur in ruby_1_9_2. I set this ticket to
1.9.x.

--
Yusuke Endoh [e-mail]
=end

[knu (Akinori MUSHA)]

=begin
Thanks for the info. Can you or anyone please check if the clause 2 is new in VC++ 2010?
If that's the case, isn't there any compatibility option we can use to restore the old behavior?

I still hope this is a bug which will eventually be fixed.
=end

[cfis (Charlie Savage)]

=begin
Just to follow up on this - I am unfortunately still seeing the same problem in ruby 1.9.2 with VC++ 2010.

  cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/md5.so md5init.obj md5ossl.obj msvcr100-ruby191.lib crypto.lib  unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:md5-i386-mswin32_100.lib -pdb:md5-i386-mswin32_100.pdb -def:md5-i386-mswin32_100.def
Creating library md5-i386-mswin32_100.lib and object md5-i386-mswin32_100.exp

md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Finish
md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Update
md5init.obj : error LNK2001: unresolved external symbol _rb_Digest_MD5_Init
../../../.ext/i386-mswin32_100/digest/md5.so : fatal error LNK1120: 3 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
Stop.
=end

[ko1 (Koichi Sasada)]

Who should handle this ticket?

[naruse (Yui NARUSE)]

• Assignee changed from knu (Akinori MUSHA) to nahi (Hiroshi Nakamura)

[nahi (Hiroshi Nakamura)]

• Target version changed from 2.0.0 to 1.9.3

[nahi (Hiroshi Nakamura)]

• Status changed from Assigned to Feedback

Charlie, I heard that it doesn't happen for Usaku at present. Do you still suffer by this problem? Can you narrow down conditions how it happens?

[cfis (Charlie Savage)]

This can be closed, it no longer happens.

[nahi (Hiroshi Nakamura)]

• Status changed from Feedback to Rejected

Thanks for the confirmation. Closing this ticket as 'Works for me'.

[naruse (Yui NARUSE)]

• Status changed from Rejected to Closed
• Backport set to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_2 branch at r52739.
Add similar patch to ext/digest/sha2_sha2ossl.c.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE to 2.0.0: REQUIRED, 2.1: DONE, 2.2: DONE

ruby_2_1 r52797 merged revision(s) 52694.