Bug #3231

Digest Does Not Build

Added by Charlie Savage almost 4 years ago. Updated over 2 years ago.

[ruby-core:29911]
Status:Rejected
Priority:Normal
Assignee:Hiroshi Nakamura
Category:ext
Target version:1.9.3
ruby -v:HEAD Backport:

Description

=begin
Revisiting this one from a few weeks back. MD5/rmd160/sha1/sha2 do not build using VC 2010. They do build with Mingw (mingw + msys). See compiler logs below.

Focusing on MD5 (they all have the same issue):

extconf.h is generated like this:

#ifndef EXTCONFH
#define EXTCONF
H
#define HAVECONFIGH 1
#define HAVEOPENSSLMD5_H 1
#endif

Then in md5init.c:

#include "digest.h"
#if defined(HAVEOPENSSLMD5_H)
#include "md5ossl.h"
#else
#include "md5.h"
#endif

Since extconf.h is never passed to the compiler, the error happens. Adding this to the top of the file:

#define HAVEOPENSSLMD5_H

Fixes the issue. Clearly that's a hack, but I don't understand how extconf.h is used in building the extension since the only place it comes into play is in the generated makefile:

$(OBJS): $(RUBYEXTCONFH)

So not sure the solution, but the problem is that having openssl installed correctly breaks the VC digest extension build.


  1. Digest/MD5

    cl -nologo -LD -Fe../../../.ext/i386-mswin32100/digest/md5.so md5init.obj md5ossl.obj msvcr100-ruby191.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws232.lib -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:md5-i386-mswin32100.lib -pdb:md5-i386-mswin32100.pdb -def:md5-i386-mswin32100.def
    Creating library md5-i386-mswin32
    100.lib and object md5-i386-mswin32100.exp
    md5init.obj : error LNK2001: unresolved external symbol _rb
    DigestMD5Finish
    md5init.obj : error LNK2001: unresolved external symbol rbDigestMD5Update
    md5init.obj : error LNK2001: unresolved external symbol rbDigestMD5Init
    ../../../.ext/i386-mswin32_100/digest/md5.so : fatal error LNK1120: 3 unresolved externals
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
    Stop.

  2. Digest/rmd160

     cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/rmd160.so rmd160init.obj rmd160ossl.obj msvcr100-ruby191.lib crypto.lib  unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:rmd160-i386-mswin32_100.lib -pdb:rmd160-i386-mswin32_100.pdb -def:rmd160-i386-mswin32_100.def
    

    Creating library rmd160-i386-mswin32100.lib and object rmd160-i386-mswin32100.exp
    rmd160init.obj : error LNK2001: unresolved external symbol rbDigestRMD160Finish
    rmd160init.obj : error LNK2001: unresolved external symbol rbDigestRMD160Update
    rmd160init.obj : error LNK2001: unresolved external symbol rbDigestRMD160Init
    ../../../.ext/i386-mswin32_100/digest/rmd160.so : fatal error LNK1120: 3 unresolved externals
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'

  3. Digest/sha1

    cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/sha1.so sha1init.obj sha1ossl.obj msvcr100-ruby191.lib crypto.lib  unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:sha1-i386-mswin32_100.lib -pdb:sha1-i386-mswin32_100.pdb -def:sha1-i386-mswin32_100.def
    

    Creating library sha1-i386-mswin32100.lib and object sha1-i386-mswin32100.exp
    sha1init.obj : error LNK2001: unresolved external symbol rbDigestSHA1Finish
    sha1init.obj : error LNK2001: unresolved external symbol rbDigestSHA1Update
    sha1init.obj : error LNK2001: unresolved external symbol rbDigestSHA1Init
    ../../../.ext/i386-mswin32_100/digest/sha1.so : fatal error LNK1120: 3 unresolved externals
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
    Stop.

  4. Digest/sha2
    cl -nologo -LD -Fe../../../.ext/i386-mswin32100/digest/sha2.so sha2init.obj sha2ossl.obj msvcr100-ruby191.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws232.lib -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:sha2-i386-mswin32100.lib -pdb:sha2-i386-mswin32100.pdb -def:sha2-i386-mswin32100.def
    Creating library sha2-i386-mswin32
    100.lib and object sha2-i386-mswin32100.exp
    sha2init.obj : error LNK2001: unresolved external symbol _rb
    DigestSHA512Finish
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA512Update
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA512Init
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA384Finish
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA384Update
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA384Init
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA256Finish
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA256Update
    sha2init.obj : error LNK2001: unresolved external symbol rbDigestSHA256Init
    ../../../.ext/i386-mswin32_100/digest/sha2.so : fatal error LNK1120: 9 unresolved externals
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
    Stop.

    c:\Development\src\ruby\ext\digest\sha2>
    =end

Makefile (7.07 KB) Charlie Savage, 05/07/2010 04:31 AM

mkmf.log (1.9 KB) Charlie Savage, 05/09/2010 08:12 AM

Makefile (7.07 KB) Charlie Savage, 05/09/2010 08:12 AM

dotdotslash.patch Magnifier - a proposed workaround for the ruby_1_9_2 branch (4.53 KB) Akinori MUSHA, 06/14/2010 09:00 PM

History

#1 Updated by Nobuyoshi Nakada almost 4 years ago

=begin
Hi,

At Sat, 1 May 2010 15:12:35 +0900,
Charlie Savage wrote in :

Since extconf.h is never passed to the compiler, the error
happens. Adding this to the top of the file:

CPPFLAGS should have -DRUBYEXTCONFH=\"$(RUBYEXTCONFH)\",
and include/ruby/ruby.h has

#ifdef RUBYEXTCONFH
#include RUBYEXTCONFH
#endif

How is CPPFLAGS defined in Makefile?

--
Nobu Nakada

=end

#2 Updated by Nobuyoshi Nakada almost 4 years ago

  • Status changed from Open to Feedback

=begin

=end

#3 Updated by Charlie Savage almost 4 years ago

=begin
Ok, I see what is happening. The include is pulling in digest/extconf.h, not digest/md5/extconf.h. Thus the compile breaks.

The makefile has this:

CPPFLAGS = -DRUBYEXTCONFH=\"$(RUBYEXTCONFH)\"

Include/ruby/ruby.h (in the source tree) has this:

#ifdef RUBYEXTCONFH
#include RUBYEXTCONFH
#endif

But if you do this:

cd /ext/digest/md5
nmake

Then for whatever reason the wrong extconf.h file is being used.

Attaching the generated makefile if it helps.

=end

#4 Updated by Nobuyoshi Nakada almost 4 years ago

  • Category set to ext

=begin
Could you show an mkmf.log?
And from where have you installed OpenSSL?
=end

#5 Updated by Charlie Savage almost 4 years ago

=begin
Sure, attached.

OpenSSL (v1.0) is installed to c:\Development\msys\local\bin and c:\Development\msys\local\include.

This is where I have other 3rd party libraries installed (iconv, zlib, libxml, etc.) and those extensions build fine as does the openssl extension. So this isn't a path issue.
=end

#6 Updated by Nobuyoshi Nakada almost 4 years ago

=begin
Hi,

At Sun, 9 May 2010 08:12:27 +0900,
Charlie Savage wrote in :

OpenSSL (v1.0) is installed to c:\Development\msys\local\bin
and c:\Development\msys\local\include.

How were those directory names given to the compiler? The
mkmf.log doesn't seem contain them.

havelibrary: checking for main() in crypto.lib... -------------------- yes
"cl -nologo -Feconftest -I../../../.ext/include/i386-mswin32
100 -I../../.././include -I../../.././ext/digest/md5 -I../../.././ext/digest/md5/.. -I../../.././include -MD -Zi -W2 -wd4996 -Od -Zm600 conftest.c msvcr100-ruby191-static.lib crypto.lib unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib -link -incremental:no -debug -opt:ref -opt:icf -libpath:. -libpath:../../.. "

haveheader: checking for openssl/md5.h... -------------------- yes
"cl -nologo -I../../../.ext/include/i386-mswin32
100 -I../../.././include -I../../.././ext/digest/md5 -I../../.././ext/digest/md5/.. -MD -Zi -W2 -wd4996 -Od -Zm600 -c conftest.c"

And from where have you downloaded and installed that OpenSSL?

--
Nobu Nakada

=end

#7 Updated by Charlie Savage almost 4 years ago

=begin
1. Open VC command prompt.
2. set INCLUDE=c:\Development\msys\local\include;%INCLUDE%
3. set LIB=c:\Development\msys\local\lib;%LIB%
4. cd c:\Development\src\ruby
5. nmake

(Note I have previously run win32/configure.bat)

Note the openssl includes and libs are being found just fine as you can see in mkmf.log
=end

#8 Updated by Charlie Savage almost 4 years ago

=begin
Missed the openssl question. Its the official release, from http://www.openssl.org/source. Builds fine, and works fine with the openssl extension as previously noted. And works fine with digest too with including extconf.h from the digest directory (versus the parent directory).
=end

#9 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Does this patch help you?
(only ext/digest/md5)

Index: ext/digest/md5/extconf.rb
===================================================================
--- ext/digest/md5/extconf.rb (revision 28034)
+++ ext/digest/md5/extconf.rb (working copy)
@@ -4,7 +4,6 @@
require "mkmf"

$defs << "-DHAVECONFIGH"
-$INCFLAGS << " -I$(srcdir)/.."

$objs = [ "md5init.#{$OBJEXT}" ]

Index: ext/digest/md5/md5init.c
===================================================================
--- ext/digest/md5/md5init.c (revision 28034)
+++ ext/digest/md5/md5init.c (working copy)
@@ -1,7 +1,7 @@
/* $RoughId: md5init.c,v 1.2 2001/07/13 19:49:10 knu Exp $ /
/
$Id$ */

-#include "digest.h"
+#include "../digest.h"
#if defined(HAVEOPENSSLMD5H)
#include "md5ossl.h"
#else
Index: ext/digest/md5/md5.h
===================================================================
--- ext/digest/md5/md5.h (revision 28034)
+++ ext/digest/md5/md5.h (working copy)
@@ -46,7 +46,7 @@
#ifndef MD5
INCLUDED
# define MD5_INCLUDED

-#include "defs.h"
+#include "../defs.h"

/*
* This code has some adaptations for the Ghostscript environment, but it

=end

#10 Updated by Charlie Savage almost 4 years ago

=begin
Yes - it does fix the issue for md5. Can this also be done for the other digest subdirectories?

Thanks for the help.

=end

#11 Updated by Yusuke Endoh almost 4 years ago

  • Status changed from Feedback to Open
  • Assignee set to Usaku NAKAMURA
  • Target version set to 1.9.2

=begin
Hi, Usaku

Could you please commit your patch?

--
Yusuke Endoh mame@tsg.ne.jp
=end

#12 Updated by Usaku NAKAMURA almost 4 years ago

  • Status changed from Open to Assigned
  • Assignee changed from Usaku NAKAMURA to Akinori MUSHA

=begin
This patch shows only concept to fix.
The maintainer should think how to do now. It's not my task.
=end

#13 Updated by Akinori MUSHA almost 4 years ago

=begin
I'll handle this.

This fix is okay for me because defs.h is intended to be for internal use inside ext/digest/.
=end

#14 Updated by Akinori MUSHA almost 4 years ago

=begin
I think I misread the patch.
Digest submodules should be able to include digest.h without having to use a relative path form, while that's not the case for defs.h.

Let me think awhile about what the real problem is here.
I won't object to putting the proposed fix into ruby19_2 for the forthcoming release, but for trunk as a permanent fix I'm not quite convinced.
=end

#15 Updated by Akinori MUSHA almost 4 years ago

=begin

=end

#16 Updated by Usaku NAKAMURA almost 4 years ago

=begin
In building process, both ext/digest/extconf.h and ext/digest/md5/extconf.h exist.
If you specify -I.. in ext/digest/md5 and #include , compiler cannot
decide to include whether ext/digest/extconf.h or ext/digest/md5/extconf.h.

Gcc gives priority to the place specified by the rear -I, but of course this
behavior is not the spec of any standards. It's merely implementation dependence.
And, VC10 actually shows different behavior.
This is the real problem.

My conclusion is that you should specify to complier what do you actually want
to include.
=end

#17 Updated by Akinori MUSHA almost 4 years ago

=begin
Thanks so much for the detailed explanation. Can I ask you something? Does VC++ search directories in random order, or does it work under any specific rules that changed in VC++ 2010?

In Unix, there is a spec in SUS that the order of specifying -I/-L options is significant for the compiler commands, and they shall search directories named in those options in the order specified.
=end

#18 Updated by Yusuke Endoh almost 4 years ago

  • Target version changed from 1.9.2 to 2.0.0

=begin
Hi,

2010/6/15 Akinori MUSHA redmine@ruby-lang.org:

Thanks so much for the detailed explanation. ?Can I ask you something? ?Does VC++ search directories in random order, or does it work under any specific rules that changed in VC++ 2010?

In Unix, there is a spec in SUS that the order of specifying -I/-L options is significant for the compiler commands, and they shall search directories named in those options in the order specified.

http://msdn.microsoft.com/ja-jp/library/36k2cdd4.aspx

The preprocessor searches for include files in the following order:

 1. In the same directory as the file that contains the #include statement.
 2. In the directories of any previously opened include files in the reverse order in which they were opened. The search starts from the directory of the include file that was opened last and continues through the directory of the include file that was opened first.
 3. Along the path specified by each /I compiler option.
 4. Along the paths specified by the INCLUDE environment variable.

The 2 seems to cause this issue.

BTW, knu seemed to apply Usaku's patch at r28341 to ruby192 branch,
so this issue would not occur in ruby
192. I set this ticket to
1.9.x.

--
Yusuke Endoh mame@tsg.ne.jp
=end

#19 Updated by Akinori MUSHA almost 4 years ago

=begin
Thanks for the info. Can you or anyone please check if the clause 2 is new in VC++ 2010?
If that's the case, isn't there any compatibility option we can use to restore the old behavior?

I still hope this is a bug which will eventually be fixed.
=end

#20 Updated by Charlie Savage over 3 years ago

=begin
Just to follow up on this - I am unfortunately still seeing the same problem in ruby 1.9.2 with VC++ 2010.

  cl -nologo -LD -Fe../../../.ext/i386-mswin32_100/digest/md5.so md5init.obj md5ossl.obj msvcr100-ruby191.lib crypto.lib  unicows.lib oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -incremental:no -debug -opt:ref -opt:icf -dll -libpath:. -libpath:../../.. -implib:md5-i386-mswin32_100.lib -pdb:md5-i386-mswin32_100.pdb -def:md5-i386-mswin32_100.def
Creating library md5-i386-mswin32_100.lib and object md5-i386-mswin32_100.exp

md5init.obj : error LNK2001: unresolved external symbol rbDigestMD5Finish
md5init.obj : error LNK2001: unresolved external symbol rbDigestMD5Update
md5init.obj : error LNK2001: unresolved external symbol rbDigestMD5Init
../../../.ext/i386-mswin32_100/digest/md5.so : fatal error LNK1120: 3 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.EXE"' : return code '0x2'
Stop.
=end

#21 Updated by Koichi Sasada almost 3 years ago

Who should handle this ticket?

#22 Updated by Yui NARUSE almost 3 years ago

  • Assignee changed from Akinori MUSHA to Hiroshi Nakamura

#23 Updated by Hiroshi Nakamura almost 3 years ago

  • Target version changed from 2.0.0 to 1.9.3

#24 Updated by Hiroshi Nakamura almost 3 years ago

  • Status changed from Assigned to Feedback

Charlie, I heard that it doesn't happen for Usaku at present. Do you still suffer by this problem? Can you narrow down conditions how it happens?

#25 Updated by Charlie Savage over 2 years ago

This can be closed, it no longer happens.

#26 Updated by Hiroshi Nakamura over 2 years ago

  • Status changed from Feedback to Rejected

Thanks for the confirmation. Closing this ticket as 'Works for me'.

Also available in: Atom PDF