[ext/openssl] Encoding of subclasses fails when it shouldn't
While skimming through ossl_asn1.c I noticed that my patch for infinite length encoding
causes problems when encoding subclasses of OpenSSL::ASN1::Sequence or OpenSSL::ASN1::Set
with infinite length.
E.g. the following fails in trunk:
sub = Class.new(OpenSSL::ASN1::Sequence)
instance = sub.new([OpenSSL::ASN1::EndOfContent.new])
instance.infinite_length = true
to_der': invalid constructed encoding (OpenSSL::ASN1::ASN1Error)'
This can be fixed with the appended code that checks for subclass relationship instead
of comparing the class directly with Set or Sequence.
#2 Updated by MartinBosslet (Martin Bosslet) over 6 years ago
The first thought that came into my mind for changing this was to provide consistency. There are several
similar operations that also use rb_is_kind_of instead of checking the class directly, mainly in
ossl_asn1_default_tag, which is used throughout the entire encoding process.
But there may also be good use cases when subclassing would make sense. For example if one would like
to have some kind of default value set before encoding the value.
A probable scenario where this would make sense could e.g. be an ASN.1 structure that comes with a SET of
certificates. Depending on the application environment, this SET might be predetermined, and to simplify
things one might want to add the predetermined certificates in case someone else forgot to do so. To
achieve this the developer could subclass OpenSSL::ASN1::Set and overwrite #to_der by setting the default
first (if needed) and then delegate to Set's implementation.
I don't know, I couldn't find a more convincing example right now, but I think the consistency argument is
more convincing anyway :)
#3 Updated by MartinBosslet (Martin Bosslet) over 6 years ago
I found a imo quite reasonable use case for sublassing Sequence or Set.
Imagine you have an instance of a rather large sequence or set permanently
stored in a variable somewhere - and this instance gets encoded to DER very
often (e.g. for building a digest of the DER bytes). To improve performance
of this operation, one could subclass sequence and cache the DER-encoded
form by lazily setting an instance variable of the subclass, so that the
actual encoding has to be done only once.