Bug #2966

CSV reports illegal quoting on line ...

Added by Seth Goings almost 2 years ago. Updated 10 months ago.

[ruby-core:28668]
Status:Rejected Start date:03/16/2010
Priority:Normal Due date:
Assignee:James Gray % Done:

0%
Category:lib
Target version:-
ruby -v:ruby 1.9.1p376 (2009-12-07 revision 26041) [i386-mswin32]

Description

I have a line of code:
@array = CSV.read("#{@dir}/#{@file}")

Which eventually reads line 24 (below) in file attached to this bug:
BBBP,004,/etc/release,"LSB_VERSION="core-2.0-noarch:core-3.0-noarch:core-2.0-x86_64:core-3.0-x86_64""

This raises a MalformedCSVError, "Illegal quoting on line #{lineno + 1}."

I understand the issue of the nested quotes and how the line is being parsed, but I don't see an issue with how this CSV line is written. Should this sort of line be parse-able with a csv library? If not, can/should the library fail nicely, instead of ruining the whole "read" call?

performance_log.csv - CSV file that fails parsing (260.8 kB) Seth Goings, 03/16/2010 01:22 am

History

#1
Updated by James Gray almost 2 years ago

  • Category set to lib
  • Status changed from Open to Rejected
> Which eventually reads line 24

Unfortunately, that line is not valid CSV data.  In CSV, quotes must be escaped (by doubling them) inside of a quoted field.  Given that, the field would need to be:  …,"LSB_VERSION=""core-2.0…

> If not, can/should the library fail nicely, instead of ruining the whole "read" call?

Without being able to correctly parse that line, it's impossible to correctly parse any data that comes after it.  You asked the library to read so data and it has become impossible to do so, so I feel the correct behavior is to raise an error.  If you will be feeding CSV invalid data, you will need to rescue these errors and take appropriate actions.

#2
Updated by Seth Goings almost 2 years ago

Is there a reason why this is able to be read in FasterCSV (prior to 1.9) then?

#3
Updated by James Gray almost 2 years ago

> Is there a reason why this is able to be read in FasterCSV (prior to 1.9) then?

You have located a bug in the new non-regular expression parser used by FasterCSV.  It's not parsable by older versions (that don't have the bug):

$ ruby -rubygems -ve 'gem("fastercsv", "=1.4.0"); require "faster_csv"; p FCSV.read(ARGV.shift)[0..24]' performance_log.csv.xls 
ruby 1.8.6 (2010-02-05 patchlevel 399) [i686-darwin10.2.0]
/usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1650:in `shift': Illegal quoting on line 24. (FasterCSV::MalformedCSVError)
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1568:in `loop'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1568:in `shift'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1513:in `each'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1524:in `to_a'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1524:in `read'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1241:in `read'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1191:in `open'
	from /usr/local/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:1241:in `read'
	from -e:1

#4
Updated by Seth Goings almost 2 years ago

Ok thanks for your help!

#5
Updated by James Gray almost 2 years ago

The bug you have found is fixed in the new FasterCSV 1.5.3 gem.

Also available in: Atom PDF