fix data lost on decompression

[komeda-shinji]

Which issue(s) this PR fixes:
Fixes #2546

in string_decompress(), gz.read loop break when unused.nil?,
but unused.nil? does not show end of stream.
It cause data lost.

loop shoud be break when eof.

[repeatedly]

Thanks for the patch!

Could you follow DCO?
And do you have a failure case to check the behaviour in test?

[komeda-shinji]

I have failure case data, but I cannot provide because it contains privacy data.
I try to create failure case data, but It is too hard and unsuccessful yet.

[komeda-shinji]

This script generates failure case data for string_decompress easily.

require 'stringio'
require 'zlib'
require 'securerandom'

def compress(data, **kwargs)
  output_io = kwargs[:output_io]
  io = output_io || StringIO.new
  Zlib::GzipWriter.wrap(io) do |gz|
    gz.write data
  end

  output_io || io.string
end

def string_decompress(compressed_data)
  io = StringIO.new(compressed_data)

  out = ''
  loop do
    gz = Zlib::GzipReader.new(io)
    out << gz.read
    unused = gz.unused
    gz.finish

    break if unused.nil?
    adjust = unused.length
    io.pos -= adjust
  end

  out
end

def create_data
  data = []
  10.times do
    data << SecureRandom.base64(1990) + "\n"
  end
  data
end
    
def find_failure
  data = []
  found = false
  loop do
    data = create_data

    indata = ''
    for d in data do
      indata << compress(d)
    end
    
    io = StringIO.new(indata)
    out = ''
    loop do
      gz = Zlib::GzipReader.new(io)
      out << gz.read
      unused = gz.unused
      gz.finish
    
      puts unused: unused.nil? ? "nil" : unused.length, eof: io.eof? ? "EOF" : "not"
      found = true if unused.nil? and not io.eof?
      unless unused.nil?
        adjust = unused.length
        io.pos -= adjust
      end
      break if io.eof
    end
    break if found
  end
  data
end

failure_case = find_failure

indata = ''
for d in failure_case do
  indata << compress(d)
end
outdata = string_decompress(indata)

puts failure_case
puts "---"
puts outdata

[repeatedly]

Use io instead of input here.

[komeda-shinji]

Oh! It's my msitake, thank you.

[repeatedly]

Thanks!
I will consider adding test with your script.