Project

General

Profile

Bug #16236

Cannot handle logrotate on Windows when one file is written from multi process/thread

Added by cosmo0920 (Hiroshi Hatake) about 2 months ago. Updated 25 days ago.

Status:
Third Party's Issue
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.4.6p354 (2019-04-01 revision 67394) [x64-mingw32]
[ruby-core:95216]

Description

TL;DR

This issue occurred in Windows version of Fluentd.
It uses ServerEngine and its logger functionality delegates into Ruby logger library.
Fluentd's logrotate functionality is originated from Ruby logger library,
Then, I found that this library does not seems to support logrotate under multi process/thread condition on Windows.

Issue Detail

When a log file which is written from the one process or thread, it works correctly:

require "logger"

logger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end

But logger writing file collision circumstances, it cannot handle logrotate correctly:

require "logger"

logger = Logger.new('test.log', 5, 30*1024)
myOtherLogger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end
...
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
.... (after that many of these lines)

Fluentd launches spawn model (which has supervisor and several workers) on Windows.
Currently, Fluentd implementation uses one logfile from supervisor and worker(s).
So, it causes log shifting operation collision.

Expected behavior

Logrorating operation works correctly under multi process/threading circumstances on Windows like as Linux environment.

History

Updated by jeremyevans0 (Jeremy Evans) about 2 months ago

Log rotation relies on renaming the current log file (app.log -> app.log.0) and the reopening the log file (app.log). This does not work on Windows in a multi-process environment because you cannot rename a file open by another process on Windows. I think it should work in a multi-thread single-process environment on Windows.

I don't think this problem is fixable in logger. You should probably use an alternative logger implementation, maybe one that logs to another single process that can handle log rotation. Alternatively, you can have each worker process use its own log file, and then combine the log files later.

Updated by cosmo0920 (Hiroshi Hatake) 25 days ago

Thanks for the information.
This issue should be Fluentd logging mechanism issue not Ruby core logger library.
Fluentd side logrotate issue is fixed.
So, it can be closed.

#4

Updated by duerst (Martin Dürst) 25 days ago

  • Status changed from Open to Third Party's Issue

Also available in: Atom PDF