Project

General

Profile

Actions

Bug #16236

closed

Cannot handle logrotate on Windows when one file is written from multi process/thread

Added by cosmo0920 (Hiroshi Hatake) over 4 years ago. Updated over 4 years ago.

Status:
Third Party's Issue
Assignee:
-
Target version:
-
ruby -v:
ruby 2.4.6p354 (2019-04-01 revision 67394) [x64-mingw32]
[ruby-core:95216]

Description

TL;DR

This issue occurred in Windows version of Fluentd.
It uses ServerEngine and its logger functionality delegates into Ruby logger library.
Fluentd's logrotate functionality is originated from Ruby logger library,
Then, I found that this library does not seems to support logrotate under multi process/thread condition on Windows.

Issue Detail

When a log file which is written from the one process or thread, it works correctly:

require "logger"

logger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end

But logger writing file collision circumstances, it cannot handle logrotate correctly:

require "logger"

logger = Logger.new('test.log', 5, 30*1024)
myOtherLogger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end
...
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
.... (after that many of these lines)

Fluentd launches spawn model (which has supervisor and several workers) on Windows.
Currently, Fluentd implementation uses one logfile from supervisor and worker(s).
So, it causes log shifting operation collision.

Expected behavior

Logrorating operation works correctly under multi process/threading circumstances on Windows like as Linux environment.

Updated by jeremyevans0 (Jeremy Evans) over 4 years ago

Log rotation relies on renaming the current log file (app.log -> app.log.0) and the reopening the log file (app.log). This does not work on Windows in a multi-process environment because you cannot rename a file open by another process on Windows. I think it should work in a multi-thread single-process environment on Windows.

I don't think this problem is fixable in logger. You should probably use an alternative logger implementation, maybe one that logs to another single process that can handle log rotation. Alternatively, you can have each worker process use its own log file, and then combine the log files later.

Updated by cosmo0920 (Hiroshi Hatake) over 4 years ago

Thanks for the information.
This issue should be Fluentd logging mechanism issue not Ruby core logger library.
Fluentd side logrotate issue is fixed.
So, it can be closed.

Actions #4

Updated by duerst (Martin Dürst) over 4 years ago

  • Status changed from Open to Third Party's Issue
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0