FileBackend in Production: C++ Log Rotation with logme

Writing log messages to a file is easy. Keeping file logging predictable after months of production use is much harder.

An active log file must not grow forever. Old logs need to be preserved for a useful period, but they also need to be deleted eventually. Finished files may need compression. At the same time, application threads should not wait for a disk write after every log message.

That is why C++ log rotation in production is more than renaming app.log to app.old.log. A complete file lifecycle is needed.

This is what FileBackend in logme provides. It can write asynchronously, rotate files by size or time, create archives, apply retention rules, and compress completed logs with gzip.

Importantly, archive discovery, retention cleanup, and compression are not performed for every log record. The normal write path stays relatively short.

What happens after LogmeI()

By default, FileBackend works asynchronously.

When a channel sends a completed record to the backend, the data does not necessarily go to the file immediately. Instead, it is placed into an internal queue. The shared file manager later writes accumulated buffers in batches.

This matters in production because the application thread usually does not wait for physical disk I/O after every message.

When the queue was previously empty, the first pending record schedules a flush. In the current implementation, the default delay is 500 ms.

If enough data accumulates before that delay expires, the backend does not keep waiting. It requests immediate processing.

As a result, small bursts can be batched, while heavy logging does not unnecessarily wait for the timer.

An explicit Flush() behaves differently. For an asynchronous FileBackend, it publishes pending data, requests immediate processing, and waits until the queued byte count reaches zero.

Therefore, Flush() is useful as a synchronization point. However, calling it after every record would remove much of the benefit of asynchronous file logging.

A practical FileBackend configuration

A production configuration can look like this:

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "async": true,
  "append": true,

  "rotation": "daily",

  "max-size": "100Mb",
  "on-size-limit": "rotate",

  "archive": "logs/archive/app.{date}.{index}.log",
  "compression": "gz",

  "retention": {
    "max-files": 30,
    "max-age": "30d",
    "max-total-size": "2Gb",
    "clean-on-start": true
  }
}

Two independent conditions can finish the active file here. A new day can start, or the file can reach its size limit.

The completed file receives an archive name. Retention rules are then applied, and gzip compression can process the archive afterwards.

Each part of this configuration solves a different production problem.

The active file and append mode

The file option defines the current active log:

"file": "logs/app.log"

If the path is relative, logme resolves it relative to the logger home directory.

If the parent directory does not exist, FileBackend attempts to create it. Therefore, using logs/app.log does not require the directory to be created manually in advance.

The normal setting is:

"append": true

When an existing file is opened, logging continues at the end of that file. Its current size is taken from the real end position.

This is usually what a production service needs. Restarting the process should not erase the previous part of the current log.

There is also a useful detail when time-based rotation is enabled. In that mode, logme forces append behavior even if append was configured differently.

That prevents an existing file for the current time period from being overwritten after a restart.

What happens at max-size

FileBackend has a built-in active-file size limit:

"max-size": "100Mb"

Without an explicit value, the current default is 8 MiB.

However, max-size alone does not define classic log rotation. The on-size-limit option controls what happens when the limit is reached.

The default is:

"on-size-limit": "truncate"

This is the historical logme behavior. The same active file remains in use, while older content is removed.

The implementation keeps the newer part of the log and writes a marker describing how many characters were discarded.

This mode is useful for a bounded local diagnostic file where long-term history is not important.

For server-side production logs, keeping history is often more useful. In that case, use:

"on-size-limit": "rotate"

Now the current file is completed and moved to the archive location.

Size rotation requires an archive template

When this is configured:

"on-size-limit": "rotate"

an archive option is required. For size-based rotation, the archive template must contain {index}.

For example:

"archive": "logs/archive/app.{index}.log"

The reason is straightforward. One busy process may produce several files within the same day or even within the same minute.

They need unique names:

app.1.log
app.2.log
app.3.log

logme also does not blindly restart numbering from one after each process restart.

FileArchivePolicy checks existing archives and restores the current index. An existing archive is not overwritten. The same applies when the matching file already exists with a .gz suffix.

This is important in production because restarting a service must not silently replace yesterday’s or today’s previous archive files.

Time-based log rotation

FileBackend can also rotate according to calendar time.

Supported modes include:

hourly
daily
weekly
monthly

Rotation can be disabled with values such as none, off, or disabled.

For example:

"rotation": "daily"

Daily rotation follows local calendar time. Hourly rotation uses the beginning of the hour. Weekly rotation treats Monday as the start of the week, while monthly rotation starts on the first day of a month.

There is one practical detail worth understanding.

FileBackend does not start a dedicated timer just to rename a file at exactly midnight. The time boundary is checked when the next log record arrives.

So a quiet service may not rotate at 00:00:00. If the next message arrives at 03:17, the previous period is completed then.

The archive name still corresponds to the logical period that has just ended. It does not simply use the arbitrary time of the first record in the new period.

For logging, this is usually the behavior you want.

Size and time rotation can work together

In production, there is often no reason to choose only one rotation rule.

For example:

{
  "rotation": "daily",
  "max-size": "100Mb",
  "on-size-limit": "rotate",
  "archive": "logs/archive/app.{date}.{index}.log"
}

The current log now ends in either of two situations.

A new day starts, or the file reaches 100 MB.

Both cases go through the same completion path. Therefore, archive naming, retention, and compression behave consistently regardless of why rotation happened.

The template:

app.{date}.{index}.log

works especially well here.

On a quiet day, you may get only:

app.2026-08-18.1.log

Under heavier load, the same day may produce:

app.2026-08-18.1.log
app.2026-08-18.2.log
app.2026-08-18.3.log

The date identifies the period, while the index identifies parts within that period.

{date}, {datetime}, and {index}

Archive names can use lifecycle placeholders.

{date} produces a value such as:

2026-08-18

{datetime} also includes the time:

2026-08-18-14-35-00

And {index} provides the archive sequence number.

For example:

"archive": "archive/server.{date}.{index}.log"

or:

"archive": "archive/server.{datetime}.{index}.log"

When a new time period begins, the index is restored for the matching archive naming pattern.

Archives from the previous day do not force the new day to continue their numbering.

This may sound like a small detail, but it makes production log directories much easier to understand.

Rotation without retention only moves the problem

A very common logging mistake is to configure rotation and stop there.

One huge app.log then turns into thousands of smaller archive files. Eventually, the disk still fills up.

In logme, this part of the lifecycle is handled by retention.

For example:

"retention": {
  "max-files": 30,
  "max-age": "30d",
  "max-total-size": "2Gb",
  "clean-on-start": true
}

These are independent limits.

max-files limits the number of matching files.

max-age removes archives that have become too old.

max-total-size removes the oldest files until the combined archive size falls below the configured limit.

The rules are applied together. In practice, the strictest combination wins.

For example, max-files: 30 does not mean “keep 30 days.” If the application creates ten archives per day, thirty files may contain only three days of history.

Therefore, max-age is useful as a separate policy. It expresses how old a log is still worth keeping.

max-total-size protects against unexpected log volume

File count alone does not always describe disk usage well.

A service may normally write 50 MB per day. Then DEBUG logging gets enabled, or a failure loop starts producing huge amounts of diagnostics.

This is where:

"max-total-size": "2Gb"

becomes useful.

If matching archive files exceed the configured total, cleanup starts with the oldest files.

This makes max-total-size a useful second line of defense. Even when the number of files is reasonable, an unusually noisy period should not consume unlimited disk space.

A zero value disables the corresponding limit. This applies to max-files, max-age, and max-total-size.

The older max-parts setting is still supported as a legacy equivalent of retention.max-files.

If both values are present, they must agree. Otherwise, the configuration is treated as invalid.

clean-on-start handles stale archives

The default is:

"clean-on-start": true

When the configuration is applied, FileBackend runs retention cleanup.

This is useful in real deployments. A service may have been stopped for weeks, while old archives remain in the directory.

At startup, the backend can immediately bring the archive set back under the configured retention policy.

After that, retention also runs when another file is completed.

Importantly, it does not scan the archive directory for every ordinary log record.

That keeps retention work away from the hot logging path.

Retention does not delete arbitrary files

Cleanup is based on the file pattern associated with a particular FileBackend.

Unrelated files in the same directory should not be removed merely because they happen to be nearby.

The active log file is also passed to the cleaner as protected. Even if its name happens to match the cleanup pattern, retention should not remove the file that is currently being written.

Therefore, the active file and archives can live in the same directory.

Still, I usually prefer a layout like this:

logs/app.log
logs/archive/app.2026-08-18.1.log
logs/archive/app.2026-08-18.2.log

It is easier for people to inspect, and archive management remains clearly separated from the current file.

Gzip is applied only to completed files

Compression is enabled with:

"compression": "gz"

or:

"compression": "gzip"

The active app.log is not compressed.

After rotation, the archive file is submitted to CompressionManager. Compression has its own queue and worker, so gzip does not run inside the normal logging call.

This is important in production. Compressing a large file can take noticeable CPU time.

There is no reason to make the thread calling LogmeI() wait for that work.

After successful compression, the archive receives a .gz suffix.

logme also considers compressed files when restoring archive indexes. Therefore, an existing:

app.2026-08-18.4.log.gz

prevents a new rotation from accidentally reusing the same fourth archive name.

There is one requirement: logme must be built with zlib support.

If USE_ZLIB is disabled, compression: "gz" can still appear in configuration, but actual gzip compression does not take place.

What if rotation fails?

Production code has to expect filesystem errors.

The archive directory may become unavailable. The disk may fill up. A rename can fail.

When completing the current file, FileBackend first attempts to prepare the archive directory. It then renames the active file to the selected archive name.

If creating the archive directory fails, the backend reports an internal error and tries to reopen the current log in append mode.

A rename failure is handled similarly. logme attempts to return to the existing active file and continue writing.

So a failed rotation does not automatically mean the existing app.log must be discarded.

However, this is still best-effort recovery.

If the filesystem no longer allows the file to be opened or written, no logging library can make the write succeed.

Filesystem logging failures should therefore still be monitored in a production environment.

Async logging does not mean the bytes are already on disk

This distinction is easy to miss.

When LogmeI() successfully submits a message to an asynchronous FileBackend, the bytes may still be waiting in the queue.

They do not have to be in the file at that exact moment.

During normal shutdown, the backend drains pending data. An explicit Flush() also lets the application wait until the queue is empty.

An abnormal process termination is different.

For example, SIGKILL gives the process no chance to execute normal shutdown code.

Therefore, a normal asynchronous file backend should not be treated as a crash-safe record of the final machine instruction.

logme has a separate crash logging API for crash-sensitive paths.

For normal application logs, asynchronous file writing is still a sensible production default.

Truncate or rotate?

The two modes solve different problems.

truncate is useful for a local diagnostic file with a strict size limit. Long-term history is secondary, and you do not want multiple files.

rotate is appropriate when previous events have diagnostic value. It enables archives, retention, and compression.

For production servers, I would normally choose rotate.

For example:

"max-size": "100Mb",
"on-size-limit": "rotate"

Combined with daily rotation, this handles two different extremes.

A quiet service gets convenient daily log files.

A very noisy service still cannot produce one enormous file during a single day.

FileBackend retention and DirectorySizeWatchdog solve different problems

logme also has another disk-control mechanism: DirectorySizeWatchdog.

It works at a broader level.

FileBackend retention manages the lifecycle of files belonging to one backend and its archive pattern.

The watchdog monitors the total size of a logging directory.

For example, five different backends may each respect their own retention policy. Together, however, they can still use too much disk space.

In that case, a directory watchdog can provide an additional global limit.

The two mechanisms do not compete with each other. In a production system, using both can be entirely reasonable.

A practical production starting point

For a typical long-running service, I would start with something close to this:

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "async": true,
  "append": true,

  "rotation": "daily",

  "max-size": "100Mb",
  "on-size-limit": "rotate",

  "archive": "logs/archive/app.{date}.{index}.log",
  "compression": "gz",

  "retention": {
    "max-files": 50,
    "max-age": "30d",
    "max-total-size": "2Gb",
    "clean-on-start": true
  }
}

The exact numbers depend on the application.

However, the overall structure is broadly useful: one clear active file, a size limit, time-based rotation, a separate archive directory, gzip compression, and several independent retention limits.

At that point, FileBackend is doing more than simply writing text into a file. It is managing the full lifecycle of production logs.

That is what C++ log rotation usually needs in practice: disk usage stays predictable, while useful diagnostic history remains available when you need it.