Collapse Macros in logme: How to Reduce Duplicate Log Errors

Eduard Mishkurov

20 hours ago

Why Duplicate Log Errors Are a Problem

Duplicate log errors are one of the most annoying problems in production diagnostics. Sometimes the real issue is not that there are too few logs. The issue is that there are too many of them.

When the same error is written in a loop, on every reconnect, on every retry, or on every request, the log quickly becomes a stream of almost identical lines.

For example, imagine that an external backend is unavailable, but the service keeps trying to connect to it:

backend connection failed
backend connection failed
backend connection failed
backend connection failed
backend connection failed
backend connection failed

Technically, every message is true. However, this kind of log is almost useless for a person.

It does not help you understand the root cause faster. Instead, it makes the log file larger, pushes nearby events away, and makes investigation harder. In many cases, the important information is not in the thousandth copy of the same error. It is usually in the events before the first failure or right after the system state changed.

This is exactly the kind of problem that collapse macros in logme are designed to solve. Their goal is not to hide errors. Their goal is to remove unnecessary noise from repeated messages.

What Collapse Does

Collapse allows logme to collapse a series of identical messages.

The first message is written immediately. This is important because the problem must be visible as soon as it happens. After that, repeated messages from the same place in the code are temporarily suppressed.

When the configured number of repeats is reached, logme writes a summary message with the number of suppressed repeats.

For example:

LogmeW_Collapse(100, "backend connection failed");

The first message is written immediately. Then identical repeated messages are suppressed. When the repeat counter reaches the configured limit, logme writes a message with the same text and repeat information:

repeated 100 times: backend connection failed

The summary still goes through the normal logme formatting path. This is important. Collapse does not create a special or unusual log format. It simply passes repeat information into the regular log record formatting flow.

As a result, the log remains readable, compact, and consistent with the rest of the system.

Why CollapseEvery Is Needed

Count-based collapse works well when repeats are frequent, but not extreme.

For example, if a reconnect attempt happens once per second, this code gives you roughly one summary per minute:

LogmeW_Collapse(60, "backend connection failed");

However, some situations are much more aggressive.

An error can be written inside a very fast loop. In that case, even a limit of one thousand repeats may not be enough. If the code can produce hundreds of thousands of identical messages per second, count-based collapse may still allow too many summary records into the log.

For this situation, logme provides a time-based version:

LogmeW_CollapseEvery(1000, "backend connection failed");

The meaning is different.

The first message is written immediately. Then identical messages are suppressed during the configured interval. When the interval has passed and the same message appears again, logme writes a record with the accumulated repeat counter.

In simple terms:

LogmeW_Collapse(1000, ...);

means:

Write a summary after one thousand repeats.

But:

LogmeW_CollapseEvery(1000, ...);

means:

Do not write this repeated error more often than once per second, but keep the number of suppressed repeats.

This is not a replacement for Every. The Every macro has a different purpose. It limits how often a message is written. CollapseEvery keeps collapse semantics: the first message is visible immediately, repeated messages are suppressed, and a later summary includes the number of accumulated repeats.

Why This Is Not Just Rate Limiting

At first glance, CollapseEvery may look like ordinary rate limiting. However, there is an important difference.

Rate limiting usually controls how often a message is printed. It does not treat repeated messages as a logical series.

For example:

LogmeW_Every(1000, ...);

is useful when you want to print a message no more than once per second. But it does not tell you how many calls were suppressed.

CollapseEvery is intended for a different case. The message is important, but its repeats create noise. Therefore, logme does not only limit output frequency. It also keeps the repeat counter.

This gives you a much more useful production log:

backend connection failed
repeated 245731 times: backend connection failed

Now you can see both facts:

The error is still happening.
The scale of the problem is very large.

For production diagnostics, this is much more useful than seeing the same line once in a while without knowing how many times it actually happened.

When to Use Collapse and CollapseEvery

Use normal Collapse when the message repeats at a moderate speed and the repeat count is the main thing you care about.

For example:

LogmeW_Collapse(10, "configuration reload failed");

This is useful when the same failure may happen several times, but it is not expected to happen hundreds of thousands of times per second.

Use CollapseEvery when the message can appear very quickly:

LogmeW_CollapseEvery(1000, "backend connection failed");

This version is especially useful for:

retry loops;
polling loops;
reconnect loops;
errors inside fast cycles;
bad system states that can generate a lot of repeated messages.

At the same time, CollapseEvery is not designed to be the cheapest possible logging call. It is not a macro that should be placed everywhere by default.

Macro uses special collapse logic. It compares message keys and stores state for the call site. Its purpose is to optimize log content in specific noisy scenarios, not to replace normal logging.

Why Identical Errors Are Not Always Identical Strings

In real logs, repeated errors often differ in small details.

For example:

request=1001 backend connection failed after 10ms
request=1002 backend connection failed after 12ms
request=1003 backend connection failed after 9ms

For a human, this is the same problem. The backend is unavailable.

However, for a simple string comparison, these are different messages. The request ID and elapsed time make every line unique.

For this case, logme provides CollapseIgnore.

LogmeW_CollapseIgnore(
  "request=[0-9]+|after [0-9]+ms"
  , 100
  , "request=%d backend connection failed after %dms"
  , requestId
  , elapsedMs
);

The regular expression is used only for comparison.

logme formats the original message first. Then it temporarily removes volatile parts from the formatted message. After that, it decides whether the message is a repeat.

The original text is still written to the log. The ignored parts affect only the comparison key.

This makes CollapseIgnore useful when each log line contains fields such as:

request ID;
correlation ID;
elapsed time;
retry number;
temporary counter;
timestamp;
changing connection ID.

These fields may make the string different, even though the underlying error is the same.

CollapseIgnoreEvery

Time-based collapse can also be combined with ignored volatile fields.

For example:

LogmeW_CollapseIgnoreEvery(
  "request=[0-9]+|after [0-9]+ms"
  , 1000
  , "request=%d backend connection failed after %dms"
  , requestId
  , elapsedMs
);

This is useful when the error repeats very quickly, but every line contains a request ID, correlation ID, counter, timestamp, or another changing field.

Without ignore logic, these messages would not be considered identical. With CollapseIgnoreEvery, logme can collapse them into one logical series.

Instead of thousands of lines with different request IDs, you can get one clear summary with the number of repeated events.

This keeps the log compact without losing the fact that the error happened many times.

What Happens When the Message Changes

Collapse works with one call site and one comparison key.

If another message comes from the same place in the code, logme treats it as the start of a new series. The pending summary for the old key is not written.

This behavior is important.

Collapse does not try to create an additional “backdated” record when the message changes. It simply starts a new series.

For example, if message A was repeated several times and then message B appears from the same call site, logme does not insert an extra summary for A before B.

The new message is written normally and becomes the new key for that call site.

This approach is simple, predictable, and easy to reason about.

Multiline Messages

Collapse works with a logical log message, not with a visual line on the screen.

If several lines should be treated as one repeated event, it is better to format them as one log message with \n inside.

For example:

backend request failed
host=api.example.com
operation=update-profile

This can be one logical event. If the same block repeats, it can be collapsed as a single message.

However, if every line is written by a separate log call, collapse will work with each line separately.

Sometimes this is fine. But for repeated blocks, it is usually better to keep related context inside one log message.

That way, logme can collapse the whole event as one unit instead of treating each line as an independent message.

When Collapse Should Not Be Used

Collapse is not suitable for every kind of log.

Do not use collapse macros for:

audit logs;
billing events;
security decisions;
legal or compliance records;
events where every individual occurrence must be preserved.

If every event must be stored as a separate fact, it must not be collapsed.

You should also be careful with CollapseIgnore.

Do not ignore fields that change the meaning of the error. For example, fields such as user, host, operation, error code, or decision result may be critical for investigation.

A simple rule is useful here:

Ignore only the parts that create noise but do not change the cause of the event.

If a field helps explain why the error happened, keep it in the comparison key.

Practical Selection Guide

logme provides several macros for different levels of message control.

Use Once when a message should be printed only one time.

You can use Every when a message should be printed no more often than a configured interval, and the repeat count is not important.

Use Collapse when a message repeats and you want a summary based on the number of repeats.

Call CollapseEvery when a message repeats too quickly and you want to limit output by time while keeping the repeat count.

Use CollapseIgnore or CollapseIgnoreEvery when messages are logically the same, but contain volatile fields that change on every call.

This creates a clear toolset:

Once                    print only once
Every                   limit by time
Collapse                collapse by repeat count
CollapseEvery           collapse by time
CollapseIgnore          collapse while ignoring volatile fields
CollapseIgnoreEvery     collapse by time while ignoring volatile fields

Simple cases are handled by simple macros. More noisy production scenarios can use the collapse family.

Conclusion

Collapse macros in logme make repeated message handling much more powerful.

Collapse aggregates repeats by count.
CollapseEvery aggregates repeats by time.
CollapseIgnore allows logme to treat messages with changing details as the same logical event.
CollapseIgnoreEvery combines both approaches and is especially useful for fast loops with volatile fields.

The main idea is simple: a log should help a person understand what happened.

If one thousand identical lines do not add new information, it is better to collapse them into one clear summary with the number of repeats.

This keeps production logs smaller, cleaner, and much easier to investigate.