Clarify source type for drain error messages in app log stream

[jorbaum]

Context

PR #633 adds an AppLogStream that emits error messages to app developers when their syslog drain is misconfigured (#579). The current implementation emits two log lines per error — one with source type LGR and one with SYS:

logclientOption := loggregator.WithAppInfo(appID, "LGR", "")
appLogStream.logClient.EmitLog(message, logclientOption)

logclientOption = loggregator.WithAppInfo(appID, "SYS", appLogStream.sourceIndex)
appLogStream.logClient.EmitLog(message, logclientOption)

This was copied from syslog_connector.go's emitLoggregatorErrorLog, which used it specifically for dropped message notifications. The new AppLogStream is used more broadly (validation errors, connectivity issues, etc.).

Problem

• Duplication: Every error produces two identical log lines in the app's log stream, which is confusing for app developers.
• Historical precedent: The old filtered_binding_fetcher emitted only with LGR. The dual pattern appears specific to the syslog connector's dropped-message alerts, not a general convention.

Proposal

Emit a single log line per error using LGR with sourceIndex for traceability:

logclientOption := loggregator.WithAppInfo(appID, "LGR", appLogStream.sourceIndex)
appLogStream.logClient.EmitLog(message, logclientOption)

Open Questions

1. Are there existing automations, dashboards, or alerting rules that depend on filtering by SYS or LGR source type for these messages?
2. If not, should syslog drain errors use LGR, SYS, or both?
3. Should the existing dual-emit in syslog_connector.go also be aligned for consistency?

References

• PR: Syslog drain app error messages in app log stream #633 | Issue: Improve application error logs about Syslog drain problems #579
• Discussion thread

[ameowlia]

@jorbaum , @geofffranks , and I discussed this at the WG forum today. We all agreed that the proposal looks good and we should de-duplicate these log lines.

Thank you @jorbaum for your work in this area!

[jorbaum]

I took another look at this and noticed that source index is set to "syslog_agent" (see syslog_agent.go), which results in the loggregator envelope having instance_id set to "syslog_agent". This seems incorrect and we should take another look.

According to loggregator-api documentation this should "refer to the instance number of the application".

Maybe setting to "0" or no value at all might be a better solution?