Application logs

Kpow uses Logback to record application logs to stdout.

Configure Kpow to write application logs to disk (or any other supported Logback appender) and control the log levels by providing a Logback configuration file.

This example turns on Jetty JAAS debug logging:

<configuration>

<statusListener class="ch.qos.logback.core.status.NopStatusListener" />

<appender name="plain" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %logger{36} – %m%n</pattern>
    </encoder>
</appender>

<appender name="json" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
            <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
            <timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
            <appendLineSeparator>true</appendLineSeparator>
            <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
                <prettyPrint>false</prettyPrint>
            </jsonFormatter>
        </layout>
    </encoder>
</appender>

<logger name="kafka" level="ERROR"/>
<logger name="org.apache.kafka" level="ERROR"/>
<logger name="org.apache.zookeeper" level="ERROR"/>
<logger name="io.confluent.kafka.serializers" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.common.configs" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.serializers" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.deserializers" level="ERROR"/>

<logger name="io.kpow.jaas.spi" level="DEBUG"/>
<logger name="org.eclipse.jetty.jaas" level="DEBUG"/>
<logger name="org.eclipse.jetty.security.authentication" level="DEBUG"/>

<if condition='isDefined("LOG_FORMAT")'>
    <then>
        <root level="INFO">
            <appender-ref ref="${LOG_FORMAT}"/>
        </root>
    </then>
    <else>
        <root level="INFO">
            <appender-ref ref="plain"/>
        </root>
    </else>
</if>
</configuration>

Provide your custom Logback configuration to Kpow as a Java system variable:

java -Dlogback.configurationFile=/kpow/custom-logback.xml -jar /kpow/kpow-latest.jar

The Kpow Docker container can be configured with the JAVA_TOOL_OPTIONS environment variable:

JAVA_TOOL_OPTIONS=-Dlogback.configurationFile=/kpow/custom-logback.xml

In this specific example we turn on debug logging for Jetty JAAS authentication methods (LDAP, etc.).

To put JAAS into debug mode also requires debug="true" in your JAAS config file:

kpow {
         org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
         debug="true"
         file="dev-resources/jaas/hash-realm.properties";
     };

Once configured, application logs are written to file and JAAS debug output is visible:

09:26:32.135 INFO  [oprtr.compute.snapshots.v2-44318b10-b719-46db-96ea-f97495cdce8f-StreamThread-2] operatr.compute-v2 – oprtr.compute.snapshots.v2: streams state changed from REBALANCING to RUNNING
09:26:32.287 INFO  [oprtr.compute.metrics.v2-6126ee6c-0253-4a5d-8d66-cbd3b0a4b0bb-StreamThread-2] operatr.compute-v2 – oprtr.compute.metrics.v2: streams state changed from REBALANCING to RUNNING
09:26:39.349 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [9] configuration captured in 204 ms
09:26:39.605 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [9] configuration captured in 256 ms
09:26:41.409 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – setupPropertyUserStore: Starting new PropertyUserStore. PropertiesFile: dev-resources/jaas/hash-realm.properties refrescallouterval: 0
09:26:41.416 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – Checking PropertyUserStore dev-resources/jaas/hash-realm.properties for admin
09:26:41.417 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – Found: admin in PropertyUserStore dev-resources/jaas/hash-realm.properties
09:26:41.429 INFO  [qtp1357054363-58] operatr.auth.jetty – login -> user: admin, roles: #{"content-administrators" "kafka-admins" "server-administrators"}
09:27:05.449 INFO  [OperatrScheduler_Worker-3] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 440 ms
09:28:05.250 INFO  [OperatrScheduler_Worker-7] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 243 ms
09:29:05.350 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 343 ms
09:30:05.257 INFO  [OperatrScheduler_Worker-5] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 250 ms

See the Logback site for full information on configuration options like rolling file appenders.