Part 5: Log analysis with beet-utils
The beet-utils command line utilities are primarily for the default compressed binary log format. Thus far in the tutorial we've been working with plain text logging, so first we'll use the JMX console to switch back to the default behavior.
For this tutorial we assume you have the command line utility zcat installed on your system. For Windows users this means Cygwin or a similar *nix-like utilities package. If you don't have zcat, you can still try the tutorial, but skip the part in Step 2 that tells you to enable Compression.
Step 1: Verify beet-utils setup
Open your favorite command interpreter (e.g. Cygwin or cmd on Windows, xterm on *nix) and change into your beet installation directory. Test the installation by running "java -jar beet-utils.jar". You should see a terse usage message:
Leave the terminal window open.
Step 2: Use jconsole to change logging format
Connect to Tomcat with jconsole and examine the beet-hello/com.mtgi.analytics/defaultTrackingManager/BeetLog mbean as described in Part 4 of the tutorial. Select the Attributes node under BeetLog and change both Binary and Compress attributes to true:
Since we're changing the log format, these changes only take effect after the next log rotation. By default this happens every night at midnight. If you are like me and doing this tutorial late at night, you can wait. Otherwise, select the Operations node under BeetLog:
Click the rotateLog button; you should see a window indicating the name of the newly archived log file.
Use a filesystem browser to examine the [TOMCAT]\logs directory. You should see both the newly archived log file (named like beet-hello-perf.xml.TIMESTAMP), and the new current log file, beet-hello-perf.bxml.gz. The file extension has been changed to .bxml.gz to reflect the new format.
Step 3: Click around beet-hello
Once again point your browser at http://localhost:8080/beet-hello. Click around on create, delete, update, clear buttons. Let the app know that you are there.
Step 4: Use jconsole to flush any in-memory events
Back in jconsole, select the Operations node under beet-hello/com.mtgi.analytics/defaultTrackingManager/BeetManager. Click the flush button to force any accumulated events to be written to the log.
The result window indicates the number of events flushed to the log. Because early in the tutorial we configured our flush interval very low (30 seconds), this will likely be 0, unless your window-changing and mouse-clicking skills are unusually well-honed.
Again use the BeetLog.rotateLog operation to rotate our log. This should create a new log named beet-hello-perf.bxml.gz.[TIMESTAMP] to use in the following steps.
Step 5: Convert the current log to other formats
Back in your command prompt window, you can convert the log to various formats using the beet-utils jar. Try the following commands:
> zcat [TOMCAT]/logs/beet-hello-perf.bxml.gz.[TIMESTAMP] | java -jar beet-utils.jar -tool xml
> zcat [TOMCAT]/logs/beet-hello-perf.bxml.gz.[TIMESTAMP] | java -jar beet-utils.jar -tool csv
> zcat [TOMCAT]/logs/beet-hello-perf.bxml.gz.[TIMESTAMP] | java -jar beet-utils.jar -tool xslt -format text -split event -xsl sql/etl/insert-events.xsl
Where of course you should replace [TOMCAT] with your Tomcat install directory, [TIMESTAMP] with the actual file extension, and use '\' instead of '/' if you are a Windows user. The last example, which applies a custom XSLT stylesheet to the event data, is a powerful way to customize event log processing. The provided XSL produces the same output as the '-tool csv' variant, but you can obviously customize its behavior.