64b6c9261e
Current folder name New folder name Book title ---------------------------------------------------------- basic-install DELETE cli-guide DELETE common common NEW admin-guide-cloud Cloud Administrators Guide docbkx-example DELETE openstack-block-storage-admin DELETE openstack-compute-admin DELETE openstack-config config-reference OpenStack Configuration Reference openstack-ha high-availability-guide OpenStack High Availabilty Guide openstack-image image-guide OpenStack Virtual Machine Image Guide openstack-install install-guide OpenStack Installation Guide openstack-network-connectivity-admin admin-guide-network OpenStack Networking Administration Guide openstack-object-storage-admin DELETE openstack-security security-guide OpenStack Security Guide openstack-training training-guide OpenStack Training Guide openstack-user user-guide OpenStack End User Guide openstack-user-admin user-guide-admin OpenStack Admin User Guide glossary NEW OpenStack Glossary bug: #1220407 Change-Id: Id5ffc774b966ba7b9a591743a877aa10ab3094c7 author: diane fleming
152 lines
6.6 KiB
XML
152 lines
6.6 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
||
<section
|
||
xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
||
xml:id="analyzing-log-files-with-swift-cli">
|
||
<title>Analyzing Log Files with Swift CLI</title>
|
||
<para>When you want quick, command-line answers to
|
||
questions about logs, you can use swift with the -o or
|
||
-output option. The -o —output option can only be used
|
||
with a single object download to redirect the data
|
||
stream to either a different file name or to STDOUT
|
||
(-). The ability to redirect the output to STDOUT
|
||
allows you to pipe “|” data without saving it to disk
|
||
first. One common use case is being able to do some
|
||
quick log file analysis. First let’s use swift to
|
||
setup some data for the examples. The “logtest”
|
||
directory contains four log files with the following
|
||
line format.</para>
|
||
<para>
|
||
<literallayout>Files:
|
||
2010-11-16-21_access.log
|
||
2010-11-16-22_access.log
|
||
2010-11-15-21_access.log
|
||
2010-11-15-22_access.log
|
||
|
||
Log lines:
|
||
Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
||
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432
|
||
</literallayout>
|
||
</para>
|
||
<para>The swift tool can easily upload the four log files
|
||
into a container named “logtest”:</para>
|
||
<para><screen>
|
||
<prompt>$</prompt> <userinput>cd logs</userinput>
|
||
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput>
|
||
</screen>
|
||
<programlisting>
|
||
2010-11-16-21_access.log
|
||
2010-11-16-22_access.log
|
||
2010-11-15-21_access.log
|
||
2010-11-15-22_access.log
|
||
</programlisting>
|
||
Get statistics on the account: <screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
|
||
<programlisting>
|
||
Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||
Containers: 1
|
||
Objects: 4
|
||
Bytes: 5888268
|
||
</programlisting>
|
||
Get statistics on the container:
|
||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen><programlisting>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||
Container: logtest
|
||
Objects: 4
|
||
Bytes: 5864468
|
||
Read ACL:
|
||
Write ACL:
|
||
</programlisting>
|
||
List all the objects in the container:
|
||
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen><programlisting>2010-11-15-21_access.log
|
||
2010-11-15-22_access.log
|
||
2010-11-16-21_access.log
|
||
2010-11-16-22_access.log</programlisting>
|
||
</para>
|
||
<para>The following examples use the -o —output option
|
||
with (-) to help answer questions about the uploaded
|
||
log files. The swift command will download an object,
|
||
stream it to awk to determine the breakdown of
|
||
requests by return code for everything during 2200 on
|
||
November 16th, 2010. Based on the log line format
|
||
column 9 is the type of request and column 12 is the
|
||
return code. After awk processes the data stream it is
|
||
piped to sort and then uniq -c to sum up the number of
|
||
occurrences for each combination of request type and
|
||
return code.</para>
|
||
<para><screen>
|
||
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest 2010-11-16-22_access.log | awk ‘{ print $9”-“$12}’ | sort | uniq -c</userinput>
|
||
</screen>
|
||
<programlisting>805 DELETE-204
|
||
12 DELETE-404
|
||
2 DELETE-409
|
||
723 GET-200
|
||
142 GET-204
|
||
74 GET-206
|
||
80 GET-304
|
||
34 GET-401
|
||
5 GET-403
|
||
18 GET-404
|
||
166 GET-412
|
||
2 GET-416
|
||
50 HEAD-200
|
||
17 HEAD-204
|
||
20 HEAD-401
|
||
8 HEAD-404
|
||
30 POST-202
|
||
25 POST-204
|
||
22 POST-400
|
||
6 POST-404
|
||
842 PUT-201
|
||
2 PUT-202
|
||
32 PUT-400
|
||
4 PUT-403
|
||
4 PUT-404
|
||
2 PUT-411
|
||
6 PUT-412
|
||
6 PUT-413
|
||
2 PUT-422
|
||
8 PUT-499</programlisting>
|
||
</para>
|
||
<para>This example uses a bash for loop with awk, swift
|
||
with its -o —output option with a hyphen (-) to find
|
||
out how many PUT requests are in each log file. First
|
||
create a list of objects by running swift with the
|
||
list command on the “logtest” container; then for each
|
||
item in the list run swift with download -o - then
|
||
pipe the output into grep to filter the put requests
|
||
and finally into wc -l to count the lines.</para>
|
||
<para>
|
||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
|
||
do echo -ne “PUTS - ” ; swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||
done</userinput></screen>
|
||
<programlisting>
|
||
2010-11-15-21_access.log - PUTS - 402
|
||
2010-11-15-22_access.log - PUTS - 1091
|
||
2010-11-16-21_access.log - PUTS - 892
|
||
2010-11-16-22_access.log - PUTS - 910
|
||
</programlisting>
|
||
</para>
|
||
<para>By adding the -p —prefix option a prefix query is
|
||
performed on the list to return only the object names
|
||
that begin with a specific string. Let’s determine out
|
||
how many PUT requests are in each object with a name
|
||
beginning with “2010-11-15”. First create a list of
|
||
objects by running swift with the list command on the
|
||
“logtest” container with the prefix option -p
|
||
2010-11-15. Then on each of item(s) returned run swift
|
||
with the download -o - then pipe the output to grep
|
||
and wc as in the previous example. The echo command is
|
||
added to display the object name.</para>
|
||
<para>
|
||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
|
||
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||
done</userinput></screen>
|
||
<programlisting>2010-11-15-21_access.log - PUTS - 402
|
||
2010-11-15-22_access.log - PUTS - 910</programlisting>
|
||
</para>
|
||
<para>The swift utility is simple, scalable, flexible and
|
||
provides useful solutions all of which are core
|
||
principles of cloud computing; with the -o output
|
||
option being just one of its many features.</para>
|
||
</section>
|