Export all journal logs

Currently we only export the devstack@ services, and then separately
export the kernel & sudo logs to syslog.txt.

This leaves a lot of logs potentially behind in the journal for
various daemons.  Just export the whole lot.

Using this output is currently very opaque and makes use of systemd
export tools that are very un-discoverable.  Add a README that will
appear alongside the journal explaining how to actually use it.  This
is a template as it would be nice to put into things like the list of
services that are in the journal, or maybe other magic.

Also make sure we export the logs since the start timestamp; currently
during a full run we drop the initial logs.

Change-Id: Id2626f9113d82c6d524039acda8a8ec74afb2081

roles/export-devstack-journal/README.rst

@@ -1,11 +1,15 @@
 Export journal files from devstack services
 
-Export the systemd journal for every devstack service in native
+This performs a number of logging collection services
-journal format as well as text.  Also, export a syslog-style file with
-kernal and sudo messages.
 
-Writes the output to the ``logs/`` subdirectory of
+* Export the systemd journal in native format
-``stage_dir``.
+* For every devstack service, export logs to text in a file named
+  ``screen-*`` to maintain legacy compatability when devstack services
+  used to run in a screen session and were logged separately.
+* Export a syslog-style file with kernel and sudo messages for legacy
+  compatability.
+
+Writes the output to the ``logs/`` subdirectory of ``stage_dir``.
 
 **Role Variables**
 

roles/export-devstack-journal/tasks/main.yaml

@@ -6,32 +6,49 @@
     state: directory
     owner: "{{ ansible_user }}"
 
-# TODO: convert this to ansible
+- name: Export legacy stack screen log files
-- name: Export journal files
   become: true
   shell:
     cmd: |
       u=""
       name=""
-      for u in `systemctl list-unit-files | grep devstack | awk '{print $1}'`; do
+      for u in $(systemctl list-unit-files | grep devstack | awk '{print $1}'); do
         name=$(echo $u | sed 's/devstack@/screen-/' | sed 's/\.service//')
         journalctl -o short-precise --unit $u | gzip - > {{ stage_dir }}/logs/$name.txt.gz
       done
 
-      # Export the journal in export format to make it downloadable
+- name: Export legacy syslog.txt
-      # for later searching. It can then be rewritten to a journal native
+  become: true
-      # format locally using systemd-journal-remote. This makes a class of
+  shell:
-      # debugging much easier. We don't do the native conversion here as
-      # some distros do not package that tooling.
-      journalctl -u 'devstack@*' -o export | \
-          xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
-
     # The journal contains everything running under systemd, we'll
     # build an old school version of the syslog with just the
     # kernel and sudo messages.
+    cmd: |
       journalctl \
           -t kernel \
           -t sudo \
           --no-pager \
           --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
         | gzip - > {{ stage_dir }}/logs/syslog.txt.gz
+
+# TODO: convert this to ansible
+#  - make a list of the above units
+#  - iterate the list here
+- name: Export journal
+  become: true
+  shell:
+    # Export the journal in export format to make it downloadable
+    # for later searching. It can then be rewritten to a journal native
+    # format locally using systemd-journal-remote. This makes a class of
+    # debugging much easier. We don't do the native conversion here as
+    # some distros do not package that tooling.
+    cmd: |
+      journalctl -o export \
+          --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
+        | xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
+
+- name: Save journal README
+  become: true
+  template:
+    src: devstack.journal.README.txt.j2
+    dest: '{{ stage_dir }}/logs/devstack.journal.README.txt'

roles/export-devstack-journal/templates/devstack.journal.README.txt.j2

@@ -0,0 +1,33 @@
+Devstack systemd journal
+========================
+
+The devstack.journal file is a copy of the systemd journal during the
+devstack run.
+
+To use it, you will need to convert it so journalctl can read it
+locally.  After downloading the file:
+
+ $ /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal
+
+Note this binary is not in the regular path.  On Debian/Ubuntu
+platforms, you will need to have the "sytemd-journal-remote" package
+installed.
+
+It should result in something like:
+
+ Finishing after writing <large number> entries
+
+You can then use journalctl to examine this file.  For example, to see
+all devstack services try:
+
+ $ journalctl --file ./output.journal -u 'devstack@*'
+
+To see just cinder API server logs restrict the match with
+
+ $ journalctl --file ./output.journal -u 'devstack@c-api'
+
+There may be many types of logs available in the journal, a command like
+
+ $ journalctl --file ./output.journal --output=json-pretty | grep "_SYSTEMD_UNIT" | sort -u
+
+can help you find interesting things to filter on.