starlingx/integ
Code Issues Proposed changes

Relocate logmgmt to stx-integ/logging/logmgmt

Browse Source 
Move content from stx-utils into stx-integ or stx-update

Packages will be relocated to

stx-update:
    enable-dev-patch
    extras

stx-integ:
    config-files/
        io-scheduler

    filesystem/
        filesystem-scripts

    grub/
        grubby

    logging/
        logmgmt

    tools/
        collector
        monitor-tools

    tools/engtools/
        hostdata-collectors
        parsers

    utilities/
        build-info
        branding   (formerly wrs-branding)
        platform-util

Change-Id: Ie4d2efc756e6294ba456bc54feb823fc8c0be9db
Story: 2002801
Task: 22687
Signed-off-by: Scott Little <scott.little@windriver.com>
This commit is contained in:
Scott Little 2018-08-01 12:18:47 -04:00
parent bab9bb6b69
commit 696e01e065
17 changed files with 1039 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
centos_pkg_dirs
View File

@ -107,3 +107,4 @@ monitoring/influxdb-extensions
kubernetes/kubernetes
kubernetes/docker-distribution
kubernetes/helm
logging/logmgmt

6
logging/logmgmt/.gitignore vendored Normal file
View File

@ -0,0 +1,6 @@
!.distro
.distro/centos7/rpmbuild/RPMS
.distro/centos7/rpmbuild/SRPMS
.distro/centos7/rpmbuild/BUILD
.distro/centos7/rpmbuild/BUILDROOT
.distro/centos7/rpmbuild/SOURCES/logmgmt*tar.gz

202
logging/logmgmt/LICENSE Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

13
logging/logmgmt/PKG-INFO Normal file
View File

@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: logmgmt
Version: 1.0
Summary: Management of /var/log filesystem
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: Management of /var/log filesystem
Platform: UNKNOWN

4
logging/logmgmt/centos/build_srpm.data Normal file
View File

@ -0,0 +1,4 @@
SRC_DIR="logmgmt"
COPY_LIST_TO_TAR="scripts"
COPY_LIST="$SRC_DIR/LICENSE"
TIS_PATCH_VER=4

73
logging/logmgmt/centos/logmgmt.spec Normal file
View File

@ -0,0 +1,73 @@
Summary: Management of /var/log filesystem
Name: logmgmt
Version: 1.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
Source1: LICENSE
BuildRequires: python-setuptools
BuildRequires: systemd-devel
Requires: systemd
Requires: python-daemon
%description
Management of /var/log filesystem
%define local_bindir /usr/bin/
%define local_etc_initd /etc/init.d/
%define local_etc_pmond /etc/pmon.d/
%define pythonroot /usr/lib64/python2.7/site-packages
%define debug_package %{nil}
%prep
%setup
# Remove bundled egg-info
rm -rf *.egg-info
%build
%{__python} setup.py build
%install
%{__python} setup.py install --root=$RPM_BUILD_ROOT \
--install-lib=%{pythonroot} \
--prefix=/usr \
--install-data=/usr/share \
--single-version-externally-managed
install -d -m 755 %{buildroot}%{local_bindir}
install -p -D -m 700 scripts/bin/logmgmt %{buildroot}%{local_bindir}/logmgmt
install -p -D -m 700 scripts/bin/logmgmt_postrotate %{buildroot}%{local_bindir}/logmgmt_postrotate
install -p -D -m 700 scripts/bin/logmgmt_prerotate %{buildroot}%{local_bindir}/logmgmt_prerotate
install -d -m 755 %{buildroot}%{local_etc_initd}
install -p -D -m 700 scripts/init.d/logmgmt %{buildroot}%{local_etc_initd}/logmgmt
install -d -m 755 %{buildroot}%{local_etc_pmond}
install -p -D -m 644 scripts/pmon.d/logmgmt %{buildroot}%{local_etc_pmond}/logmgmt
install -p -D -m 664 scripts/etc/systemd/system/logmgmt.service %{buildroot}%{_unitdir}/logmgmt.service
%post
/usr/bin/systemctl enable logmgmt.service >/dev/null 2>&1
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%doc LICENSE
%{local_bindir}/*
%{local_etc_initd}/*
%dir %{local_etc_pmond}
%{local_etc_pmond}/*
%{_unitdir}/logmgmt.service
%dir %{pythonroot}/%{name}
%{pythonroot}/%{name}/*
%dir %{pythonroot}/%{name}-%{version}.0-py2.7.egg-info
%{pythonroot}/%{name}-%{version}.0-py2.7.egg-info/*

202
logging/logmgmt/logmgmt/LICENSE Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

6
logging/logmgmt/logmgmt/logmgmt/__init__.py Normal file
View File

@ -0,0 +1,6 @@
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""

263
logging/logmgmt/logmgmt/logmgmt/logmgmt.py Normal file
View File

@ -0,0 +1,263 @@
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
###################
# IMPORTS
###################
import logging
import logging.handlers
import time
import os
import subprocess
import glob
import re
import sys
from daemon import runner
import prepostrotate
###################
# CONSTANTS
###################
LOG_DIR = '/var/lib/logmgmt'
LOG_FILE = LOG_DIR + '/logmgmt.log'
PID_FILE = '/var/run/logmgmt.pid'
LOG_FILE_MAX_BYTES = 1024*1024
LOG_FILE_BACKUP_COUNT = 5
PERCENT_FREE_CRITICAL = 10
PERCENT_FREE_MAJOR = 20
LOGROTATE_PERIOD = 600 # Every ten minutes
###################
# METHODS
###################
def start_polling():
logmgmt_daemon = LogMgmtDaemon()
logmgmt_runner = runner.DaemonRunner(logmgmt_daemon)
logmgmt_runner.daemon_context.umask = 0o022
logmgmt_runner.do_action()
def handle_exception(exc_type, exc_value, exc_traceback):
"""
Exception handler to log any uncaught exceptions
"""
logging.error("Uncaught exception",
exc_info=(exc_type, exc_value, exc_traceback))
sys.__excepthook__(exc_type, exc_value, exc_traceback)
###################
# CLASSES
###################
class LogMgmtDaemon():
""" Daemon process representation of
the /var/log monitoring program
"""
def __init__(self):
# Daemon-specific init
self.stdin_path = '/dev/null'
self.stdout_path = '/dev/null'
self.stderr_path = '/dev/null'
self.pidfile_path = PID_FILE
self.pidfile_timeout = 5
self.monitored_files = []
self.unmonitored_files = []
self.last_logrotate = 0
self.last_check = 0
def configure_logging(self, level=logging.DEBUG):
my_exec = os.path.basename(sys.argv[0])
if not os.path.exists(LOG_DIR):
os.mkdir(LOG_DIR, 0755)
log_format = '%(asctime)s: ' \
+ my_exec + '[%(process)s]: ' \
+ '%(filename)s(%(lineno)s): ' \
+ '%(levelname)s: %(message)s'
fmt = logging.Formatter(fmt=log_format)
# Use python's log rotation, rather than logrotate
handler = logging.handlers.RotatingFileHandler(
LOG_FILE,
maxBytes=LOG_FILE_MAX_BYTES,
backupCount=LOG_FILE_BACKUP_COUNT)
my_logger = logging.getLogger()
my_logger.setLevel(level)
handler.setFormatter(fmt)
handler.setLevel(level)
my_logger.addHandler(handler)
# Log uncaught exceptions to file
sys.excepthook = handle_exception
def run(self):
self.configure_logging()
while True:
self.check_var_log()
# run/poll every 1 min
time.sleep(60)
def get_percent_free(self):
usage = os.statvfs('/var/log')
return ((usage.f_bavail * 100) / usage.f_blocks)
def get_monitored_files(self):
self.monitored_files = []
try:
output = subprocess.check_output(['/usr/sbin/logrotate', '-d', '/etc/logrotate.conf'],
stderr=subprocess.STDOUT)
for line in output.split('\n'):
fields = line.split()
if len(fields) > 0 and fields[0] == "considering":
self.monitored_files.extend(glob.glob(fields[2]))
self.monitored_files.extend(glob.glob(fields[2] + '.[0-9].gz'))
self.monitored_files.extend(glob.glob(fields[2] + '.[0-9][0-9].gz'))
except:
logging.error('Failed to determine monitored files')
def get_unmonitored_files(self):
self.unmonitored_files = []
try:
output = subprocess.check_output(['find', '/var/log', '-type', 'f'])
for fname in output.split('\n'):
if fname in self.monitored_files:
continue
# Ignore some files
if '/var/log/puppet' in fname \
or '/var/log/dmesg' in fname \
or '/var/log/rabbitmq' in fname \
or '/var/log/lastlog' in fname:
continue
if os.path.exists(fname):
self.unmonitored_files.append(fname)
except:
logging.error('Failed to determine unmonitored files')
def purge_files(self, index):
pattern = re.compile('.*\.([0-9]*)\.gz')
for fname in sorted(self.monitored_files):
result = pattern.match(fname)
if result:
if int(result.group(1)) >= index:
logging.info("Purging file: %s" % fname)
try:
os.remove(fname)
except OSError as e:
logging.error('Failed to remove file: %s', e)
def run_logrotate(self):
self.last_logrotate = int(time.time())
try:
subprocess.check_call(['/usr/sbin/logrotate', '/etc/logrotate.conf'])
except:
logging.error('Failed logrotate')
def run_logrotate_forced(self):
self.last_logrotate = int(time.time())
try:
subprocess.check_call(['/usr/sbin/logrotate', '-f', '/etc/logrotate.conf'])
except:
logging.error('Failed logrotate -f')
def timecheck(self):
# If we're more than a couple of mins since the last timecheck,
# there could have been a large time correction, which would skew
# our timing. Reset the logrotate timestamp to ensure we don't miss anything
now = int(time.time())
if self.last_check > now or (now - self.last_check) > 120:
self.last_logrotate = 0
self.last_check = now
def check_var_log(self):
self.timecheck()
try:
prepostrotate.ensure_bash_log_locked_down()
except Exception as e:
logging.exception('Failed to ensure bash.log locked', e)
pf = self.get_percent_free()
if pf > PERCENT_FREE_CRITICAL:
# We've got more than 10% free space, so just run logrotate every ten minutes
now = int(time.time())
if self.last_logrotate > now or (now - self.last_logrotate) > LOGROTATE_PERIOD:
logging.info("Running logrotate")
self.run_logrotate()
return
logging.warning("Reached critical disk usage for /var/log: %d%% free" % pf)
# We're running out of disk space, so we need to start deleting files
for index in range(20, 11, -1):
logging.info("/var/log is %d%% free. Purging rotated .%d.gz files to free space" % (pf, index))
self.get_monitored_files()
self.purge_files(index)
pf = self.get_percent_free()
if pf >= PERCENT_FREE_MAJOR:
# We've freed up enough space. Do a logrotate and leave
logging.info("/var/log is %d%% free. Running logrotate" % pf)
self.run_logrotate()
return
# We still haven't freed up enough space, so try a logrotate
logging.info("/var/log is %d%% free. Running logrotate" % pf)
self.run_logrotate()
pf = self.get_percent_free()
if pf >= PERCENT_FREE_MAJOR:
return
# Try a forced rotate
logging.info("/var/log is %d%% free. Running forced logrotate" % pf)
self.run_logrotate_forced()
pf = self.get_percent_free()
if pf >= PERCENT_FREE_MAJOR:
return
# Start deleting unmonitored files
self.get_monitored_files()
self.get_unmonitored_files()
logging.info("/var/log is %d%% free. Deleting unmonitored files to free space" % pf)
for fname in sorted(self.unmonitored_files, key=os.path.getsize, reverse=True):
logging.info("Deleting unmonitored file: %s" % fname)
try:
os.remove(fname)
except OSError as e:
logging.error('Failed to remove file: %s', e)
pf = self.get_percent_free()
if pf >= PERCENT_FREE_MAJOR:
logging.info("/var/log is %d%% free." % pf)
return
# Nothing else to be done
logging.info("/var/log is %d%% free." % pf)
return

60
logging/logmgmt/logmgmt/logmgmt/prepostrotate.py Normal file
View File

@ -0,0 +1,60 @@
"""
Copyright (c) 2017 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
###################
# IMPORTS
###################
import array
import fcntl
import struct
import glob
EXT2_APPEND_FL = 0x00000020
EXT4_EXTENTS_FL = 0x00080000
EXT_IOC_SETFLAGS = 0x40086602
EXT_IOC_GETFLAGS = 0x80086601
def _is_file_append_only(filename):
buf = array.array('h', [0])
with open(filename, 'r') as f:
fcntl.ioctl(f.fileno(), EXT_IOC_GETFLAGS, buf)
has_append_only = (buf.tolist()[0] & EXT2_APPEND_FL) == EXT2_APPEND_FL
return has_append_only
def _set_file_attrs(filename, attrs):
flags = struct.pack('i', attrs)
with open(filename, 'r') as f:
fcntl.ioctl(f.fileno(), EXT_IOC_SETFLAGS, flags)
def chattr_add_append_only(filename):
_set_file_attrs(filename, EXT2_APPEND_FL | EXT4_EXTENTS_FL)
def chattr_remove_append_only(filename):
_set_file_attrs(filename, EXT4_EXTENTS_FL)
def prerotate():
for filename in glob.glob("/var/log/bash.log*"):
if _is_file_append_only(filename):
chattr_remove_append_only(filename)
def postrotate():
for filename in glob.glob("/var/log/bash.log*"):
if not _is_file_append_only(filename):
chattr_add_append_only(filename)
def ensure_bash_log_locked_down():
# need the same functionality as postrotate
postrotate()

19
logging/logmgmt/logmgmt/setup.py Normal file
View File

@ -0,0 +1,19 @@
#!/usr/bin/env python
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import setuptools
setuptools.setup(name='logmgmt',
version='1.0.0',
description='logmgmt',
license='Apache-2.0',
packages=['logmgmt'],
entry_points={
}
)

18
logging/logmgmt/scripts/bin/logmgmt Normal file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env python
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import sys
try:
from logmgmt import logmgmt
except EnvironmentError as e:
print >> sys.stderr, "Error importing logmgmt: ", str(e)
sys.exit(1)
logmgmt.start_polling()

19
logging/logmgmt/scripts/bin/logmgmt_postrotate Normal file
View File

@ -0,0 +1,19 @@
#!/usr/bin/env python
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import sys
try:
from logmgmt import prepostrotate
except EnvironmentError as e:
print >> sys.stderr, "Error importing prepostrotate: ", str(e)
sys.exit(1)
prepostrotate.postrotate()

19
logging/logmgmt/scripts/bin/logmgmt_prerotate Normal file
View File

@ -0,0 +1,19 @@
#!/usr/bin/env python
"""
Copyright (c) 2014 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import sys
try:
from logmgmt import prepostrotate
except EnvironmentError as e:
print >> sys.stderr, "Error importing prepostrotate: ", str(e)
sys.exit(1)
prepostrotate.prerotate()

14
logging/logmgmt/scripts/etc/systemd/system/logmgmt.service Normal file
View File

@ -0,0 +1,14 @@
[Unit]
Description=Titanium Cloud Log Management
After=network.target syslog-ng.service iscsid.service sw-patch.service
Before=config.service pmon.service
[Service]
Type=forking
ExecStart=/etc/init.d/logmgmt start
ExecStop=/etc/init.d/logmgmt stop
ExecReload=/etc/init.d/logmgmt restart
PIDFile=/var/run/logmgmt.pid
[Install]
WantedBy=multi-user.target

96
logging/logmgmt/scripts/init.d/logmgmt Normal file
View File

@ -0,0 +1,96 @@
#!/bin/sh
#
# Copyright (c) 2014 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
### BEGIN INIT INFO
# Provides: logmgmt
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Daemon for /var/log management
# Description: Daemon for /var/log management
### END INIT INFO
DESC="logmgmt"
DAEMON="/usr/bin/logmgmt"
RUNDIR="/var/run"
PIDFILE=$RUNDIR/$DESC.pid
start()
{
if [ -e $PIDFILE ]; then
PIDDIR=/prod/$(cat $PIDFILE)
if [ -d ${PIDFILE} ]; then
echo "$DESC already running."
exit 0
else
echo "Removing stale PID file $PIDFILE"
rm -f $PIDFILE
fi
fi
echo -n "Starting $DESC..."
mkdir -p $RUNDIR
start-stop-daemon --start --quiet \
--pidfile ${PIDFILE} --exec ${DAEMON} start
#--make-pidfile
if [ $? -eq 0 ]; then
echo "done."
else
echo "failed."
exit 1
fi
}
stop()
{
echo -n "Stopping $DESC..."
start-stop-daemon --stop --quiet --pidfile $PIDFILE
if [ $? -eq 0 ]; then
echo "done."
else
echo "failed."
fi
rm -f $PIDFILE
}
status()
{
pid=`cat $PIDFILE 2>/dev/null`
if [ -n "$pid" ]; then
if ps -p $pid &>/dev/null ; then
echo "$DESC is running"
exit 0
else
echo "$DESC is not running but has pid file"
exit 1
fi
fi
echo "$DESC is not running"
exit 3
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart|force-reload|reload)
stop
start
;;
status)
status
;;
*)
echo "Usage: $0 {start|stop|force-reload|restart|reload|status}"
exit 1
;;
esac
exit 0

24
logging/logmgmt/scripts/pmon.d/logmgmt Normal file
View File

@ -0,0 +1,24 @@
;
; Copyright (c) 2014-2016 Wind River Systems, Inc.
;
; SPDX-License-Identifier: Apache-2.0
;
[process]
process = logmgmt
pidfile = /var/run/logmgmt.pid
script = /etc/init.d/logmgmt
style = lsb ; ocf or lsb
severity = minor ; Process failure severity
; critical : host is failed
; major : host is degraded
; minor : log is generated
restarts = 5 ; Number of back to back unsuccessful restarts before severity assertion
interval = 10 ; Number of seconds to wait between back-to-back unsuccessful restarts
debounce = 20 ; Number of seconds the process needs to run before declaring
; it as running O.K. after a restart.
; Time after which back-to-back restart count is cleared.
startuptime = 10 ; Seconds to wait after process start before starting the debounce monitor
mode = passive ; Monitoring mode: passive (default) or active
; passive: process death monitoring (default: always)
; active: heartbeat monitoring, i.e. request / response messaging