removed slogging references from docs
This commit is contained in:
John Dickinson
parent
5949217ce1
commit
5490c514fe
@ -42,7 +42,6 @@ Overview and Concepts
|
||||
overview_reaper
|
||||
overview_auth
|
||||
overview_replication
|
||||
overview_stats
|
||||
ratelimit
|
||||
overview_large_objects
|
||||
|
||||
|
||||
@ -1,192 +0,0 @@
|
||||
==================
|
||||
Swift stats system
|
||||
==================
|
||||
|
||||
The swift stats system is composed of three parts parts: log creation, log
|
||||
uploading, and log processing. The system handles two types of logs (access
|
||||
and account stats), but it can be extended to handle other types of logs.
|
||||
|
||||
---------
|
||||
Log Types
|
||||
---------
|
||||
|
||||
***********
|
||||
Access logs
|
||||
***********
|
||||
|
||||
Access logs are the proxy server logs. Rackspace uses syslog-ng to redirect
|
||||
the proxy log output to an hourly log file. For example, a proxy request that
|
||||
is made on August 4, 2010 at 12:37 gets logged in a file named 2010080412.
|
||||
This allows easy log rotation and easy per-hour log processing.
|
||||
|
||||
*********************************
|
||||
Account / Container DB stats logs
|
||||
*********************************
|
||||
|
||||
DB stats logs are generated by a stats system process.
|
||||
swift-account-stats-logger runs on each account server (via cron) and walks
|
||||
the filesystem looking for account databases. When an account database is
|
||||
found, the logger selects the account hash, bytes_used, container_count, and
|
||||
object_count. These values are then written out as one line in a csv file. One
|
||||
csv file is produced for every run of swift-account-stats-logger. This means
|
||||
that, system wide, one csv file is produced for every storage node. Rackspace
|
||||
runs the account stats logger every hour. Therefore, in a cluster of ten
|
||||
account servers, ten csv files are produced every hour. Also, every account
|
||||
will have one entry for every replica in the system. On average, there will be
|
||||
three copies of each account in the aggregate of all account stat csv files
|
||||
created in one system-wide run. The swift-container-stats-logger runs in a
|
||||
similar fashion, scanning the container dbs.
|
||||
|
||||
----------------------
|
||||
Log Processing plugins
|
||||
----------------------
|
||||
|
||||
The swift stats system is written to allow a plugin to be defined for every
|
||||
log type. Swift includes plugins for both access logs and storage stats logs.
|
||||
Each plugin is responsible for defining, in a config section, where the logs
|
||||
are stored on disk, where the logs will be stored in swift (account and
|
||||
container), the filename format of the logs on disk, the location of the
|
||||
plugin class definition, and any plugin-specific config values.
|
||||
|
||||
The plugin class definition defines three methods. The constructor must accept
|
||||
one argument (the dict representation of the plugin's config section). The
|
||||
process method must accept an iterator, and the account, container, and object
|
||||
name of the log. The keylist_mapping accepts no parameters.
|
||||
|
||||
-------------
|
||||
Log Uploading
|
||||
-------------
|
||||
|
||||
swift-log-uploader accepts a config file and a plugin name. It finds the log
|
||||
files on disk according to the plugin config section and uploads them to the
|
||||
swift cluster. This means one uploader process will run on each proxy server
|
||||
node and each account server node. To not upload partially-written log files,
|
||||
the uploader will not upload files with an mtime of less than two hours ago.
|
||||
Rackspace runs this process once an hour via cron.
|
||||
|
||||
--------------
|
||||
Log Processing
|
||||
--------------
|
||||
|
||||
swift-log-stats-collector accepts a config file and generates a csv that is
|
||||
uploaded to swift. It loads all plugins defined in the config file, generates
|
||||
a list of all log files in swift that need to be processed, and passes an
|
||||
iterable of the log file data to the appropriate plugin's process method. The
|
||||
process method returns a dictionary of data in the log file keyed on (account,
|
||||
year, month, day, hour). The log-stats-collector process then combines all
|
||||
dictionaries from all calls to a process method into one dictionary. Key
|
||||
collisions within each (account, year, month, day, hour) dictionary are
|
||||
summed. Finally, the summed dictionary is mapped to the final csv values with
|
||||
each plugin's keylist_mapping method.
|
||||
|
||||
The resulting csv file has one line per (account, year, month, day, hour) for
|
||||
all log files processed in that run of swift-log-stats-collector.
|
||||
|
||||
|
||||
================================
|
||||
Running the stats system on SAIO
|
||||
================================
|
||||
|
||||
#. Create a swift account to use for storing stats information, and note the
|
||||
account hash. The hash will be used in config files.
|
||||
|
||||
#. Edit /etc/rsyslog.d/10-swift.conf::
|
||||
|
||||
# Uncomment the following to have a log containing all logs together
|
||||
#local1,local2,local3,local4,local5.* /var/log/swift/all.log
|
||||
|
||||
$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
|
||||
local1.*;local1.!notice ?HourlyProxyLog
|
||||
|
||||
local1.*;local1.!notice /var/log/swift/proxy.log
|
||||
local1.notice /var/log/swift/proxy.error
|
||||
local1.* ~
|
||||
|
||||
#. Edit /etc/rsyslog.conf and make the following change::
|
||||
$PrivDropToGroup adm
|
||||
|
||||
#. `mkdir -p /var/log/swift/hourly`
|
||||
#. `chown -R syslog.adm /var/log/swift`
|
||||
#. `chmod 775 /var/log/swift /var/log/swift/hourly`
|
||||
#. `service rsyslog restart`
|
||||
#. `usermod -a -G adm <your-user-name>`
|
||||
#. Relogin to let the group change take effect.
|
||||
#. Create `/etc/swift/log-processor.conf`::
|
||||
|
||||
[log-processor]
|
||||
swift_account = <your-stats-account-hash>
|
||||
user = <your-user-name>
|
||||
|
||||
[log-processor-access]
|
||||
swift_account = <your-stats-account-hash>
|
||||
container_name = log_data
|
||||
log_dir = /var/log/swift/hourly/
|
||||
source_filename_pattern = ^
|
||||
(?P<year>[0-9]{4})
|
||||
(?P<month>[0-1][0-9])
|
||||
(?P<day>[0-3][0-9])
|
||||
(?P<hour>[0-2][0-9])
|
||||
.*$
|
||||
class_path = swift.stats.access_processor.AccessLogProcessor
|
||||
user = <your-user-name>
|
||||
|
||||
[log-processor-stats]
|
||||
swift_account = <your-stats-account-hash>
|
||||
container_name = account_stats
|
||||
log_dir = /var/log/swift/stats/
|
||||
class_path = swift.stats.stats_processor.StatsLogProcessor
|
||||
devices = /srv/1/node
|
||||
mount_check = false
|
||||
user = <your-user-name>
|
||||
|
||||
[log-processor-container-stats]
|
||||
swift_account = <your-stats-account-hash>
|
||||
container_name = container_stats
|
||||
log_dir = /var/log/swift/stats/
|
||||
class_path = swift.stats.stats_processor.StatsLogProcessor
|
||||
processable = false
|
||||
devices = /srv/1/node
|
||||
mount_check = false
|
||||
user = <your-user-name>
|
||||
|
||||
#. Add the following under [app:proxy-server] in `/etc/swift/proxy-server.conf`::
|
||||
|
||||
log_facility = LOG_LOCAL1
|
||||
|
||||
#. Create a `cron` job to run once per hour to create the stats logs. In
|
||||
`/etc/cron.d/swift-stats-log-creator`::
|
||||
|
||||
0 * * * * <your-user-name> /usr/local/bin/swift-account-stats-logger /etc/swift/log-processor.conf
|
||||
|
||||
#. Create a `cron` job to run once per hour to create the container stats logs. In
|
||||
`/etc/cron.d/swift-container-stats-log-creator`::
|
||||
|
||||
5 * * * * <your-user-name> /usr/local/bin/swift-container-stats-logger /etc/swift/log-processor.conf
|
||||
|
||||
#. Create a `cron` job to run once per hour to upload the stats logs. In
|
||||
`/etc/cron.d/swift-stats-log-uploader`::
|
||||
|
||||
10 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf stats
|
||||
|
||||
#. Create a `cron` job to run once per hour to upload the stats logs. In
|
||||
`/etc/cron.d/swift-stats-log-uploader`::
|
||||
|
||||
15 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf container-stats
|
||||
|
||||
#. Create a `cron` job to run once per hour to upload the access logs. In
|
||||
`/etc/cron.d/swift-access-log-uploader`::
|
||||
|
||||
5 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf access
|
||||
|
||||
#. Create a `cron` job to run once per hour to process the logs. In
|
||||
`/etc/cron.d/swift-stats-processor`::
|
||||
|
||||
30 * * * * <your-user-name> /usr/local/bin/swift-log-stats-collector /etc/swift/log-processor.conf
|
||||
|
||||
After running for a few hours, you should start to see .csv files in the
|
||||
log_processing_data container in the swift stats account that was created
|
||||
earlier. This file will have one entry per account per hour for each account
|
||||
with activity in that hour. One .csv file should be produced per hour. Note
|
||||
that the stats will be delayed by at least two hours by default. This can be
|
||||
changed with the new_log_cutoff variable in the config file. See
|
||||
`log-processor.conf-sample` for more details.
|
||||
@ -44,17 +44,6 @@ use = egg:swift#account
|
||||
# The replicator also performs reclamation
|
||||
# reclaim_age = 86400
|
||||
|
||||
[account-stats]
|
||||
# You can override the default log routing for this app here (don't use set!):
|
||||
# log_name = account-stats
|
||||
# log_facility = LOG_LOCAL0
|
||||
# log_level = INFO
|
||||
# cf_account = AUTH_7abbc116-8a07-4b63-819d-02715d3e0f31
|
||||
# container_name = account_stats
|
||||
# proxy_server_conf = /etc/swift/proxy-server.conf
|
||||
# log_facility = LOG_LOCAL0
|
||||
# log_level = INFO
|
||||
|
||||
[account-auditor]
|
||||
# You can override the default log routing for this app here (don't use set!):
|
||||
# log_name = account-auditor
|
||||
|
||||
Loading…
Reference in New Issue
Block a user