removed slogging references from docs
This commit is contained in:
John Dickinson
parent
5949217ce1
commit
5490c514fe
@ -42,7 +42,6 @@ Overview and Concepts
|
|||||||
overview_reaper
|
overview_reaper
|
||||||
overview_auth
|
overview_auth
|
||||||
overview_replication
|
overview_replication
|
||||||
overview_stats
|
|
||||||
ratelimit
|
ratelimit
|
||||||
overview_large_objects
|
overview_large_objects
|
||||||
|
|
||||||
|
|||||||
@ -1,192 +0,0 @@
|
|||||||
==================
|
|
||||||
Swift stats system
|
|
||||||
==================
|
|
||||||
|
|
||||||
The swift stats system is composed of three parts parts: log creation, log
|
|
||||||
uploading, and log processing. The system handles two types of logs (access
|
|
||||||
and account stats), but it can be extended to handle other types of logs.
|
|
||||||
|
|
||||||
---------
|
|
||||||
Log Types
|
|
||||||
---------
|
|
||||||
|
|
||||||
***********
|
|
||||||
Access logs
|
|
||||||
***********
|
|
||||||
|
|
||||||
Access logs are the proxy server logs. Rackspace uses syslog-ng to redirect
|
|
||||||
the proxy log output to an hourly log file. For example, a proxy request that
|
|
||||||
is made on August 4, 2010 at 12:37 gets logged in a file named 2010080412.
|
|
||||||
This allows easy log rotation and easy per-hour log processing.
|
|
||||||
|
|
||||||
*********************************
|
|
||||||
Account / Container DB stats logs
|
|
||||||
*********************************
|
|
||||||
|
|
||||||
DB stats logs are generated by a stats system process.
|
|
||||||
swift-account-stats-logger runs on each account server (via cron) and walks
|
|
||||||
the filesystem looking for account databases. When an account database is
|
|
||||||
found, the logger selects the account hash, bytes_used, container_count, and
|
|
||||||
object_count. These values are then written out as one line in a csv file. One
|
|
||||||
csv file is produced for every run of swift-account-stats-logger. This means
|
|
||||||
that, system wide, one csv file is produced for every storage node. Rackspace
|
|
||||||
runs the account stats logger every hour. Therefore, in a cluster of ten
|
|
||||||
account servers, ten csv files are produced every hour. Also, every account
|
|
||||||
will have one entry for every replica in the system. On average, there will be
|
|
||||||
three copies of each account in the aggregate of all account stat csv files
|
|
||||||
created in one system-wide run. The swift-container-stats-logger runs in a
|
|
||||||
similar fashion, scanning the container dbs.
|
|
||||||
|
|
||||||
----------------------
|
|
||||||
Log Processing plugins
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
The swift stats system is written to allow a plugin to be defined for every
|
|
||||||
log type. Swift includes plugins for both access logs and storage stats logs.
|
|
||||||
Each plugin is responsible for defining, in a config section, where the logs
|
|
||||||
are stored on disk, where the logs will be stored in swift (account and
|
|
||||||
container), the filename format of the logs on disk, the location of the
|
|
||||||
plugin class definition, and any plugin-specific config values.
|
|
||||||
|
|
||||||
The plugin class definition defines three methods. The constructor must accept
|
|
||||||
one argument (the dict representation of the plugin's config section). The
|
|
||||||
process method must accept an iterator, and the account, container, and object
|
|
||||||
name of the log. The keylist_mapping accepts no parameters.
|
|
||||||
|
|
||||||
-------------
|
|
||||||
Log Uploading
|
|
||||||
-------------
|
|
||||||
|
|
||||||
swift-log-uploader accepts a config file and a plugin name. It finds the log
|
|
||||||
files on disk according to the plugin config section and uploads them to the
|
|
||||||
swift cluster. This means one uploader process will run on each proxy server
|
|
||||||
node and each account server node. To not upload partially-written log files,
|
|
||||||
the uploader will not upload files with an mtime of less than two hours ago.
|
|
||||||
Rackspace runs this process once an hour via cron.
|
|
||||||
|
|
||||||
--------------
|
|
||||||
Log Processing
|
|
||||||
--------------
|
|
||||||
|
|
||||||
swift-log-stats-collector accepts a config file and generates a csv that is
|
|
||||||
uploaded to swift. It loads all plugins defined in the config file, generates
|
|
||||||
a list of all log files in swift that need to be processed, and passes an
|
|
||||||
iterable of the log file data to the appropriate plugin's process method. The
|
|
||||||
process method returns a dictionary of data in the log file keyed on (account,
|
|
||||||
year, month, day, hour). The log-stats-collector process then combines all
|
|
||||||
dictionaries from all calls to a process method into one dictionary. Key
|
|
||||||
collisions within each (account, year, month, day, hour) dictionary are
|
|
||||||
summed. Finally, the summed dictionary is mapped to the final csv values with
|
|
||||||
each plugin's keylist_mapping method.
|
|
||||||
|
|
||||||
The resulting csv file has one line per (account, year, month, day, hour) for
|
|
||||||
all log files processed in that run of swift-log-stats-collector.
|
|
||||||
|
|
||||||
|
|
||||||
================================
|
|
||||||
Running the stats system on SAIO
|
|
||||||
================================
|
|
||||||
|
|
||||||
#. Create a swift account to use for storing stats information, and note the
|
|
||||||
account hash. The hash will be used in config files.
|
|
||||||
|
|
||||||
#. Edit /etc/rsyslog.d/10-swift.conf::
|
|
||||||
|
|
||||||
# Uncomment the following to have a log containing all logs together
|
|
||||||
#local1,local2,local3,local4,local5.* /var/log/swift/all.log
|
|
||||||
|
|
||||||
$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
|
|
||||||
local1.*;local1.!notice ?HourlyProxyLog
|
|
||||||
|
|
||||||
local1.*;local1.!notice /var/log/swift/proxy.log
|
|
||||||
local1.notice /var/log/swift/proxy.error
|
|
||||||
local1.* ~
|
|
||||||
|
|
||||||
#. Edit /etc/rsyslog.conf and make the following change::
|
|
||||||
$PrivDropToGroup adm
|
|
||||||
|
|
||||||
#. `mkdir -p /var/log/swift/hourly`
|
|
||||||
#. `chown -R syslog.adm /var/log/swift`
|
|
||||||
#. `chmod 775 /var/log/swift /var/log/swift/hourly`
|
|
||||||
#. `service rsyslog restart`
|
|
||||||
#. `usermod -a -G adm <your-user-name>`
|
|
||||||
#. Relogin to let the group change take effect.
|
|
||||||
#. Create `/etc/swift/log-processor.conf`::
|
|
||||||
|
|
||||||
[log-processor]
|
|
||||||
swift_account = <your-stats-account-hash>
|
|
||||||
user = <your-user-name>
|
|
||||||
|
|
||||||
[log-processor-access]
|
|
||||||
swift_account = <your-stats-account-hash>
|
|
||||||
container_name = log_data
|
|
||||||
log_dir = /var/log/swift/hourly/
|
|
||||||
source_filename_pattern = ^
|
|
||||||
(?P<year>[0-9]{4})
|
|
||||||
(?P<month>[0-1][0-9])
|
|
||||||
(?P<day>[0-3][0-9])
|
|
||||||
(?P<hour>[0-2][0-9])
|
|
||||||
.*$
|
|
||||||
class_path = swift.stats.access_processor.AccessLogProcessor
|
|
||||||
user = <your-user-name>
|
|
||||||
|
|
||||||
[log-processor-stats]
|
|
||||||
swift_account = <your-stats-account-hash>
|
|
||||||
container_name = account_stats
|
|
||||||
log_dir = /var/log/swift/stats/
|
|
||||||
class_path = swift.stats.stats_processor.StatsLogProcessor
|
|
||||||
devices = /srv/1/node
|
|
||||||
mount_check = false
|
|
||||||
user = <your-user-name>
|
|
||||||
|
|
||||||
[log-processor-container-stats]
|
|
||||||
swift_account = <your-stats-account-hash>
|
|
||||||
container_name = container_stats
|
|
||||||
log_dir = /var/log/swift/stats/
|
|
||||||
class_path = swift.stats.stats_processor.StatsLogProcessor
|
|
||||||
processable = false
|
|
||||||
devices = /srv/1/node
|
|
||||||
mount_check = false
|
|
||||||
user = <your-user-name>
|
|
||||||
|
|
||||||
#. Add the following under [app:proxy-server] in `/etc/swift/proxy-server.conf`::
|
|
||||||
|
|
||||||
log_facility = LOG_LOCAL1
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to create the stats logs. In
|
|
||||||
`/etc/cron.d/swift-stats-log-creator`::
|
|
||||||
|
|
||||||
0 * * * * <your-user-name> /usr/local/bin/swift-account-stats-logger /etc/swift/log-processor.conf
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to create the container stats logs. In
|
|
||||||
`/etc/cron.d/swift-container-stats-log-creator`::
|
|
||||||
|
|
||||||
5 * * * * <your-user-name> /usr/local/bin/swift-container-stats-logger /etc/swift/log-processor.conf
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to upload the stats logs. In
|
|
||||||
`/etc/cron.d/swift-stats-log-uploader`::
|
|
||||||
|
|
||||||
10 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf stats
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to upload the stats logs. In
|
|
||||||
`/etc/cron.d/swift-stats-log-uploader`::
|
|
||||||
|
|
||||||
15 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf container-stats
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to upload the access logs. In
|
|
||||||
`/etc/cron.d/swift-access-log-uploader`::
|
|
||||||
|
|
||||||
5 * * * * <your-user-name> /usr/local/bin/swift-log-uploader /etc/swift/log-processor.conf access
|
|
||||||
|
|
||||||
#. Create a `cron` job to run once per hour to process the logs. In
|
|
||||||
`/etc/cron.d/swift-stats-processor`::
|
|
||||||
|
|
||||||
30 * * * * <your-user-name> /usr/local/bin/swift-log-stats-collector /etc/swift/log-processor.conf
|
|
||||||
|
|
||||||
After running for a few hours, you should start to see .csv files in the
|
|
||||||
log_processing_data container in the swift stats account that was created
|
|
||||||
earlier. This file will have one entry per account per hour for each account
|
|
||||||
with activity in that hour. One .csv file should be produced per hour. Note
|
|
||||||
that the stats will be delayed by at least two hours by default. This can be
|
|
||||||
changed with the new_log_cutoff variable in the config file. See
|
|
||||||
`log-processor.conf-sample` for more details.
|
|
||||||
@ -44,17 +44,6 @@ use = egg:swift#account
|
|||||||
# The replicator also performs reclamation
|
# The replicator also performs reclamation
|
||||||
# reclaim_age = 86400
|
# reclaim_age = 86400
|
||||||
|
|
||||||
[account-stats]
|
|
||||||
# You can override the default log routing for this app here (don't use set!):
|
|
||||||
# log_name = account-stats
|
|
||||||
# log_facility = LOG_LOCAL0
|
|
||||||
# log_level = INFO
|
|
||||||
# cf_account = AUTH_7abbc116-8a07-4b63-819d-02715d3e0f31
|
|
||||||
# container_name = account_stats
|
|
||||||
# proxy_server_conf = /etc/swift/proxy-server.conf
|
|
||||||
# log_facility = LOG_LOCAL0
|
|
||||||
# log_level = INFO
|
|
||||||
|
|
||||||
[account-auditor]
|
[account-auditor]
|
||||||
# You can override the default log routing for this app here (don't use set!):
|
# You can override the default log routing for this app here (don't use set!):
|
||||||
# log_name = account-auditor
|
# log_name = account-auditor
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user