442 Commits

Author SHA1 Message Date
janonymous
8eca02d3a1 Replace xrange() with six.moves.range()
xrange is no longer available in py3.
replacing xrange with range from six.moves

Change-Id: Ib863c316a0724bd9c4f53c2e5a8d1bcd42c4dc5a
2015-07-03 01:00:28 +05:30
Jenkins
f5d6f8c23a Merge "Add policy support to dispersion tools" 2015-06-25 09:42:00 +00:00
paul luse
e6165a7879 Add policy support to dispersion tools
Doesn't work for anything other than policy 0. updated to allow user
to specify policy name on cmd line (as with object-info) which
then makes populate/report work with 3x, 2x, or EC style policies

Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
Closes-Bug: 1458688
2015-06-23 02:14:02 -07:00
Mark Seger
af734b3fb6 Change usage help and Attention messages to warnings
Change-Id: I1396aaffe36e739606f15f7fef37b11bd83f1fc1
2015-06-03 15:32:25 -04:00
paul luse
647b66a2ce Erasure Code Reconstructor
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
  - There is no notion of update() or update_deleted().
  - There is a single job processor
  - Jobs are processed partition by partition.
  - At the end of processing a rebalanced or handoff partition, the
    reconstructor will remove successfully reverted objects if any.

And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
2015-04-14 00:52:17 -07:00
Lorcan
0a46793662 Add swift-recon feature to track swift-drive-audit error count
This is a follow-on from a previous commit which added recon info
for swift-drive-audit (https://review.openstack.org/#/c/122468/).

Here, the "--drievaudit" option is added to swift-recon tool. This
feature gives the statistics for the system-wide drive errors flagged
by swift-drive-audit. An example of the output is as follows:
(verbose mode)

swift-recon --driveaudit -v
===============================================================================
--> Starting reconnaissance on 5 hosts
===============================================================================
[2015-03-11 17:13:39] Checking drive-audit errors
-> http://1.2.3.4:6000/recon/driveaudit: {'drive_audit_errors': 14}
-> http://1.2.3.5:6000/recon/driveaudit: {'drive_audit_errors': 0}
-> http://1.2.3.6:6000/recon/driveaudit: {'drive_audit_errors': 37}
-> http://1.2.3.7:6000/recon/driveaudit: {'drive_audit_errors': 101}
-> http://1.2.3.8:6000/recon/driveaudit: {'drive_audit_errors': 0}
[drive_audit_errors] low: 0, high: 101, avg: 30.4, total: 152, Failed: 0.0%, no_result: 0, reported: 5
===============================================================================

Change-Id: Ia16c52a9d613eeb3de1a5a428d88dd1233631912
2015-03-23 11:38:32 +00:00
Jenkins
49586c6ed7 Merge "Prevent redundant commenting by drive-audit" 2015-02-03 22:33:18 +00:00
Takashi Kajinami
a270dca239 Prevent redundant commenting by drive-audit
The drive-audit detects error log about a device and comments out it
in /etc/fstab. When the error log is generated several times, it
comments out the line for each time.
This patch makes drive-audit to check if the device is already
commented out, and prevents redundant commenting out.

Change-Id: Ia542d35b58552dde0f324bb9c42531f98c9058fa
2015-02-02 22:01:29 +09:00
Samuel Merritt
3ac43e8299 Allow per-policy overrides in object replicator.
The replicator already supports --devices and --partitions to restrict
its operation to a subset of devices and partitions. However,
operators don't always want to replicate a partition in all policies
since different policies (usually) have different rings.

For example, if I know that policy 0's partition 1234 is has no
replicas on primary nodes due to over-aggressive rebalancing, I really
want to find a node where the partition isa and make the replicator
push it onto the primaries. However, if I haven't been messing with
policy 1's ring, its partition 1234 is fine. With the existing
replicator args, I get both or neither; this commit lets me get just
the useful one.

Change-Id: Ib1d58fdd228a6ee7865321e65d7c04a891fa5c49
2015-01-22 16:10:22 -08:00
Keshava Bharadwaj
1ffe6b3953 Adds console logging to swift-drive-audit
This patch adds console logging ability to swift-drive-audit.
There are cases where logging to console is necessary when drive-audit
is done. This can be consumed for flagging errors in monitoring tools
such as icinga.

DocImpact
Change-Id: Ia1e1effcbd89bd2cf6d5b8c64019f1647c736a3a
2014-12-18 14:19:12 +05:30
Cedric Dos Santos
a8751ae557 Correct misspelled words
In some files I found misspelling words.

bin/swift-reconciler-enqueue#l26
   primarly => primarily
swift/account/backend.py#l309
   ommited => omitted
swift/container/replicator.py#l158
   successfull => successful
test/unit/account/test_backend.py#1450
   non_existant_policy_index => non_existent_policy_index
test/unit/account/test_backend.py#1451
   'test-non-existant-policy'=> 'test-non-existent-policy'
test/unit/account/test_backend.py#1453
   non_existant_policy_index => non_existent_policy_index

Change-Id: I976236e3200a6fbdc20be464acff182b6cface81
2014-11-25 15:44:30 +00:00
Daisuke Morita
2792fe81a9 Show the sum of every policy's amount in /recon/async
After the release of Swift ver. 2.0.0, some recon responses do not
show each policy's information yet. To make things worse, some recon
results only count on policy-0's score, therefore the total is not
shown in the recon results.

With this patch, async_pending count of recon results becomes
policy-aware. Suppose a number of async_pending files for policy-0 is 2
and a number for policy-1 is 3, recon sums up every policy's amount
as follows.

$ curl http://<host>:<port>/recon/async
{"async_pending": 5} # It showed 2 before this commit

Related-Bug: 1375332
Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412
2014-11-14 20:47:04 +09:00
Filippo Giunchedi
dec9448c87 swift-drive-audit: pass logger to get_errors
Closes-Bug: #1379767
Change-Id: Ie8ec5e9ffb977ab39aedeb878e6aae9f0b62530b
2014-10-10 12:51:53 +01:00
Jenkins
8647f86494 Merge "Add new features to swift-drive-audit" 2014-10-10 09:24:41 +00:00
Lorcan
cb20763893 Add new features to swift-drive-audit
This patch adds two new features to swift-drive-audit. The first
is an option in the drive-audit.conf file that allows the operator
to prevent the drives ever being unmounted automatically,
regardless of the amount of errors present. This could be of
benefit in very small systems consisting of only one or two drives
where the operator would like to manually unmount/fix the
particular drive(s) and minimise any potential downtime.

The second is another option in drive-audit.conf that allows the
operator to select a recon directory. This directory will then
have a drive.recon file which will keep an up-to-date record of
the swift drives and any errors associated with them. An example
of the output would be as follows:

{"/srv/node/disk2": "0", "/srv/node/disk3": "25", "/srv/node/disk0": "0",
"/srv/node/disk1": "0", "/srv/node/disk10": "0", "/srv/node/disk7": "0",
"/srv/node/disk4": "137", "/srv/node/disk5": "0", "/srv/node/disk8": "0",
"/srv/node/disk9": "0", "/srv/node/disk6": "0", "/srv/node/disk11": "60"}

This would allow the operator to monitor the errors on the swift
drives without having to spend time searching through logs. Also, if
this is accepted, it should be possible to add an option to
swift-recon that would keep track of this at a system level.

Change-Id: Ib5dacf8622b7363e070c274c7c30c8ead448a055
2014-09-25 16:27:25 +01:00
Jenkins
607e2ed2e3 Merge "Add "--no-overlap" option to swift-dispersion populate" 2014-09-22 15:44:14 +00:00
Lorcan
4faf170270 Add "--no-overlap" option to swift-dispersion populate
This change allows the user to use a "--no-overlap" parameter when
running the tool multiple times. It will increase the coverage by
whatever is specified in the dispersion_coverage field of the conf
file in a manner where existing container/objects are left in place
and no partition is populated more than once.

Related-Bug: #1233045

Change-Id: I139fed2f4c967ba18d073b7ecd1e946ed4da1271
2014-09-19 10:44:30 +01:00
John Dickinson
b7281cf2c5 make the bind_port config setting required
In a long-term effort to change the recommended ports for Swift,
the first step is to require the bind_port in config files. Later,
we can change the recommended setting.

Anyone currently explicitly setting the ports will not be affected.
Anyone not setting the ports will need to specify them to match their
rings.

DocImpact

Change-Id: Icca83a263acdd0afc9016424a3e9f8c15e944789
2014-09-08 07:28:43 -07:00
Samuel Merritt
4f2bb9f271 Make swift-form-signature testable
Moved the body of bin/swift-form-signature into
swift/cli/form_signature.py, like was done with swift-ring-builder and
others. Added a couple basic tests; there's not 100% coverage, but
it's better than the 0% coverage we had before.

It's almost a straight forklift, but I changed exit() calls to return
statements.

Change-Id: Ie2f702c070da24d9cdface83b9e838e9e2965085
2014-07-24 14:38:53 -07:00
Yuan Zhou
6cc10d17de Update bin scripts to be storage policy aware
swift-container-info:
    Print policy container info

swift-object-info:
    Allow to specify storage policy name when looking for object info
    Notify if there is missmatch between ring location and the actual
    object path in filesystem

swift-get-nodes:
    Allow to specify storage policy name when looking for account/
    container/object ring location
    Notify if there is missmatch between ring and the policy

Lookup policy name in swift.conf; 'Legacy' container will use
policy-0's name; 'Unknown' is shown if policy not found in swift.conf

DocImpact
Implements: blueprint storage-policies
Change-Id: I450d40dc6e2d8f759187dff36d658e52737ae2a5
2014-06-18 20:57:09 -07:00
Clay Gerrard
3fc4d6f91d Add container-reconciler daemon
This daemon will take objects that are in the wrong storage policy and
move them to the right ones, or delete requests that went to the wrong
storage policy and apply them to the right ones. It operates on a
queue similar to the object-expirer's queue.

Discovering that the object is in the wrong policy will be done in
subsequent commits by the container replicator; this is the daemon
that handles them once they happen.

Like the object expirer, you only need to run one of these per cluster
see etc/container-reconciler.conf.

DocImpact
Implements: blueprint storage-policies
Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
2014-06-18 17:31:39 -07:00
Jenkins
c384d76c57 Merge "Add --quoted option to swift-temp-url" 2014-06-03 05:10:28 +00:00
Chuck Thier
0a122c1575 Add targeted config loading to swift-init
This allows an easier and more explicit way to tell swift-init to run on
specific servers.  For example with an SAIO, this allows you to do
something like:

swift-init object-server.1 reload

to reload just the 1st object server.  A more real world example is when
you are running separate servers for replication.  In this example you
might have an object-server/public.conf and
object-server/replication.conf.  With this change you can do something
like:

swift-init object-server.replication reload

to just reload the replication server.

DocImpact
Change-Id: I5c6046b5ee28e17dadfc5fc53d1d872d9bb8fe48
2014-05-19 14:43:50 +00:00
Clay Gerrard
b7659bee26 Add --quoted option to swift-temp-url
If you have a path with special characters it may be easier to hand them to
swift-temp-url prequoted than try to escape them on the command line.  By the
time common.middleware.tempurl gets ahold of the path it's unquoted so we do
the same before calculating the hmac but still use the pre-quoted path output
to the commandline.

Change-Id: Ia1a9666e487b1e70e4db7cd597bc6a1027e3e918
2014-05-02 20:11:12 -07:00
Luis de Bethencourt
3af5216543 Sort the log file dates in reverse order
We want to show log files in descending order.

Change-Id: I6f79d9f5a73afa43efec36c3ca99afcde3fe7813
Closes-Bug: #1311405
2014-04-25 11:19:47 -04:00
Yuan Zhou
39f5eab890 Clean up swift-{account, container}-info
Reuse common code; add unit tests; ensured coverage was at 100%.

Change-Id: Id6fcc7cb07fd178e00d43968e3e2cc03226fdc05
2014-04-03 09:54:58 -04:00
Jenkins
6e44a8499a Merge "Update swift-get-nodes to use storage_directory util func" 2014-03-31 19:36:01 +00:00
Jenkins
a7d54467cd Merge "Added swift-account-info tool." 2014-03-31 18:00:17 +00:00
Jenkins
a59a98f96c Merge "minor cleanups for swift-container-info" 2014-03-31 15:42:09 +00:00
Madhuri Kumari
c90ede29ff Added swift-account-info tool.
This is a very simple swift tool to retrieve information
of an account that is located on the storage node.
One can call the tool with a given account db file
as it is stored on the storage node system.
It will then return several information about that account.

Change-Id: Ibfeee790adc000fc177b4b3c03d22ff785fda325
2014-03-31 10:05:52 +05:30
Jenkins
cdbf4f1f91 Merge "Removed hard coded location of ring" 2014-03-26 21:17:19 +00:00
Clay Gerrard
12c9dbe5f7 minor cleanups for swift-container-info
Change-Id: I4d0687f64524093e2ffaa4f8d31bcff70dca145e
2014-03-26 13:26:13 -07:00
Yuan Zhou
4b7bab9fc9 Update swift-get-nodes to use storage_directory util func
Change-Id: I074d6c5f62e5218427c72b3e11de40da4ecfc0ab
2014-03-26 10:47:22 +08:00
Madhuri Kumari
a37f90b3f1 Removed hard coded location of ring
In bin/swift-object-info, removed the hard coded location of ring.

Change-Id: Ie328633f5f4de755c9dcc4bb52341bb97087419d
2014-03-25 16:14:00 +05:30
Madhuri Kumari
6b441fabd9 Added swift-container-info tool.
This is a very simple swift tool to retrieve information
of a container that is located on the storage node.
One can call the tool with a given container db file
as it is stored on the storage node system.
It will then return several information about that container.

Change-Id: Ifebaed6c51a9ed5fbc0e7572bb43ef05d7dd254b
2014-03-22 17:30:20 +05:30
Eamonn O'Toole
793489b80d Allow specification of object devices for audit
In object audit "once" mode we are allowing the user to specify
a sub-set of devices to audit using the "--devices" command-line
option.  The sub-set is specified as a comma-separated list.  This
patch is taken from a larger patch to enable parallel processing
in the object auditor.

We've had to modify recon so that it will work properly with this
change to "once" mode.    We've modified dump_recon_cache()
so that it will store nested dictionaries, in other words it will
store a recon cache entry such as {'key1': {'key2': {...}}}. When
the object auditor is run in "once" mode with "--devices" set the
object_auditor_stats_ALL and ZBF entries look like:
{'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When
swift-recon is run, it hunts through the nested dicts to find the
appropriate entries.  The object auditor recon cache entries are set
to {} at the beginning of each audit cycle, and individual disk
entries are cleared from cache at the end of each disk's audit cycle.

DocImpact

Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
2014-03-11 14:17:08 +00:00
Jenkins
0bfbbdc5ee Merge "Fix misspellings in swift" 2014-02-21 01:36:53 +00:00
Shane Wang
a94be9443d Fix misspellings in swift
Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -

Change-Id: I6594fc4ca5ae10bd30eac8a2f2493a376adcadee
Closes-Bug: #1257295
2014-02-20 16:15:48 +08:00
Christian Schwede
39d22ccec7 Don't create bin/* files magically
Just use import to make scripts available in bin/ instead of
creating these during setup.py install.

Change-Id: I7318bbb77f6564ed58736887e711e1c497873471
2014-02-13 10:51:27 +00:00
Jenkins
8902aef591 Merge "Remove swiftclient dependency" 2014-02-11 02:18:06 +00:00
Christian Schwede
1f3ae6d8da Remove swiftclient dependency
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.

The dependency in swift/container/sync.py has also been removed.

Implements: blueprint remove-swiftclient-dependency

Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
2014-02-06 09:44:58 +00:00
Christian Schwede
24657b2b39 Add tests for swift-ring-builder
Add some tests for essential methods in swift-ring-builder.
Tests for removing or changing device settings are executed
with different search values to cover many possible command
line arguments.

Currently tested methods:

- create ring
- add device
- remove device
- set weight
- set info
- set min_part_hours
- set replicas

Tests use swift.common.ring.RingBuilder to verify actions.

Catching and testing output from print statements is not
tested, because this requires redirecting sys.stdout during
tests and that might have some sideeffects for testing tools.

bin/swift-ring-builder has been moved to swift/cli/ringbuilder.py
and slightly modified to work as before (mainly due to no more
existing global variables since that part of the code has been
moved inside a main() function).

Change-Id: Ia63f59a8faca1fad990784f27532ca07a2125454
2014-02-05 16:20:09 +00:00
Jenkins
b1cacf4c93 Merge "Let swift-object-info skip etag verification" 2014-02-05 01:55:49 +00:00
Christian Schwede
cd4b4da8b6 Add some tests for bin/swift-recon
Fix also minor bug in zone filtering when zone set to 0.

Moved bin/swift-recon to swift/cli/recon.py, which makes
it possible to import it without using some scary hacks.
bin/swift-recon is now created by setup.py install.

Closes-Bug: #1261692
Change-Id: Id0729991c8ece73604467480dbf93fec7d8eb196
2014-01-31 15:34:37 +00:00
Jenkins
6b8f845c98 Merge "Make swift-recon usable on hosts without IPv6" 2014-01-31 00:07:22 +00:00
Christian Schwede
866c568cd6 Make swift-recon usable on hosts without IPv6
Fixes a bug when swift-recon --sockstat is used on hosts without
IPv6 support.

Tested by disabling IPv6 on Ubuntu 12.04 LTS:

Add "ipv6.disable=1" to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
sudo update-grub
sudo /sbin/reboot

Closes-Bug: 1270711
Change-Id: Ib31a059e412ac68ca0a3faef4201fec7560d1178
2014-01-20 07:13:20 +00:00
Samuel Merritt
a7e0a9d57d Fix swift-dispersion-report when partitions overlap
swift-dispersion-report tries to avoid checking a partition more than
once, so it keeps track of partitions already queried and skips
duplicates.

swift-dispersion-report also keeps track of the number of successful
responses; it counts the number of expected replicas to find, and also
counts the number of replicas actually found, and tells the operator
if the numbers differ.

However, in the case that a partition was duplicated, the
expected-responses counter was incremented, but the actual check was
skipped, so it looked as though some copies were missing. Now we only
increment the expected-responses counter if we're actually going to
perform the check.

Change-Id: I22ac2b8066b62ca7c8ebf099c9f602118bb1a298
2014-01-17 11:45:57 -08:00
Samuel Merritt
92fc917b4b Let swift-object-info skip etag verification
Now swift-object-info has a "-n" option that will cause the etag
verification to be skipped; on large objects, the etag verification
takes the vast majority of the runtime, and sometimes you just want to
know which account owns the 5 GB monstrosity without waiting around while
its checksum is verified.

Change-Id: Id284570633eb7b98046cdb948d7c37a152de1195
2014-01-16 12:55:50 -08:00
Jenkins
db63240d7e Merge "Remove swiftclient dep on direct_client" 2014-01-07 21:24:27 +00:00
Florian Hines
f9d14971c2 Don't report async pendings on exception
If we encounter an exception trying to gather async pendings 'async'
doesn't exist and the cronjob ends up erroring out and leaving behind a
stale lock file.

Change-Id: I70a6d3f00bd2c9ce742e6d16af93804280707040
2014-01-04 00:19:22 -06:00