8995 Commits

Author SHA1 Message Date
Zuul
06460cc8a5 Merge "Do not require the close method for response iterator" 2020-11-03 06:29:01 +00:00
Zuul
b9a404b4d1 Merge "ec: Add an option to write fragments with legacy crc" 2020-11-02 23:03:49 +00:00
Zuul
c75e23e81f Merge "Add tasks_per_second option to expirer" 2020-10-30 06:55:44 +00:00
Zuul
d9902abb19 Merge "Imported Translations from Zanata" 2020-10-29 23:25:24 +00:00
Takashi Kajinami
7b5868266a Do not require the close method for response iterator
Since fb08d477eb7bf5e678b9cd99b44a435842a7dfbf was merged, storlets
invocation on proxy fails because the close method is called even if
iterator has list_iterator type which doesn't have the close method.

This change replaces the direct call of the close method by
close_if_possible, to avoid Exception if the response iterator is
list or tuple, which don't support the close method.

Change-Id: I851cbb17465b5693dda5206aeae0a3191c9fe4f4
Closes-Bug: #1901453
2020-10-29 15:39:34 -07:00
OpenStack Proposal Bot
f665a6d423 Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Ic8a087c600ef0624ee815a330119ccc12c287f0f
2020-10-29 10:27:00 +00:00
Tim Burke
95212ac725 Add a slightly more-useful repr for DiskFiles
Change-Id: Ica2b637c9ee015c2cf129715e33d1ed1968b0a94
2020-10-27 11:55:05 -07:00
Clay Gerrard
b05ad82959 Add tasks_per_second option to expirer
This allows operators to throttle expirers as needed.

Partial-Bug: #1784753
Change-Id: If75dabb431bddd4ad6100e41395bb6c31a4ce569
2020-10-23 10:24:52 -05:00
Tim Burke
6b7eecab5d Get unit tests passing on py39
They finally got rid of Thread.isAlive, having added Thread.is_alive as
an alias back in 2.6.

array.tostring is also gone; use array.tofile instead, but only on py3.

Change-Id: I0b7e426c60118a5df879a8908729d7feb12dfc1b
2020-10-18 21:42:21 -07:00
Zuul
127bf9707c Merge "New proxy logging field for wire status" 2020-10-14 04:39:13 +00:00
OpenStack Proposal Bot
055e7b93a5 Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Iaff6cdd9e8de2589a56113867550e35d52cc068f
2020-10-11 08:24:38 +00:00
Clay Gerrard
fb08d477eb New proxy logging field for wire status
Capture the on the wire status code for logging because we change the
logged status code sometimes.

Closes-Bug: #1896518
Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df
2020-10-07 01:01:40 +00:00
Zuul
44ac59ddc3 Merge "py3: Fix non-ascii chars in staticweb listings" 2020-10-07 00:30:20 +00:00
Zuul
73c0c72735 Merge "probe-tests: Avoid a DuplicateSectionError on py3" 2020-10-07 00:11:43 +00:00
Zuul
4236a6efa3 Merge "Use bigger GreenPool for concurrent EC" 2020-10-03 06:54:16 +00:00
Zuul
4441a54ad1 Merge "wsgi: Ensure _response_headers is a list" 2020-10-03 05:27:17 +00:00
Zuul
227cb64788 Merge "Log storage policy with s3api requests" 2020-10-03 02:19:52 +00:00
Zuul
ed99099917 Merge "Fail short reads in SegmentedIterable" 2020-10-03 02:17:46 +00:00
Tim Burke
0d2604c110 wsgi: Ensure _response_headers is a list
It's reasonably common that apps or middlewares may send back headers
via dict.items(), but WSGIContext users often expect the headers to be
a list.  Convert it for them, to avoid errors like

    AttributeError: 'dict_items' object has no attribute 'append'

Change-Id: I4d061fad4da370c1cbb77ab78a55133319ea2dd7
2020-10-02 16:42:50 -07:00
Clay Gerrard
f4a34c2998 Log storage policy with s3api requests
Change-Id: I5fe5ab31d6b2d9f7b6ecb3bfa246433a78e54808
2020-10-01 17:18:13 -07:00
Zuul
454fceea18 Merge "Update master for stable/victoria" 2020-10-01 23:54:12 +00:00
Renich Bon Ćirić
6f4143d996 Clean up apache deployment docs
docs: Removing the use of NameVirtualHost from the apache examples

It's not used anymore. It's deprecated in fact: https://httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost

Change-Id: I76999cfacc10a244024ee0cca66dda95a0169a67

docs: Added more spacing to the apache2 examples

They're easier to read and a bit less bloated.

Change-Id: I5e21a66018b7ef309918fbbde93f2494286d291e

docs: Switching to /srv/www to be more FHS 3.0 conformat

It's more modern and well supported to use /srv/www now in place of
/var/www.

Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html

docs: Added user, group and display name in WSGI examples

This properly sets the user and group for the wsgi processes in the
examples; as well as adding a display name for easier identification.

Change-Id: Ie5783081e4054e5b2fbf3a856716101a1aaf61b8

docs: Replace apachectl for systemctl commands

It's safe to asume that all modern distros; supported by OpenStack, will
have systemd implemented. It's better to favor systemctl in those cases.

Change-Id: Ic0d2e47c1ac53502ce638d6fc2424ab9df037262

docs: Emphasis to file paths and command options

I've enclosed configuration options or parameters in interpreted text
quotes.

Also, I've enclosed fiel paths with inline literal quotes.

Change-Id: Iec54b7758bce01fc8e8daff48498383cb70c62ce

docs: Fixed wording used to indicate the restart of apache

Just a little commit to make it clearer of what we're gonna do.

Change-Id: Id5ab3e94519bcfe1832b92e456a1d1fa81dd54e3
2020-10-01 16:04:55 -05:00
Renich Bon Ćirić
38ab4593f4 docs: Switching to /srv/www from /var/www to be more FHS 3.0 conformat
It's more modern and well supported to use /srv/www now in place of
/var/www.

Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html
2020-10-01 15:55:40 -05:00
179a6ffb8f Update master for stable/victoria
Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: I229f74529a718be2dfd9627a9b60f677b67578c6
Sem-Ver: feature
2020-10-01 10:29:23 -07:00
Tim Burke
599f63e762 ec: Add an option to write fragments with legacy crc
When upgrading from liberasurecode<=1.5.0, you may want to continue
writing legacy CRCs until all nodes are upgraded and capabale of reading
fragments with zlib CRCs.

Starting in liberasurecode>=1.6.2, we can use the environment variable
LIBERASURECODE_WRITE_LEGACY_CRC to control whether we write zlib or
legacy CRCs, but for many operators it's easier to manage swift configs
than environment variables. Add a new option, write_legacy_ec_crc, to the
proxy-server app and object-reconstructor; if set to true, ensure legacy
frags are written.

Note that more daemons instantiate proxy-server apps than just the
proxy-server. The complete set of impacted daemons should be:

  * proxy-server
  * object-reconstructor
  * container-reconciler
  * any users of internal-client.conf

UpgradeImpact
=============
To ensure a smooth liberasurecode upgrade:

 1. Determine whether your cluster writes legacy or zlib CRCs. Depending
    on the order in which shared libraries are loaded, your servers may
    already be reading and writing zlib CRCs, even with old
    liberasurecode. In that case, no special action is required and
    WRITING LEGACY CRCS DURING THE UPGRADE WILL CAUSE AN OUTAGE.
    Just upgrade liberasurecode normally. See the closed bug for more
    information and a script to determine which CRC is used.
 2. On all nodes, ensure Swift is upgraded to a version that includes
    write_legacy_ec_crc support and write_legacy_ec_crc is enabled on
    all daemons.
 3. On each node, upgrade liberasurecode and restart Swift services.
    Because of (2), they will continue writing legacy CRCs which will
    still be readable by nodes that have not yet upgraded.
 4. Once all nodes are upgraded, remove the write_legacy_ec_crc option
    from all configs across all nodes. After restarting daemons, they
    will write zlib CRCs which will also be readable by all nodes.

Change-Id: Iff71069f808623453c0ff36b798559015e604c7d
Related-Bug: #1666320
Closes-Bug: #1886088
Depends-On: https://review.opendev.org/#/c/738959/
2020-09-30 16:49:59 -07:00
Tim Burke
9a0bac4ceb Fail short reads in SegmentedIterable
...and check for it *before* doing the MD5 check. We see this happen
ocassionally, but as best I can tell, it's always due to a
ChunkReadTimeout popping in the proxy that it can't recover from.

Change-Id: If238725bbec4fc3f6c8d000599c735a7c4972f7d
2020-09-28 13:56:00 -07:00
Clay Gerrard
d17ee8ba14 Pass thread_locals through to frag queue feeder
... so that we don't loose txn-id on error messages

Change-Id: Ic9bd98be115b8d97cf5ab879e01cee25aad497af
2020-09-25 11:57:21 -05:00
Clay Gerrard
5f95e1bece Use bigger GreenPool for concurrent EC
We're getting some blockage trying to feed backup requests in waterfall
EC because the pool_size was limited to the initial batch of requests.
This was (un?)fortunately working out in practice because there were
lots of initial primary fragment requests and some would inevitably be
quick enough to make room for the pending feeder requests.  But when
enough of the initial requests were slow (network issue at the proxy?)
we wouldn't have the expected number of pending backup requests
in-flight.  Since concurrent EC should never make extra requests to
non-primaries (at least not until an existing primary request
completes) ec_n_unique_fragments makes a reasonable cap for the pool.

Drive-bys:

 * Don't make concurrent_ec_extra_requests unless you have enabled
   concurrent_gets.
 * Improved mock_http_connect extra requests tracking formatting
 * FakeStatus __repr__'s w/ status code in AssertionErrors

Change-Id: Iec579ed874ef097c659dc80fff1ba326b6da05e9
2020-09-25 09:47:40 -05:00
aolivo
004052dc65 Fixing S3 credential caching region support
Change-Id: I53c7c2bebb61c48615e96f4fbf766a70d458bcb4
2020-09-21 17:46:20 -05:00
Tim Burke
41335f7b47 Revert "Make rolling-upgrade job non-voting"
This reverts commit 63e6c91ab86f3a832d7af9dc081ee90b73d8fa04.

Change-Id: I1af54679c48474d3aa375c5d1192871e9bb16c7d
2020-09-19 09:05:25 -07:00
Zuul
24d64a1b22 Merge "py3: Fix swift-dispersion-populate" 2020-09-19 01:42:30 +00:00
Zuul
60530753b3 Merge "Run swift-tox-func-encryption-py37 job in the gate" 2020-09-18 17:12:39 +00:00
Tim Burke
318c1a5660 Run swift-tox-func-encryption-py37 job in the gate
This used to be the name of the py2 job, but now even that's gone;
I'm not sure how Zuul let us get away with this.

Change-Id: I1a2f22d592fc7245e40c645000026e4ba1fca528
Related-Change: Ia9ae0fc226dfc9b40157faebac100c10a9180c62
2020-09-17 15:46:52 -07:00
Tim Burke
d6c6ab764e Authors/ChangeLog for 2.26.0
Change-Id: Ia8e31ed0d5aefe67f2f926dc92d9acd6c0c98007
victoria-em
2020-09-17 11:56:22 -07:00
Zuul
4b2d8c2258 Merge "proxy: Include thread_locals when spawning _fragment_GET_request" 2020-09-17 03:33:55 +00:00
Zuul
f5e03ee973 Merge "xlo: Drain error responses" 2020-09-15 22:11:21 +00:00
Tim Burke
a8d2146266 xlo: Drain error responses
Otherwise, they get logged as 499s. While we're at it, also drain DLO
bodies (if they're small).

Change-Id: I7b54f25f42577020b10029c74f8fc01fa6fc591e
2020-09-15 12:51:16 -07:00
Tim Burke
f8a9a6e116 s3api: Make quota-exceeded errors more obvious
Change-Id: Ia8db40227343e9c4555267c62072a1c9bfc28c66
Closes-Bug: #1893811
2020-09-11 15:37:33 -07:00
Tim Burke
5b8073c770 py3: Fix swift-dispersion-populate
We fixed swift-dispersion-report already; -populate needed the same fix
or else it'd hit a "maximum recursion depth exceeded" error.

Change-Id: I2d22e087a88c9e8003621feb26460ab6e5ce2a57
Related-Change: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745
Closes-Bug: #1895346
Related-Bug: #1863680
2020-09-11 19:35:32 +00:00
Zuul
4fefdf4d83 Merge "gate: Make rolling upgrade job work with either 60xx or 62xx ports" 2020-09-10 18:52:49 +00:00
Zuul
3f0e9490a6 Merge "Remove some useless swob.Request attr setting" 2020-09-10 03:11:36 +00:00
Tim Burke
ae8a89f042 gate: Make rolling upgrade job work with either 60xx or 62xx ports
Change-Id: If14f53993e6a0554345b327415812423b6b57434
Related-Change: If5dd79f926fa51a58b3a732b212b484a7e9f00db
2020-09-09 15:47:00 -07:00
Tim Burke
d5625abf60 proxy: Include thread_locals when spawning _fragment_GET_request
Otherwise, we miss out on transaction id and client IP information when
timeouts pop.

Closes-Bug: #1892421
Change-Id: I6dea3ccf780bcc703db8447a2ef13c33838ff12d
2020-09-08 15:00:02 -07:00
Clay Gerrard
754defc39c Client should retry when there's just one 404 and a bunch of errors
During a rebalance, it's expected that we may get a 404 for data that
does exist elsewhere in the cluster. Normally this isn't a problem; the
proxy sees the 404, keeps digging, and one of the other primaries will
serve the response.

Previously, if the other replicas were heavily loaded, the proxy would
see a bunch of timeouts and the fresh (empty) primary, treat the 404 as
good, and send that on to the client.

Now, have the proxy throw out that first 404 (provided it doesn't have a
timestamp); it will then return a 503 to the client, indicating that it
should try again.

Add a new (per-policy) proxy-server config option,
rebalance_missing_suppression_count; operators may use this to increase
the number of 404-no-timestamp responses to discard if their rebalances
are going faster than replication can keep up, or set it to zero to
return to the previous behavior.

Change-Id: If4bd39788642c00d66579b26144af8f116735b4d
2020-09-08 14:33:09 -07:00
Zuul
06091172c2 Merge "ec: Close down some unused responses more quickly" 2020-09-08 21:27:23 +00:00
Tim Burke
ba46eb0e3d py3: Fix non-ascii chars in staticweb listings
Change-Id: Ifcf810f009a8036f250a09eec0d9a65a77342129
Closes-Bug: #1894357
2020-09-06 22:04:06 -07:00
Zuul
cca5e8b1de Merge "Make all concurrent_get options per-policy" 2020-09-04 19:31:12 +00:00
Tim Burke
6e9e41d9b1 Remove some useless swob.Request attr setting
None of it was getting passed through to the WSGI env, so it had no
effect on anything.

Change-Id: Id0b9ea0b0d1cd03bb75233953abadbb17fadc426
2020-09-04 11:40:55 -07:00
Zuul
20e1544ad8 Merge "Extend concurrent_gets to EC GET requests" 2020-09-04 14:22:20 +00:00
Zuul
a7f3fe93ab Merge "Start to decouple Object GET path" 2020-09-04 14:22:17 +00:00