infra-specs/specs/migrate_askbot.rst

Copyright 2015 OpenStack Foundation

This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

Migrate ask.openstack.org to infra

https://storyboard.openstack.org/#!/story/2000158

The OpenStack Q&A site located at ask.openstack.org currently running in an instance deployed by a third-party. The main goal of this migration to make the entire deployment process repeatable, and move the project to proper infrastructure puppet repositories.

Problem Description

The ask.openstack.org site was deployed manually and contains the minimal application stack required for running the site. Currently the site is missing regular backups, security updates and deployment documentation.

Proposed Change

Create the missing puppet modules, prepare the changes in openstack-infra/system-config repo.

Alternatives

Leave ask.openstack.org as is.

Implementation

Assignee(s)

Primary assignee:

marton-kiss

Gerrit Topic

Use Gerrit topic "askbot-site" for all patches related to this spec.

git review -t askbot-site

Work Items

• Lower ask.openstack.org DNS TTL to 300
• Create puppet-askbot split-out module
• Add vamsee-solr module and puppet-askbot to modules.env
• Make the system-config changes, add ask.pp
• Add SSL certificates and passwords to hiera
• Launch new ask.openstack.org server
• Restore database and static files from original ask.openstack.org site
• Silent testing with /etc/hosts override
• Backup / restore of original ask.openstack.org data
• Update DNS entry of ask.openstack.org with the new server address
• Redirect html traffic using nginx to new ask.openstack.org to avoid db sync issues
• Restore ask.openstack.org DNS TTL to 3600

Repositories

A new puppet-askbot repository will need to be created, along with updates to system-config to consume this module.

Servers

An ask.openstack.org will need to be created.

DNS Entries

The ask.openstack.org zone must be point to the newly created server as the last step of this migration process.

Documentation

Askbot documentation need to be added to ci.openstack.org documentation.

Security

The services will run on Ubuntu, so core operating system not requires any special attention.

The application stack have some elements that must be deployed from tar.gz or pypy instead of OS packages:

• Apache Solr (4.7.2)
• askbot

Testing

Askbot don't have integration tests implemented. After instance creation and initial data migration, I suggest to do a 1-2 week long silent test of the UI and address upcoming bugs during that period.

Backup / recovery steps

Backup assets

Askbot holds all of the site data in a postgresql database, and all static files in the filesystem. In case of the existing deployment it is available at /srv/ask/dumps directory. The dumps are created by the shell script srv/ask/node2/cron/backup.sh executed daily by cron.

Invoke management commands

The management cli tool is available under /srv/askbot-sites/slot0/config, so to invoke management commands you need to enter this directory first.

$ cd /srv/askbot-sites/slot0/config
$ python manage.py <command> <args>

Stop celeryd and apache, jetty services

$ sudo service askbot-celeryd stop
$ sudo service apache2 stop
$ sudo service jetty stop

Restore database dump

Recreate an empty database:

$ sudo su - postgres -c 'dropdb askbotdb'
$ sudo su - postgres -c 'createdb askbotdb --owner=ask'

Restore from backup:

$ psql -d askbotdb -h localhost -U ask -W -f /path/to/last-dump.sql

Notice you will be prompted for the ask_db_password from hiera.

Sync db and migrate:

$ cd /srv/askbot-sites/slot0/config
$ sudo python manage.py syncdb
$ sudo python manage.py migrate

Notice this step is required to apply schema changes between askbot versions.

Start celeryd

$ sudo service askbot-celeryd start
$ sudo service apache2 start

Rebuild solr indexes

$ sudo service jetty start
$ cd /srv/askbot-sites/slot0/config
$ sudo python manage.py askbot_rebuild_index -l en
$ sudo python manage.py askbot_rebuild_index -l zh

Test the solr deployment, query string "sahara":

$ curl "http://127.0.0.1:8983/solr/core-en/select/?fq=django_ct%3A%28askbot.thread%29&rows=10&q=%28sahara%29&start=0&wt=json&fl=%2A+score"

Notice this query must return a non-empty resultset.

Restart celeryd

$ sudo service askbot-celeryd restart
$ sudo service apache2 restart

Restore static files

Static files must be extracted into /srv/askbot-sites/slot0/upfiles directory. It mostly holds profile pictures, and site logo, so if pictures not showing up in the site those files are missing, or have a wrong file permission.

$ cd /srv/askbot-sites/slot0
$ sudo rm -rf upfiles
$ sudo tar xf /path/to/last-upfiles.tar --strip-components=2

Dependencies

We are using vamsee-solr module 0.0.7 from puppetforge, and it is forcing Us to use solr 4.7.2 because 4.10.x requires some extra patches to work and this upgrade also means a schema change.