android13/external/cldr/tools/scripts/ansible

Ansible for Survey Tool

These are ansible scripts for setup and maintenance of the Survey Tool.

Scope

Right now, the test setup mostly controls OpenLiberty, but not the nginx proxy due to public port issues (https).

Setup

Setup: Control system

This is your local system, where you control the others from.

• Install Ansible https://ansible.com
• Install some prereqs:

ansible-galaxy install -r requirements.yml

• Make sure you can ssh into all of the needed systems. For example, ssh cldr-ref.unicode.org should succeed without needing a password.

• You should be able to run ansible all -m ping and get something back like the following:

cldr-ref.unicode.org | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

Setup: Managed systems

• Install python3. Make sure python --version or python3 --version returns "Python 3…"

• TODO: these shouldn't be needed, but they are. Here's the entire install command:

sudo apt-get update && sudo apt-get install python3 python-apt python3-pymysql

Setup: surveytool keypair

Create a RSA keypair with no password for the buildbot:

mkdir -p ./local-vars
ssh-keygen -t rsa -b 4096 -f ./local-vars/surveytool -P '' -C 'surveytool deploy'

The contents of the local-vars/surveytool.pub file is used for the key: parameter below in local.yml. The local-vars/surveytool private key is used in the secret RSA_KEY_SURVEYTOOL.

Then setup github secrets as shown:

• SMOKETEST_HOST - hostname of smoketest
• SMOKETEST_PORT - port of smoketest
• RSA_KEY_SURVEYTOOL - contents of local-vars/surveytool (the secret key)
• SMOKETEST_KNOWNHOSTS - run ssh-keyscan smoketest.example.com where smoketest.example.com is the name of the smoketest server. Put the results into this secret. One of these lines should match ~/.ssh/known_hosts on your own system when you ssh into smoketest. Try grep -i smoke ~/.ssh/known_hosts

Create a folder "cldrbackup" inside local-vars

mkdir -p ./local-vars/cldrbackup

Add three files inside local-vars/cldrbackup-vars: id_rsa, id_rsa.pub, and known_hosts. These must correspond to the public key for cldrbackup on corp.unicode.org. Copy existing versions if you have them. Otherwise, create new ones with ssh-keygen -t rsa and copy the public key to corp.unicode.org with ssh-copy-id -i ~/.ssh/id_rsa cldrbackup@corp.unicode.org

Setup: Config file

• Create a file local-vars/local.yml matching the example values in test-local-vars/local.yml but with secure passwords instead of hunter42, ...!

cldradmin_pw: hunter46 # needs to match cldradmin pw below
mysql_users:
  # this is the account used by the survey tool itself
  # password will match /var/lib/openliberty/usr/servers/cldr/server.env
  - name: surveytool
    host: localhost
    password: hunter42
    priv: 'cldrdb.*:ALL'
  # this is the account used for administrative tasks
  # password will match /home/cldradmin/.my.sql
  - name: cldradmin
    password: hunter46
    priv: 'cldrdb.*:ALL/*.*:PROCESS'
    append_privs: yes
# this is the account used for deployment
surveytooldeploy:
  # TODO: surveytooldeploy.password appears to be unused?
  password: hunter43
  # vap will match CLDR_VAP in /srv/st/config/cldr.properties
  vap: hunter44
  # testpw will match CLDR_TESTPW in /srv/st/config/cldr.properties
  testpw: hunter45
  oldversion: 39
  newversion: 40
  key: ssh-rsa …  ( SSH key goes here)
  certbot_admin_email: surveytool@unicode.org
  certbot_certs:
    - domains:
      - cldr-ref.unicode.org

Setup: cldrcc

mkdir -p local-vars/cldrcc
ssh-keygen -t rsa -b 2048 -C 'CLDR Commit Checker' -f local-vars/cldrcc/id_rsa

Configure

Run the setup playbook.

ansible-playbook --check setup-playbook.yml

This is in dry run mode. When it looks good to you, take the --check out and run it again.

You can also use the -l cldr-smoke.unicode.org option to limit the operation to a single host.

Local Test

• install vagrant and some provider such as virtualbox or libvirt

vagrant up

• To log into the new host, run vagrant ssh

• To iterate, trying to reapply ansible, run vagrant provision --provision-with=ansible

• to deploy ST to this, use the following:

(cd ../.. ; mvn package) # go to the tools folder and build ST (cldr-apps.war, etc.) if not already built
vagrant ssh -- sudo -u surveytool /usr/local/bin/deploy-to-tomcat.sh $(git rev-parse HEAD) < ../../cldr-apps/target/cldr-apps.war

• Now you should be able to login at http://127.0.0.1:8880/cldr-apps/

• If you need to get directly to the tomcat server, use:

vagrant ssh -- -L 9080:127.0.0.1:9080
# leave this shell window open.

Then, you can go to http://127.0.0.1:9080 and directly access tomcat.