Differences

This shows you the differences between two versions of the page.

--- howto-hyperv-ci [2015/08/07 14:26]
gsamfira
+++ howto-hyperv-ci [2015/09/25 16:03] (current)
ibalutoiu
@@ Line 11: / Line 11: @@
   * Gerrit account
   * Upstream port 29418 must be accessible
+  * A VLAN range large enough to allow tempests tests to run successfully. This vlan range needs to be allocated before the job starts, and it needs not to conflict with any other job.
+  * A list of MAC addresses that represent data-ports and external-ports. This is how the deployer correctly configures networking on the nodes.
 We are going to go through all the steps required to stand up a new Hyper-V CI. At the end of this guide, you should have:
@@ Line 20: / Line 22: @@
-Note, this guide does not cover high availability. That is a topic for a different howto.
+What's missing:
+  * High availability
+  * Log collection
+There is some level of manual configuration currently required, which will be automated in the future (relation between jenkins and zuul to take care of configuring the gearman connection for example).
 ===== Deploying juju =====
@@ Line 81: / Line 88: @@
 ===== Deploying the infrastructure =====
-Next, we need to deploy jenkins, zuul and active-directory. Jenkins charm exists upstream, and we will use it. The zuul charm is a bit custom made for this CI, so we need to clone it our github repository.
+Next, we need to deploy jenkins, zuul and active-directory. Jenkins charm exists upstream, and we will use it. The zuul charm is custom made for this CI, and not yet uploaded to the charm store so we need to clone it our github repository.
 So lets clone necessary charms locally:
@@ Line 130: / Line 137: @@
-Using that file we will now deploy our charms. Take note, charms will be configured to use the information you provide.
+Using that file we will now deploy our charms. Take note, charms will be configured to use the information you provide:
+  juju-deployer -L -S -c filename.yaml
 At the end you should have something like this:
@@ Line 156: / Line 165: @@
 The zuul charm sets up most of what we need automatically. However, given that jenkins allows the user to create arbitrary jobs, the layout.yaml may not fit with the job names you created. So after creating the jenkins jobs (not yet automated), you will need to tweak the layout.yaml for zuul.
-So lets access jenkins now:
+===== Preparing Jenkins =====
+==== Install prerequisites ====
+At this point, jenkins should be set up, including all the required plugins. We still need to clone the tools that will deploy the actual tests. This will also be automated in the future. There are a few areas that the upstream charm does not cover, which warrants a fork. For now, lets do it manually:
+  juju ssh jenkins/0
+you will automatically be logged in with the ubuntu user. This user has sudo rights. Lets set up some prerequisites:
+  sudo apt-add-repository -y ppa:juju/stable
+  sudo apt-get update
+  sudo apt-get -y install juju-deployer juju-core python-gevent python-jujuclient git
+  sudo -u jenkins -i
+  mkdir ~/scripts && cd ~/scripts
+  git clone https://github.com/cloudbase/common-ci.git
+  cd common-ci && git checkout redmondridge
+  mkdir -p ~/charms/trusty && cd ~/charms/trusty
+  git clone https://github.com/cloudbase/devstack-charm.git devstack
+  mkdir -p ~/charms/win2012r2 && cd ~/charms/win2012r2
+  git clone git@bitbucket.org:cloudbase/active-directory-charm.git active-directory
+  cd active-directory
+  ./download-juju-powershell-modules.sh
+  mkdir -p ~/charms/win2012hvr2 && cd ~/charms/win2012hvr2
+  git clone https://github.com/cloudbase/hyperv-charm.git hyper-v-ci
+  exit
+  sudo ln -s /var/lib/jenkins/scripts/common-ci/deployer/deployer.py /usr/bin/ci-deployer
+  exit
+We need to copy over the juju environment to the jenkins server so that it can deploy juju charms:
+<code bash>
+ADDRESS=$(juju run --unit jenkins/0 "unit-get public-address")
+scp -i ~/.juju/ssh/juju_id_rsa -r /home/ubuntu/.juju ubuntu@$ADDRESS:~/
+juju ssh jenkins/0 "sudo cp -a /home/ubuntu/.juju /var/lib/jenkins/.juju"
+juju ssh jenkins/0 "sudo chown jenkins:jenkins -R /var/lib/jenkins/.juju"
+juju ssh jenkins/0 "sudo chmod 700 -R /var/lib/jenkins/.juju"
+</code>
+==== Configure Jenkins ====
+With the prerequisites set up, we need to configure jenkins. Lets start by configuring the gearman connection. Both this and setting up the jenkins jobs are done via the web UI for now. This will change when we fork the charm to allow defining jobs and configuring plugins we need from the charm itself. Find out the jenkins address:
   juju status --format oneline jenkins
-This hsould show you the public address of the jenkins nodes deployed:
+This should show you the public address of the jenkins nodes deployed:
   $ juju status --format oneline jenkins
@@ Line 166: / Line 222: @@
   - jenkins/0: stunning-digestion.maas (started) 8080/tcp
-If you've set up the MaaS node as your local resolver, then you should be able to resolve the randomly nenerated hostname (stunning-digestion.maas in this case). If not, you can resolve it by simply doing:
+If you've set up the MaaS node as your local resolver, then you should be able to resolve the randomly generated hostname (stunning-digestion.maas in this case). If not, you can resolve it by simply doing:
   nslookup stunning-digestion.maas <maas ip>
@@ Line 180: / Line 236: @@
 Navigate over to http://10.255.251.177:8080 and log in using:
   * username: admin (default)
   * password: SuperSecretPassword (the one specified in infra.yaml)
+Now lets check that jenkins can connect to the gearman server provided by zuul. First we need to get the zuul address:
+  ubuntu@ubuntu-maas:~$ juju status --format line zuul
+  - zuul/0: exalted-nerve.maas (started)
+Navigate to the jenkins configuration page:
+ {{ :manage_jenkins.png?900 |}}
+and search for **gearman**:
+{{ :gearman_settings.png?900 |}}
+Click on **Test connection** to make sure jenkins can connect, and save the settings.
+==== Create jenkins job ====
+Time to set up the jenkins jobs.
+Click on Create new job:
+{{ :new_job.png?900 |}}
+Select Freestyle project and set "nova-hyperv-tempest" as a name:
+{{ :job_name.png?900 |}}
+Select Add build step, and select "Execute Shell":
+{{ :build_step.png?900 |}}
+Add the following content to the job:
+<code bash>
+#!/bin/bash
+function exec_with_retry () {
+    local max_retries=$1
+    local interval=${2}
+    local cmd=${@:3}
+    local counter=0
+    while [ $counter -lt $max_retries ]; do
+        local exit_code=0
+        eval $cmd || exit_code=$?
+        if [ $exit_code -eq 0 ]; then
+            return 0
+        fi
+        let counter=counter+1
+        if [ -n "$interval" ]; then
+            sleep $interval
+        fi
+    done
+    return $exit_code
+}
+set -x
+set +e
+BUNDLE_LOCATION=$(mktemp)
+UUID=$ZUUL_UUID
+cat << EOF > $BUNDLE_LOCATION
+nova:
+  overrides:
+    data-port: "space delimited mac address list"
+    external-port: "space delimited mac address list"
+    zuul-branch: $ZUUL_BRANCH
+    zuul-change: "$ZUUL_CHANGE"
+    zuul-project: $ZUUL_PROJECT
+    zuul-ref: $ZUUL_REF
+    zuul-url: $ZUUL_URL
+  relations:
+  - - devstack-$UUID
+    - hyper-v-ci-$UUID
+  - - hyper-v-ci-$UUID
+    - active-directory
+  services:
+    active-directory:
+      branch: https://github.com/cloudbase/active-directory.git
+      charm: local:win2012r2/active-directory
+      num_units: 1
+      options:
+        domain-name: cloudbase.local
+        password: Passw0rd
+    devstack-$UUID:
+      branch: https://github.com/cloudbase/devstack-charm.git
+      charm: local:trusty/devstack
+      num_units: 1
+      options:
+        disabled-services: horizon n-novnc n-net n-cpu ceilometer-acompute
+        enable-plugin: networking-hyperv|https://github.com/stackforge/networking-hyperv.git
+        enabled-services: rabbit mysql key n-api n-crt n-obj n-cond n-sch n-cauth
+          neutron q-svc q-agt q-dhcp q-l3 q-meta q-lbaas q-fwaas q-metering q-vpn
+          g-api g-reg cinder c-api c-vol c-sch c-bak s-proxy s-object s-container
+          s-account heat h-api h-api-cfn h-api-cw h-eng tempest
+        extra-packages: build-essential libpython-all-dev python-all python-dev python3-all
+          python3.4-dev g++ g++-4.8 pkg-config libvirt-dev
+        extra-python-packages: networking-hyperv
+        heat-image-url: http://10.255.251.252/Fedora.vhdx
+        test-image-url: http://10.255.251.252/cirros.vhdx
+        vlan-range: 2500:2550
+    hyper-v-ci-$UUID:
+      branch: https://github.com/cloudbase/hyperv-charm
+      charm: local:win2012hvr2/hyper-v-ci
+      num_units: 2
+      options:
+        download-mirror: http://64.119.130.115/bin
+        extra-python-packages: setuptools SQLAlchemy==0.9.8 wmi oslo.i18n==1.7.0 pbr==1.2.0
+        git-user-email: hyper-v_ci@microsoft.com
+        git-user-name: Hyper-V CI
+        pypi-mirror: "space delimited pypi mirrors list"
+EOF
+export JUJU_REPOSITORY=$HOME/charms
+ci-deployer deploy --template $BUNDLE_LOCATION --search-string $UUID
+build_exit_code=$?
+if [[ $build_exit_code -eq 0 ]]; then
+	#run tempest
+	source nodes
+	project=$(basename $ZUUL_PROJECT)
+	exec_with_retry 5 2 ssh -tt -o 'PasswordAuthentication=no' -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' -i ~/.juju/ssh/juju_id_rsa ubuntu@$DEVSTACK "git clone https://github.com/cloudbase/common-ci.git /home/ubuntu/common-ci"
+    clone_exit_code=$?
+	exec_with_retry 5 2 ssh -tt -o 'PasswordAuthentication=no' -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' -i ~/.juju/ssh/juju_id_rsa ubuntu@$DEVSTACK "git -C /home/ubuntu/common-ci checkout redmondridge"
+	checkout_exit_code=$?
+	exec_with_retry 5 2 ssh -tt -o 'PasswordAuthentication=no' -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' -i ~/.juju/ssh/juju_id_rsa ubuntu@$DEVSTACK "mkdir -p /home/ubuntu/tempest"
+	ssh -tt -o 'PasswordAuthentication=no' -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' -i ~/.juju/ssh/juju_id_rsa ubuntu@$DEVSTACK "/home/ubuntu/common-ci/devstack/bin/run-all-tests.sh --include-file /home/ubuntu/common-ci/devstack/tests/$project/included_tests.txt --exclude-file /home/ubuntu/common-ci/devstack/tests/$project/excluded_tests.txt --isolated-file /home/ubuntu/common-ci/devstack/tests/$project/isolated_tests.txt --tests-dir /opt/stack/tempest --parallel-tests 4 --max-attempts 4"
+	tests_exit_code=$?
+	exec_with_retry 5 2 ssh -tt -o 'PasswordAuthentication=no' -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' -i ~/.juju/ssh/juju_id_rsa ubuntu@$DEVSTACK "/home/ubuntu/devstack/unstack.sh"
+fi
+#collect logs form Windows nodes
+#collect logs from devstack node
+#upload logs to logs server
+#destroy charms, services and used nodes.
+ci-deployer teardown --search-string $ZUUL_UUID
+cleanup_exit_code=$?
+if [[ $build_exit_code -ne 0 ]]; then
+	echo "CI Error while deploying environment"
+	exit 1
+fi
+if [[ $clone_exit_code -ne 0 ]]; then
+	echo "CI Error while cloning the scripts repository"
+	exit 1
+fi
+if [[ $checkout_exit_code -ne 0 ]]; then
+	echo "CI Error while checking out the required branch of the scripts repository"
+	exit 1
+fi
+if [[ $tests_exit_code -ne 0 ]]; then
+	echo "Tempest tests execution finished with a failure status"
+	exit 1
+fi
+if [[ $cleanup_exit_code -ne 0 ]]; then
+	echo "CI Error while cleaning up the testing environment"
+	exit 1
+fi
+exit 0
+</code>
+There are a few things you should remember about this job:
+  * the VLAN range needs to be computed before the job starts
+  * the MAC address list needs to be known beforehand.
+  * the extra python packages should be used only during CI quirkiness (requirements include a bad version of PBR for example).
+In this PoC, we have a static list of MAC addresses and VLAN ranges, but this should be queried from a single source of truth (say an API that hosts resources such as VLAN ranges and MAC addresses).
+===== Configuring up Zuul =====
+As mentioned before, this is a manual step that will change once we integrate the jenkins and zuul charms. For now lets make sure layout.yaml is correctly configured. Edit **/etc/zuul/layout.yaml** and set the following content:
+<code yaml>
+includes:
+  - python-file: openstack_functions.py
+pipelines:
+  - name: check
+    description: Newly uploaded patchsets enter this pipeline to be validated. Voting is neutral, only comments with results are sent.
+    failure-message: Build failed. For rechecking only on the Hyper-V CI, add a review comment with check hyper-v
+    manager: IndependentPipelineManager
+    precedence: low
+    trigger:
+      gerrit:
+        - event: patchset-created
+        - event: change-restored
+        - event: comment-added
+          comment_filter: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*(recheck|reverify|check hyper-v)
+    dequeue-on-new-patchset: true
+jobs:
+  - name: ^.*$
+    parameter-function: set_log_url
+  - name: nova-hyperv-tempest
+    voting: false
+  - name: neutron-hyperv-tempest
+    voting: false
+  - name: check-noop
+    voting: false
+projects:
+  - name: openstack/nova
+    check:
+      - nova-hyperv-tempest
+</code>
+Save the file and restart zuul:
+<code bash>
+restart zuul-merger
+restart zuul-server
+</code>
+You should be all done (If I have not forgotten anything). Make sure you check zuul logs for patch update status. This will be the likely place to look if you get no jobs in jenkins at all.

Cloudbase Solutions Wiki

User Tools

Site Tools

Differences

Page Tools