Difference between revisions of "OpenStack deployment via packstack from RDO"

From Christoph's Personal Wiki
Jump to: navigation, search
(→‎Open vSwitch (OVS))
 
(33 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
This article will cover the steps involved in getting OpenStack deployed using "<code>packstack</code>" from [https://www.rdoproject.org/ RDO].
+
This article will cover the steps involved in getting [[:Category:OpenStack|OpenStack]] deployed using "<code>packstack</code>" from [https://www.rdoproject.org/ RDO].
  
−
NOTE: Article under construction.
+
<div style="margin: 10px; padding: 5px; border: 2px solid red;">'''WARNING:''' This article is very much under construction. Proceed with caution until this banner has been removed.</div>
  
 
* Deploy assumptions:
 
* Deploy assumptions:
 
** OS: [[CentOS]] 7.1 (64-bit; 7.1.1503 Core)
 
** OS: [[CentOS]] 7.1 (64-bit; 7.1.1503 Core)
 
** OpenStack release: "Kilo" (April 2015)
 
** OpenStack release: "Kilo" (April 2015)
 +
** RDO release: rdo-release-kilo-1 (noarch; 12 May 2015)
  
 
==Single node==
 
==Single node==
−
''Note: Using neutron with a flat network driver.''
+
''Note: Using neutron with a "flat" network driver.''
  
−
  $ sysctl -a | grep ip_forward #=> 1
+
  $ sysctl -a | grep ip_forward #=> 1 (see: [[sysctl]])
−
  $ sestatus #=> set to "permissive"
+
  $ sestatus #=> set to "permissive" (see: [[selinux]])
 
  $ systemctl stop NetworkManager.service
 
  $ systemctl stop NetworkManager.service
 
  $ systemctl disable NetworkManager.service
 
  $ systemctl disable NetworkManager.service
 +
$ chkconfig network on
 +
$ systemctl start network.service
  
 
  $ yum update -y
 
  $ yum update -y
 +
 +
* Install the latest RDO release package:
 
  $ yum install -y <nowiki>https://rdoproject.org/repos/rdo-release.rpm</nowiki>
 
  $ yum install -y <nowiki>https://rdoproject.org/repos/rdo-release.rpm</nowiki>
 +
 +
However, note that, since we want to make sure we are using the OpenStack ''Kilo'' release (the latest, as of writing, is ''Liberty''), it is better to explicitly specify which release we wish to use with the following instead:
 +
$ yum install -y <nowiki>http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm</nowiki>
 +
 
  $ yum install -y openstack-packstack
 
  $ yum install -y openstack-packstack
  
Line 45: Line 54:
 
  ONBOOT=yes
 
  ONBOOT=yes
 
  EOF
 
  EOF
 +
 +
* Modify the following config parameters to define a logical name for our external physical L2 segment, as "extnet" (this will be referenced as a provider network when we create the external networks below) and support more than just a vxlan driver (since we will be using a "flat" driver for this deploy):
  
 
  $ openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings extnet:br-ex
 
  $ openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings extnet:br-ex
 
  $ openstack-config --set /etc/neutron/plugin.ini ml2 type_drivers vxlan,flat,vlan
 
  $ openstack-config --set /etc/neutron/plugin.ini ml2 type_drivers vxlan,flat,vlan
  
 +
* Restart all related network services:
 
  $ service network restart
 
  $ service network restart
 
  $ service neutron-openvswitch-agent restart
 
  $ service neutron-openvswitch-agent restart
Line 55: Line 67:
 
* Bug fix:
 
* Bug fix:
 
  $ ovs-vsctl br-set-external-id br-ex bridge-id br-ex
 
  $ ovs-vsctl br-set-external-id br-ex bridge-id br-ex
−
  $ service neutron-plugin-openvswitch-agent restart
+
  $ service neutron-openvswitch-agent restart
 +
$ neutron agent-list
 +
$ nova service-list
  
 +
* The following credentials, endpoints, etc. will be used for all OpenStack utilities (e.g., <code>`nova`</code>, <code>`neutron`</code>, etc.) or API calls for the remainder of this article section:
 
  $ cat keystonerc_admin  
 
  $ cat keystonerc_admin  
 
  unset OS_SERVICE_TOKEN
 
  unset OS_SERVICE_TOKEN
Line 74: Line 89:
 
                       --provider:physical_network extnet \
 
                       --provider:physical_network extnet \
 
                       --router:external \
 
                       --router:external \
−
                       --shared
+
                       --shared \
 
                       external_network
 
                       external_network
 
  $ neutron subnet-create --name public_subnet \
 
  $ neutron subnet-create --name public_subnet \
Line 82: Line 97:
 
  $ neutron net-create private_network
 
  $ neutron net-create private_network
 
  $ neutron subnet-create --name private_subnet \
 
  $ neutron subnet-create --name private_subnet \
−
                         --allocation-pool start=10.10.1.100,end=10.10.1.200
+
                         --allocation-pool start=10.10.1.100,end=10.10.1.200 \
 
                         --gateway=10.10.1.1 private_network 10.10.1.0/24
 
                         --gateway=10.10.1.1 private_network 10.10.1.0/24
 
  $ neutron router-create router1
 
  $ neutron router-create router1
Line 92: Line 107:
 
  $ keystone user-create --name demo --tenant demo --pass "password" --email demo@example.com --enabled true
 
  $ keystone user-create --name demo --tenant demo --pass "password" --email demo@example.com --enabled true
  
−
* Populate glance with initial image:
+
* Populate glance with an initial image:
 
  $ CIRROS_IMAGE_NAME=cirros-0.3.4-x86_64
 
  $ CIRROS_IMAGE_NAME=cirros-0.3.4-x86_64
 
  $ CIRROS_IMAGE_URL="<nowiki>http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img</nowiki>"
 
  $ CIRROS_IMAGE_URL="<nowiki>http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img</nowiki>"
Line 109: Line 124:
 
  $ nova secgroup-add-rule base TCP 80 80 0.0.0.0/0
 
  $ nova secgroup-add-rule base TCP 80 80 0.0.0.0/0
 
  $ nova secgroup-add-rule base ICMP -1 -1 0.0.0.0/0
 
  $ nova secgroup-add-rule base ICMP -1 -1 0.0.0.0/0
 +
$ nova secgroup-list-rules base
 +
 +
* Create keypair:
 +
$ nova keypair-add admin >/root/admin.pem
 +
$ chmod 600 /root/admin.pem
 +
$ nova keypair-list
  
 
* Create a very small ("nano") flavor for use in testing (spins up faster, uses less resources, etc.):
 
* Create a very small ("nano") flavor for use in testing (spins up faster, uses less resources, etc.):
Line 116: Line 137:
 
* Setup environment variables in order to keep track of UUIDs, etc.:
 
* Setup environment variables in order to keep track of UUIDs, etc.:
 
  $ INSTANCE_NAME=rdo-test-01
 
  $ INSTANCE_NAME=rdo-test-01
−
  $ GLANCE_IMAGE_ID=$(glance image-list | \grep ${CIRROS_IMAGE_NAME} | awk '{print $2}')
+
  $ GLANCE_IMAGE_ID=$(glance image-list|awk '/ '${CIRROS_IMAGE_NAME}' /{print $2}')
−
  $ PRIVATE_NET_ID=$(neutron net-list | \grep private_network | awk '{print $2}')
+
  $ PRIVATE_NET_ID=$(neutron net-list|awk '/ private_network /{print $2}')
  
 
* Spin up a nova instance (VM):
 
* Spin up a nova instance (VM):
 
  $ nova boot --flavor m1.nano --image ${GLANCE_IMAGE_ID} --nic net-id=${PRIVATE_NET_ID} \
 
  $ nova boot --flavor m1.nano --image ${GLANCE_IMAGE_ID} --nic net-id=${PRIVATE_NET_ID} \
 
             --key-name admin --security-groups base ${INSTANCE_NAME}
 
             --key-name admin --security-groups base ${INSTANCE_NAME}
−
  $ INSTANCE_ID=$(nova list | \grep ${INSTANCE_NAME} | awk '{print $2}')
+
  $ INSTANCE_ID=$(nova list|awk '/ '${INSTANCE_NAME}' /{print $2}')
  
 
* Associate a floating IP with the new instance (this "floating IP" is how the instance communicates with the Internet):
 
* Associate a floating IP with the new instance (this "floating IP" is how the instance communicates with the Internet):
Line 128: Line 149:
 
  $ FLOATINGIP_ID=
 
  $ FLOATINGIP_ID=
 
  $ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
 
  $ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
−
                             --device_id ${INSTANCE_ID} | \grep compute | awk '{print $2}')
+
                             --device_id ${INSTANCE_ID}|awk '/compute.*/{print $2}')
 
  $ neutron floatingip-associate ${FLOATINGIP_ID} ${NEUTRON_COMPUTE_PORT_ID}
 
  $ neutron floatingip-associate ${FLOATINGIP_ID} ${NEUTRON_COMPUTE_PORT_ID}
 
  $ neutron floatingip-show ${FLOATINGIP_ID}
 
  $ neutron floatingip-show ${FLOATINGIP_ID}
 +
$ neutron floatingip-list
 +
$ nova floating-ip-list
 +
 +
* Log into instance:
 +
$ ssh -i /root/admin.pem cirros@10.1.100.x # use associated floating IP
 +
 +
===Troubleshooting===
 +
* Make sure the "<code>openvswitch</code>" kernel module is installed and configured properly:
 +
$ lsmod | grep openvswitch
 +
$ modinfo openvswitch
 +
 +
* Poke around your Open vSwitch setup:
 +
$ ovs-vsctl show
 +
$ ovs-vsctl list-br
 +
$ ovs-vsctl list-ports br-ex # => e.g., em1, phy-br-ex, qg-20ffa8ce-2f
 +
$ ovs-ofctl dump-flows br-ex
 +
$ ovs-ofctl dump-ports br-ex
 +
$ ovs-ofctl show br-int
 +
$ ovs-vsctl list bridge
 +
$ ovs-vsctl list port
 +
$ ovs-vsctl list interface
 +
 +
* Show ethernet (MAC) addresses the Forwarding DataBase (fdb) knows about for a given bridge (e.g., "<code>br-int</code>"):
 +
$ ovs-appctl fdb/show br-int
 +
 +
* Look through ovs db logs to see what/how/when ports, bridges, interfaces, etc. were created for a given instance:
 +
$ ovsdb-tool show-log | grep ${INSTANCE_ID}
 +
 +
# example output:
 +
record 467: 2015-10-06 23:37:05.589
 +
  "ovs-vsctl: /bin/ovs-vsctl --timeout=120 \
 +
      -- --if-exists del-port qvo55bfa719-2b \
 +
      -- add-port br-int qvo55bfa719-2b \
 +
      -- set Interface qvo55bfa719-2b \
 +
        external-ids:iface-id=55bfa719-2b39-4793-823e-4a79405d9256 \
 +
        external-ids:iface-status=active \
 +
        external-ids:attached-mac=fa:16:3e:b8:dc:e2 \
 +
        external-ids:vm-uuid=0e6c8821-22f3-4c6a-95b9-0fbf855e82e0"
 +
 +
$ brctl show
 +
$ brctl showmacs qbrc648c3ca-76
 +
$ ps afux|grep [d]nsmasq
 +
$ cat /proc/$(pidof dnsmasq)/cmdline | tr '\0' '\n'
 +
$ ps afux|grep qemu
 +
 +
$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
 +
$ ip netns exec ${QROUTER} ip a
 +
$ ip netns exec ${QROUTER} router
 +
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
 +
$ ip netns exec ${QROUTER} iptables -S -t nat
 +
 +
* Work inside the DHCP network namespace:
 +
$ ip netns identify $(pidof dnsmasq)
 +
#~OR~
 +
$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
 +
$ ip netns exec ${QDHCP} ip r
 +
 +
* Log into instance via the DHCP network namespace (using the private {non-floating IP} network IP assigned to the instance):
 +
$ ip netns exec ${QDHCP} ssh cirros@10.10.1.x # use actual IP; password = "cubswin:)"
 +
 +
* Check host's iptables rules related to our instance:
 +
$ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
 +
                            --device_id ${INSTANCE_ID} | \grep compute | awk '{print $2}')
 +
$ iptables -L neutron-openvswi-i$(echo ${NEUTRON_COMPUTE_PORT_ID} |awk '{print substr($0,1,10)}')
 +
 +
# example output (trimmed):
 +
RETURN    tcp  --  anywhere            anywhere            tcp dpt:http
 +
RETURN    tcp  --  anywhere            anywhere            tcp dpt:ssh
 +
RETURN    icmp --  anywhere            anywhere
 +
 +
The above rules should match our Nova secgroup "base" rules we defined previously. I.e.,
 +
$ nova secgroup-list-rules base
 +
+-------------+-----------+---------+-----------+--------------+
 +
| IP Protocol | From Port | To Port | IP Range  | Source Group |
 +
+-------------+-----------+---------+-----------+--------------+
 +
| tcp        | 80        | 80      | 0.0.0.0/0 |              |
 +
| tcp        | 22        | 22      | 0.0.0.0/0 |              |
 +
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
 +
+-------------+-----------+---------+-----------+--------------+
 +
 +
* Capture traffic to/from host and/or instance:
 +
$ tcpdump -nni eth0 \( dst host 10.0.0.19 and port 22 \)
 +
$ tcpdump -nni eth0 icmp
 +
$ tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
 +
 +
* See what virsh knows about our instance:
 +
$ VIRSH_INSTANCE_ID=$(nova show ${INSTANCE_ID} |\grep OS-EXT-SRV-ATTR:instance_name|awk '{print $4}')
 +
$ virsh dumpxml instance-00000003 | grep -A3 bridge
 +
$ grep -A4 'type="bridge"' /var/lib/nova/instances/${INSTANCE_ID}/libvirt.xml
 +
$ xmllint --xpath '//devices/interface[@type="bridge"]/source[@bridge]' libvirt.xml
 +
 +
===Open vSwitch (OVS)===
 +
 +
<pre>
 +
                +--------------------------------------+
 +
                |                cirros                |
 +
                |              +------+              |
 +
                |              | eth0 | (10.10.1.102) |
 +
                +---------------+------+---------------+
 +
                                    |
 +
                                    |
 +
                          +-----------------+
 +
Virtual NIC (tap)          | tap-55bfa719-2b |
 +
                          +-----------------+
 +
                                    |
 +
                                    |
 +
                          +-----------------+
 +
Linux bridge              | qbr-55bfa719-2b |
 +
                          +-----------------+
 +
                                    |
 +
                                    |
 +
                          +-----------------+
 +
                          | qvb-55bfa719-2b |
 +
                          +-----------------+
 +
veth pair                          |
 +
                          +-----------------+
 +
                          | qvo-55bfa719-2b |
 +
          +---------------+-----------------+--------------+
 +
OVS Bridge |    10.10.1.0/24    br-int                    |
 +
          +--+----------------+-----------+----------------+
 +
            / | qr-b24d8155-69 | \      / | tap0d8a8773-84 | \
 +
          /  +----------------+  \    /  +----------------+  \
 +
          /                        \  /                        \
 +
          |        qrouter-        |  |      Namespace:        |
 +
          |        9f36....        |  |        qdhcp-          |
 +
          |                        |  |                        |
 +
          |      10.1.100.16      |    \______________________/
 +
          \  +----------------+  /
 +
          \  | qg-20ffa8ce-2f |  /
 +
          +--+----------------+--+     
 +
          |    10.1.100.0/23    |
 +
OVS Bridge |        br-ex        |
 +
          +----------------------+
 +
</pre>
 +
 +
The names for the OVS ports and interfaces are made from a prefix (qr-, qg-, or tap) followed by the first 11 chars of the Neutron port they represent.
 +
 +
* Inspect the ethernet bridge ("Linux bridge") configuration in the Linux kernel:
 +
$ brctl show
 +
bridge name    bridge id          STP enabled  interfaces
 +
qbr55bfa719-2b  8000.fe163e32aa55    no        qvb55bfa719-2b
 +
                                                tap55bfa719-2b
 +
 +
$ brctl showmacs qbr55bfa719-2b
 +
port no  mac addr          is local?  ageing timer
 +
  2      fe:16:3e:32:aa:55    yes          0.00
 +
  2      fe:16:3e:32:aa:55    yes          0.00
 +
  1      fe:be:6e:86:cc:9f    yes          0.00
 +
  1      fe:be:6e:86:cc:9f    yes          0.00
 +
 +
The "<code>bridge id</code>" ("<code>8000.fe163e32aa55</code>") should match one of the MAC addresses listed above. That is,
 +
8000.fe163e32aa55 => fe:16:3e:32:aa:55 (on port #2)
 +
 +
Every bridge has an id associated with it; this is an eight-byte number, the first two bytes being the bridge priority, which we can set manually, and the next six bytes are the MAC address of the bridge. Under Linux, the default bridge priority is 32768 (<code>`printf '%x\n' 32768` #=> 8000</code>). The bridge's MAC address is that of the lowest numbered MAC address of all the bridge's ports. We generally represent the bridge ID as a two part hexadecimal number, the bridge ID followed by the MAC address as the fractional part. For example, <code>8000.fe163e32aa55</code> is the ID of a bridge with a priority of 32768 (8000 hex) and a MAC address of <code>fe:16:3e:32:aa:55</code>.
 +
 +
* Get a list of network namespaces:
 +
$ ip netns list
 +
qrouter-9f3646b2-cfa0-435b-9708-5854656b797f # router1
 +
qdhcp-7f8821c4-7f8a-4a48-ac17-15f27a32a60c  # private_network
 +
 +
$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
 +
$ ip netns exec ${QROUTER} ip a
 +
$ ip netns exec ${QROUTER} route -n
 +
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
 +
$ ip netns exec ${QROUTER} iptables -S -t nat
 +
 +
$ ip netns exec ${QROUTER} ip r
 +
default via 10.1.100.1 dev qg-20ffa8ce-2f
 +
10.1.100.0/23 dev qg-20ffa8ce-2f  proto kernel  scope link  src 10.1.100.16
 +
10.10.1.0/24 dev qr-b24d8155-69  proto kernel  scope link  src 10.10.1.1
 +
 +
* Check interfaces configured within the network namespace:
 +
$ ip netns exec ${QROUTER} ip a|grep -E 'state|inet'
 +
...
 +
32: qr-b24d8155-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
 +
    inet 10.10.1.1/24 brd 10.10.1.255 scope global qr-b24d8155-69
 +
36: qg-20ffa8ce-2f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
 +
    inet 10.1.100.16/23 brd 10.1.101.255 scope global qg-20ffa8ce-2f
 +
    inet 10.1.100.19/32 brd 10.1.100.19 scope global qg-20ffa8ce-2f
 +
 +
The agent has bound 10.10.1.1 (the <code>gateway_ip</code> of our <code>private_subnet</code>) to <code>qr-b24d8155-69</code> (<code>router1</code>'s interface for <code>private_subnet</code>). DHCP pushes this IP to instances connected to <code>private_subnet</code> for their default gateway.
 +
 +
The agent has also bound IP addresses 10.1.100.16 (for gateway SNAT) and 10.1.100.19 (the floating IP) to the gateway interface <code>qg-20ffa8ce-2f</code>.
 +
 +
* Verify agent has also enabled IP forwarding within the namespace:
 +
$ ip netns exec ${QROUTER} sysctl net.ipv4.ip_forward
 +
net.ipv4.ip_forward = 1
 +
 +
* Look at the Network Address Translation (NAT) rules within the network namespace:
 +
$ ip netns exec ${QROUTER} iptables-save|grep -i nat
 +
...
 +
-A neutron-l3-agent-OUTPUT -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
 +
-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
 +
-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19
 +
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
 +
-A neutron-l3-agent-snat -o qg-20ffa8ce-2f -j SNAT --to-source 10.1.100.16
 +
 +
Thus, a connection initiated from within instance to the outside will hit this rule in the NAT table (post routing):
 +
-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19
 +
 +
And connection initiated from the outside to the instance's floating IP will hit this rule:
 +
-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
 +
 +
$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
 +
$ ip netns exec ${QDHCP} ip r
 +
 +
$ grep tap /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/interface
 +
tap0d8a8773-84
 +
 +
$ cat /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/opts
 +
tag:tag0,option:router,10.10.1.1
 +
tag:tag0,option:dns-server,10.10.1.100
 +
 +
cirros$ route -n
 +
Kernel IP routing table
 +
Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
 +
0.0.0.0        10.10.1.1      0.0.0.0        UG    0      0        0 eth0
 +
10.10.1.0      0.0.0.0        255.255.255.0  U    0      0        0 eth0
 +
 +
cirros$ cat /etc/resolv.conf
 +
search openstacklocal
 +
nameserver 8.8.8.8
  
 +
===Metadata===
 
* Direct access to Nova metadata:
 
* Direct access to Nova metadata:
 
:see: [http://blog.oddbit.com/2014/01/14/direct-access-to-nova-metadata/ for details]
 
:see: [http://blog.oddbit.com/2014/01/14/direct-access-to-nova-metadata/ for details]
 +
Note: You need to have the "hmac" and "hashlib" Python modules installed (they are by default on most distros). You also need to have [[crudini]] installed (or you could use regex with grep, [[sed]], [[awk]], etc.).
 
  $ SHARED_SECRET=$(crudini --get /etc/nova/nova.conf neutron metadata_proxy_shared_secret)
 
  $ SHARED_SECRET=$(crudini --get /etc/nova/nova.conf neutron metadata_proxy_shared_secret)
 
  $ META_SIGNATURE=$(python -c 'import hmac,hashlib;print hmac.new("'${SHARED_SECRET}'",\
 
  $ META_SIGNATURE=$(python -c 'import hmac,hashlib;print hmac.new("'${SHARED_SECRET}'",\
 
                               "'${INSTANCE_ID}'",hashlib.sha256).hexdigest()')
 
                               "'${INSTANCE_ID}'",hashlib.sha256).hexdigest()')
−
  $ ADMIN_TENANT_ID=$(keystone tenant-list | \grep admin | awk '{print $2}')
+
  $ ADMIN_TENANT_ID=$(keystone tenant-list | awk '/admin/{print $2}')
 
  $ ENDPOINT=<nowiki>http://10.1.100.15:8775</nowiki>
 
  $ ENDPOINT=<nowiki>http://10.1.100.15:8775</nowiki>
 
  $ curl -s -H "x-instance-id:${INSTANCE_ID}" \
 
  $ curl -s -H "x-instance-id:${INSTANCE_ID}" \
Line 164: Line 409:
 
security-groups
 
security-groups
 
</pre>
 
</pre>
 +
 +
==See also==
 +
* [[OpenStack deployment with vagrant]]
  
 
==External links==
 
==External links==
 +
* [https://www.rdoproject.org/ RDO project]
 
* [https://github.com/christophchamp/xtof-openstack-rdo-packstack xtof-openstack-rdo-packstack] on GitHub
 
* [https://github.com/christophchamp/xtof-openstack-rdo-packstack xtof-openstack-rdo-packstack] on GitHub
  
 
[[Category:OpenStack]]
 
[[Category:OpenStack]]

Latest revision as of 21:41, 8 April 2016

This article will cover the steps involved in getting OpenStack deployed using "packstack" from RDO.

WARNING: This article is very much under construction. Proceed with caution until this banner has been removed.
  • Deploy assumptions:
    • OS: CentOS 7.1 (64-bit; 7.1.1503 Core)
    • OpenStack release: "Kilo" (April 2015)
    • RDO release: rdo-release-kilo-1 (noarch; 12 May 2015)

Single node

Note: Using neutron with a "flat" network driver. 

$ sysctl -a | grep ip_forward #=> 1 (see: sysctl)
$ sestatus #=> set to "permissive" (see: selinux)
$ systemctl stop NetworkManager.service
$ systemctl disable NetworkManager.service
$ chkconfig network on
$ systemctl start network.service

$ yum update -y
  • Install the latest RDO release package:
$ yum install -y https://rdoproject.org/repos/rdo-release.rpm

However, note that, since we want to make sure we are using the OpenStack Kilo release (the latest, as of writing, is Liberty), it is better to explicitly specify which release we wish to use with the following instead: 

$ yum install -y http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm

$ yum install -y openstack-packstack

$ packstack --allinone --provision-demo=n

$ cp /etc/sysconfig/network-scripts/ifcfg-eth0 /root/ # backup
$ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=FF:FF:FF:FF:FF:FF
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-ex
ONBOOT=yes
EOF

$ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-br-ex 
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
MACADDR=FF:FF:FF:FF:FF:FF
BOOTPROTO=static
IPADDR=10.1.100.15
#PREFIX=23
NETMASK=255.255.254.0
GATEWAY=10.1.100.1
DNS1=8.8.8.8
DNS2=8.8.4.4
ONBOOT=yes
EOF
  • Modify the following config parameters to define a logical name for our external physical L2 segment, as "extnet" (this will be referenced as a provider network when we create the external networks below) and support more than just a vxlan driver (since we will be using a "flat" driver for this deploy):
$ openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings extnet:br-ex
$ openstack-config --set /etc/neutron/plugin.ini ml2 type_drivers vxlan,flat,vlan
  • Restart all related network services:
$ service network restart
$ service neutron-openvswitch-agent restart
$ service neutron-server restart
  • Bug fix:
$ ovs-vsctl br-set-external-id br-ex bridge-id br-ex
$ service neutron-openvswitch-agent restart
$ neutron agent-list
$ nova service-list
  • The following credentials, endpoints, etc. will be used for all OpenStack utilities (e.g., `nova`, `neutron`, etc.) or API calls for the remainder of this article section:
$ cat keystonerc_admin 
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=<password>
export OS_AUTH_URL=http://10.1.100.15:5000/v2.0
export PS1='[\u@\h \W(keystone_admin)]\$ '

export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne

$ . keystonerc_admin # source the admin environment
  • Setup networks:
$ neutron net-create --provider:network_type flat \
                     --provider:physical_network extnet \
                     --router:external \
                     --shared \
                     external_network
$ neutron subnet-create --name public_subnet \
                        --enable_dhcp=False \
                        --allocation-pool start=10.1.100.16,end=10.1.100.20 \
                        --gateway=10.1.100.1 external_network 10.1.100.0/23
$ neutron net-create private_network
$ neutron subnet-create --name private_subnet \
                        --allocation-pool start=10.10.1.100,end=10.10.1.200 \
                        --gateway=10.10.1.1 private_network 10.10.1.0/24
$ neutron router-create router1
$ neutron router-interface-add router1 private_subnet
$ neutron router-gateway-set router1 external_network
  • Create new (non-admin) tenant and user:
$ keystone tenant-create --name demo --description "demo tenant" --enabled true
$ keystone user-create --name demo --tenant demo --pass "password" --email demo@example.com --enabled true
  • Populate glance with an initial image:
$ CIRROS_IMAGE_NAME=cirros-0.3.4-x86_64
$ CIRROS_IMAGE_URL="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img"
$ curl ${CIRROS_IMAGE_URL} | \
    glance image-create --name="${CIRROS_IMAGE_NAME}" \
                        --is-public=true \
                        --container-format=bare \
                        --disk-format=qcow2
$ glance image-list
  • Create basic security groups/rules (to allow basic networking traffic in/out of VMs):
$ nova secgroup-create all "Allow all tcp ports"
$ nova secgroup-add-rule all TCP 1 65535 0.0.0.0/0
$ nova secgroup-create base "Allow Base Access"
$ nova secgroup-add-rule base TCP 22 22 0.0.0.0/0
$ nova secgroup-add-rule base TCP 80 80 0.0.0.0/0
$ nova secgroup-add-rule base ICMP -1 -1 0.0.0.0/0
$ nova secgroup-list-rules base
  • Create keypair:
$ nova keypair-add admin >/root/admin.pem
$ chmod 600 /root/admin.pem
$ nova keypair-list
  • Create a very small ("nano") flavor for use in testing (spins up faster, uses less resources, etc.):
$ nova flavor-create m1.nano 42 64 0 1 # <name> <id> <ram> <disk> <vcpus>
$ nova flavor-list
  • Setup environment variables in order to keep track of UUIDs, etc.:
$ INSTANCE_NAME=rdo-test-01
$ GLANCE_IMAGE_ID=$(glance image-list|awk '/ '${CIRROS_IMAGE_NAME}' /{print $2}')
$ PRIVATE_NET_ID=$(neutron net-list|awk '/ private_network /{print $2}')
  • Spin up a nova instance (VM):
$ nova boot --flavor m1.nano --image ${GLANCE_IMAGE_ID} --nic net-id=${PRIVATE_NET_ID} \
            --key-name admin --security-groups base ${INSTANCE_NAME}
$ INSTANCE_ID=$(nova list|awk '/ '${INSTANCE_NAME}' /{print $2}')
  • Associate a floating IP with the new instance (this "floating IP" is how the instance communicates with the Internet):
$ neutron floatingip-create external_network
$ FLOATINGIP_ID=
$ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
                            --device_id ${INSTANCE_ID}|awk '/compute.*/{print $2}')
$ neutron floatingip-associate ${FLOATINGIP_ID} ${NEUTRON_COMPUTE_PORT_ID}
$ neutron floatingip-show ${FLOATINGIP_ID}
$ neutron floatingip-list
$ nova floating-ip-list
  • Log into instance:
$ ssh -i /root/admin.pem cirros@10.1.100.x # use associated floating IP

Troubleshooting

  • Make sure the "openvswitch" kernel module is installed and configured properly:
$ lsmod | grep openvswitch
$ modinfo openvswitch
  • Poke around your Open vSwitch setup:
$ ovs-vsctl show
$ ovs-vsctl list-br
$ ovs-vsctl list-ports br-ex # => e.g., em1, phy-br-ex, qg-20ffa8ce-2f
$ ovs-ofctl dump-flows br-ex
$ ovs-ofctl dump-ports br-ex
$ ovs-ofctl show br-int
$ ovs-vsctl list bridge
$ ovs-vsctl list port
$ ovs-vsctl list interface
  • Show ethernet (MAC) addresses the Forwarding DataBase (fdb) knows about for a given bridge (e.g., "br-int"):
$ ovs-appctl fdb/show br-int
  • Look through ovs db logs to see what/how/when ports, bridges, interfaces, etc. were created for a given instance:
$ ovsdb-tool show-log | grep ${INSTANCE_ID}

# example output:
record 467: 2015-10-06 23:37:05.589 
  "ovs-vsctl: /bin/ovs-vsctl --timeout=120 \
     -- --if-exists del-port qvo55bfa719-2b \
     -- add-port br-int qvo55bfa719-2b \
     -- set Interface qvo55bfa719-2b \
       external-ids:iface-id=55bfa719-2b39-4793-823e-4a79405d9256 \
       external-ids:iface-status=active \
       external-ids:attached-mac=fa:16:3e:b8:dc:e2 \
       external-ids:vm-uuid=0e6c8821-22f3-4c6a-95b9-0fbf855e82e0"

$ brctl show
$ brctl showmacs qbrc648c3ca-76
$ ps afux|grep [d]nsmasq
$ cat /proc/$(pidof dnsmasq)/cmdline | tr '\0' '\n'
$ ps afux|grep qemu

$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
$ ip netns exec ${QROUTER} ip a
$ ip netns exec ${QROUTER} router
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
$ ip netns exec ${QROUTER} iptables -S -t nat
  • Work inside the DHCP network namespace:
$ ip netns identify $(pidof dnsmasq)
#~OR~
$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
$ ip netns exec ${QDHCP} ip r
  • Log into instance via the DHCP network namespace (using the private {non-floating IP} network IP assigned to the instance):
$ ip netns exec ${QDHCP} ssh cirros@10.10.1.x # use actual IP; password = "cubswin:)"
  • Check host's iptables rules related to our instance:
$ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
                            --device_id ${INSTANCE_ID} | \grep compute | awk '{print $2}')
$ iptables -L neutron-openvswi-i$(echo ${NEUTRON_COMPUTE_PORT_ID} |awk '{print substr($0,1,10)}')

# example output (trimmed):
RETURN     tcp  --  anywhere             anywhere             tcp dpt:http
RETURN     tcp  --  anywhere             anywhere             tcp dpt:ssh
RETURN     icmp --  anywhere             anywhere

The above rules should match our Nova secgroup "base" rules we defined previously. I.e., 

$ nova secgroup-list-rules base
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp         | 80        | 80      | 0.0.0.0/0 |              |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+
  • Capture traffic to/from host and/or instance:
$ tcpdump -nni eth0 \( dst host 10.0.0.19 and port 22 \)
$ tcpdump -nni eth0 icmp
$ tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
  • See what virsh knows about our instance:
$ VIRSH_INSTANCE_ID=$(nova show ${INSTANCE_ID} |\grep OS-EXT-SRV-ATTR:instance_name|awk '{print $4}')
$ virsh dumpxml instance-00000003 | grep -A3 bridge
$ grep -A4 'type="bridge"' /var/lib/nova/instances/${INSTANCE_ID}/libvirt.xml
$ xmllint --xpath '//devices/interface[@type="bridge"]/source[@bridge]' libvirt.xml

Open vSwitch (OVS)

                +--------------------------------------+
                |                cirros                |
                |               +------+               |
                |               | eth0 | (10.10.1.102) |
                +---------------+------+---------------+
                                    |
                                    |
                           +-----------------+
Virtual NIC (tap)          | tap-55bfa719-2b |
                           +-----------------+
                                    |
                                    |
                           +-----------------+
Linux bridge               | qbr-55bfa719-2b |
                           +-----------------+
                                    |
                                    |
                           +-----------------+
                           | qvb-55bfa719-2b |
                           +-----------------+
veth pair                           |
                           +-----------------+
                           | qvo-55bfa719-2b |
           +---------------+-----------------+--------------+
OVS Bridge |     10.10.1.0/24     br-int                    |
           +--+----------------+-----------+----------------+
            / | qr-b24d8155-69 | \       / | tap0d8a8773-84 | \
           /  +----------------+  \     /  +----------------+  \
          /                        \   /                        \
          |        qrouter-        |   |      Namespace:        |
          |        9f36....        |   |        qdhcp-          |
          |                        |   |                        |
          |      10.1.100.16       |    \______________________/
          \   +----------------+   /
           \  | qg-20ffa8ce-2f |  /
           +--+----------------+--+       
           |    10.1.100.0/23     |
OVS Bridge |        br-ex         |
           +----------------------+

The names for the OVS ports and interfaces are made from a prefix (qr-, qg-, or tap) followed by the first 11 chars of the Neutron port they represent.

  • Inspect the ethernet bridge ("Linux bridge") configuration in the Linux kernel:
$ brctl show
bridge name     bridge id          STP enabled  interfaces
qbr55bfa719-2b  8000.fe163e32aa55     no        qvb55bfa719-2b
                                                tap55bfa719-2b

$ brctl showmacs qbr55bfa719-2b
port no  mac addr           is local?  ageing timer
  2      fe:16:3e:32:aa:55    yes          0.00
  2      fe:16:3e:32:aa:55    yes          0.00
  1      fe:be:6e:86:cc:9f    yes          0.00
  1      fe:be:6e:86:cc:9f    yes          0.00

The "bridge id" ("8000.fe163e32aa55") should match one of the MAC addresses listed above. That is, 

8000.fe163e32aa55 => fe:16:3e:32:aa:55 (on port #2)

Every bridge has an id associated with it; this is an eight-byte number, the first two bytes being the bridge priority, which we can set manually, and the next six bytes are the MAC address of the bridge. Under Linux, the default bridge priority is 32768 (`printf '%x\n' 32768` #=> 8000). The bridge's MAC address is that of the lowest numbered MAC address of all the bridge's ports. We generally represent the bridge ID as a two part hexadecimal number, the bridge ID followed by the MAC address as the fractional part. For example, 8000.fe163e32aa55 is the ID of a bridge with a priority of 32768 (8000 hex) and a MAC address of fe:16:3e:32:aa:55.

  • Get a list of network namespaces:
$ ip netns list
qrouter-9f3646b2-cfa0-435b-9708-5854656b797f # router1
qdhcp-7f8821c4-7f8a-4a48-ac17-15f27a32a60c   # private_network

$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
$ ip netns exec ${QROUTER} ip a
$ ip netns exec ${QROUTER} route -n
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
$ ip netns exec ${QROUTER} iptables -S -t nat

$ ip netns exec ${QROUTER} ip r
default via 10.1.100.1 dev qg-20ffa8ce-2f 
10.1.100.0/23 dev qg-20ffa8ce-2f  proto kernel  scope link  src 10.1.100.16 
10.10.1.0/24 dev qr-b24d8155-69  proto kernel  scope link  src 10.10.1.1
  • Check interfaces configured within the network namespace:
$ ip netns exec ${QROUTER} ip a|grep -E 'state|inet'
...
32: qr-b24d8155-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    inet 10.10.1.1/24 brd 10.10.1.255 scope global qr-b24d8155-69
36: qg-20ffa8ce-2f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    inet 10.1.100.16/23 brd 10.1.101.255 scope global qg-20ffa8ce-2f
    inet 10.1.100.19/32 brd 10.1.100.19 scope global qg-20ffa8ce-2f

The agent has bound 10.10.1.1 (the gateway_ip of our private_subnet) to qr-b24d8155-69 (router1's interface for private_subnet). DHCP pushes this IP to instances connected to private_subnet for their default gateway.

The agent has also bound IP addresses 10.1.100.16 (for gateway SNAT) and 10.1.100.19 (the floating IP) to the gateway interface qg-20ffa8ce-2f.

  • Verify agent has also enabled IP forwarding within the namespace:
$ ip netns exec ${QROUTER} sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
  • Look at the Network Address Translation (NAT) rules within the network namespace:
$ ip netns exec ${QROUTER} iptables-save|grep -i nat
...
-A neutron-l3-agent-OUTPUT -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-20ffa8ce-2f -j SNAT --to-source 10.1.100.16

Thus, a connection initiated from within instance to the outside will hit this rule in the NAT table (post routing): 

-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19

And connection initiated from the outside to the instance's floating IP will hit this rule: 

-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102

$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
$ ip netns exec ${QDHCP} ip r

$ grep tap /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/interface
tap0d8a8773-84

$ cat /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/opts
tag:tag0,option:router,10.10.1.1
tag:tag0,option:dns-server,10.10.1.100

cirros$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.1.1       0.0.0.0         UG    0      0        0 eth0
10.10.1.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0

cirros$ cat /etc/resolv.conf 
search openstacklocal
nameserver 8.8.8.8

Metadata

  • Direct access to Nova metadata:
see: for details

Note: You need to have the "hmac" and "hashlib" Python modules installed (they are by default on most distros). You also need to have crudini installed (or you could use regex with grep, sed, awk, etc.). 

$ SHARED_SECRET=$(crudini --get /etc/nova/nova.conf neutron metadata_proxy_shared_secret)
$ META_SIGNATURE=$(python -c 'import hmac,hashlib;print hmac.new("'${SHARED_SECRET}'",\
                             "'${INSTANCE_ID}'",hashlib.sha256).hexdigest()')
$ ADMIN_TENANT_ID=$(keystone tenant-list | awk '/admin/{print $2}')
$ ENDPOINT=http://10.1.100.15:8775
$ curl -s -H "x-instance-id:${INSTANCE_ID}" \
          -H "x-tenant-id:${ADMIN_TENANT_ID}" \
          -H "x-instance-id-signature:${META_SIGNATURE}" \
          ${ENDPOINT}/latest/meta-data

# RESPONSE:
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

See also

External links

Retrieved from "http://wiki.christophchamp.com/index.php?title=OpenStack_deployment_via_packstack_from_RDO&oldid=6552"