OpenStack deployment via packstack from RDO
From Christoph's Personal Wiki
This article will cover the steps involved in getting OpenStack deployed using "packstack" from RDO.
WARNING: This article is very much under construction. Proceed with caution until this banner has been removed.
- Deploy assumptions:
- OS: CentOS 7.1 (64-bit; 7.1.1503 Core)
- OpenStack release: "Kilo" (April 2015)
- RDO release: rdo-release-kilo-1 (noarch; 12 May 2015)
Contents
Single node
Note: Using neutron with a "flat" network driver.
$ sysctl -a | grep ip_forward #=> 1 (see: sysctl) $ sestatus #=> set to "permissive" (see: selinux) $ systemctl stop NetworkManager.service $ systemctl disable NetworkManager.service $ chkconfig network on $ systemctl start network.service
$ yum update -y
- Install the latest RDO release package:
$ yum install -y https://rdoproject.org/repos/rdo-release.rpm
However, note that, since we want to make sure we are using the OpenStack Kilo release (the latest, as of writing, is Liberty), it is better to explicitly specify which release we wish to use with the following instead:
$ yum install -y http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm
$ yum install -y openstack-packstack
$ packstack --allinone --provision-demo=n
$ cp /etc/sysconfig/network-scripts/ifcfg-eth0 /root/ # backup $ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=FF:FF:FF:FF:FF:FF TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex ONBOOT=yes EOF
$ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-br-ex DEVICE=br-ex DEVICETYPE=ovs TYPE=OVSBridge MACADDR=FF:FF:FF:FF:FF:FF BOOTPROTO=static IPADDR=10.1.100.15 #PREFIX=23 NETMASK=255.255.254.0 GATEWAY=10.1.100.1 DNS1=8.8.8.8 DNS2=8.8.4.4 ONBOOT=yes EOF
- Modify the following config parameters to define a logical name for our external physical L2 segment, as "extnet" (this will be referenced as a provider network when we create the external networks below) and support more than just a vxlan driver (since we will be using a "flat" driver for this deploy):
$ openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings extnet:br-ex $ openstack-config --set /etc/neutron/plugin.ini ml2 type_drivers vxlan,flat,vlan
- Restart all related network services:
$ service network restart $ service neutron-openvswitch-agent restart $ service neutron-server restart
- Bug fix:
$ ovs-vsctl br-set-external-id br-ex bridge-id br-ex $ service neutron-openvswitch-agent restart $ neutron agent-list $ nova service-list
- The following credentials, endpoints, etc. will be used for all OpenStack utilities (e.g.,
`nova`,`neutron`, etc.) or API calls for the remainder of this article section:
$ cat keystonerc_admin unset OS_SERVICE_TOKEN export OS_USERNAME=admin export OS_PASSWORD=<password> export OS_AUTH_URL=http://10.1.100.15:5000/v2.0 export PS1='[\u@\h \W(keystone_admin)]\$ ' export OS_TENANT_NAME=admin export OS_REGION_NAME=RegionOne
$ . keystonerc_admin # source the admin environment
- Setup networks:
$ neutron net-create --provider:network_type flat \
--provider:physical_network extnet \
--router:external \
--shared \
external_network
$ neutron subnet-create --name public_subnet \
--enable_dhcp=False \
--allocation-pool start=10.1.100.16,end=10.1.100.20 \
--gateway=10.1.100.1 external_network 10.1.100.0/23
$ neutron net-create private_network
$ neutron subnet-create --name private_subnet \
--allocation-pool start=10.10.1.100,end=10.10.1.200 \
--gateway=10.10.1.1 private_network 10.10.1.0/24
$ neutron router-create router1
$ neutron router-interface-add router1 private_subnet
$ neutron router-gateway-set router1 external_network
- Create new (non-admin) tenant and user:
$ keystone tenant-create --name demo --description "demo tenant" --enabled true $ keystone user-create --name demo --tenant demo --pass "password" --email demo@example.com --enabled true
- Populate glance with an initial image:
$ CIRROS_IMAGE_NAME=cirros-0.3.4-x86_64
$ CIRROS_IMAGE_URL="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img"
$ curl ${CIRROS_IMAGE_URL} | \
glance image-create --name="${CIRROS_IMAGE_NAME}" \
--is-public=true \
--container-format=bare \
--disk-format=qcow2
$ glance image-list
- Create basic security groups/rules (to allow basic networking traffic in/out of VMs):
$ nova secgroup-create all "Allow all tcp ports" $ nova secgroup-add-rule all TCP 1 65535 0.0.0.0/0 $ nova secgroup-create base "Allow Base Access" $ nova secgroup-add-rule base TCP 22 22 0.0.0.0/0 $ nova secgroup-add-rule base TCP 80 80 0.0.0.0/0 $ nova secgroup-add-rule base ICMP -1 -1 0.0.0.0/0 $ nova secgroup-list-rules base
- Create keypair:
$ nova keypair-add admin >/root/admin.pem $ chmod 600 /root/admin.pem $ nova keypair-list
- Create a very small ("nano") flavor for use in testing (spins up faster, uses less resources, etc.):
$ nova flavor-create m1.nano 42 64 0 1 # <name> <id> <ram> <disk> <vcpus> $ nova flavor-list
- Setup environment variables in order to keep track of UUIDs, etc.:
$ INSTANCE_NAME=rdo-test-01
$ GLANCE_IMAGE_ID=$(glance image-list|awk '/ '${CIRROS_IMAGE_NAME}' /{print $2}')
$ PRIVATE_NET_ID=$(neutron net-list|awk '/ private_network /{print $2}')
- Spin up a nova instance (VM):
$ nova boot --flavor m1.nano --image ${GLANCE_IMAGE_ID} --nic net-id=${PRIVATE_NET_ID} \
--key-name admin --security-groups base ${INSTANCE_NAME}
$ INSTANCE_ID=$(nova list|awk '/ '${INSTANCE_NAME}' /{print $2}')
- Associate a floating IP with the new instance (this "floating IP" is how the instance communicates with the Internet):
$ neutron floatingip-create external_network
$ FLOATINGIP_ID=
$ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
--device_id ${INSTANCE_ID}|awk '/compute.*/{print $2}')
$ neutron floatingip-associate ${FLOATINGIP_ID} ${NEUTRON_COMPUTE_PORT_ID}
$ neutron floatingip-show ${FLOATINGIP_ID}
$ neutron floatingip-list
$ nova floating-ip-list
- Log into instance:
$ ssh -i /root/admin.pem cirros@10.1.100.x # use associated floating IP
Troubleshooting
- Make sure the "
openvswitch" kernel module is installed and configured properly:
$ lsmod | grep openvswitch $ modinfo openvswitch
- Poke around your Open vSwitch setup:
$ ovs-vsctl show $ ovs-vsctl list-br $ ovs-vsctl list-ports br-ex # => e.g., em1, phy-br-ex, qg-20ffa8ce-2f $ ovs-ofctl dump-flows br-ex $ ovs-ofctl dump-ports br-ex $ ovs-ofctl show br-int $ ovs-vsctl list bridge $ ovs-vsctl list port $ ovs-vsctl list interface
- Show ethernet (MAC) addresses the Forwarding DataBase (fdb) knows about for a given bridge (e.g., "
br-int"):
$ ovs-appctl fdb/show br-int
- Look through ovs db logs to see what/how/when ports, bridges, interfaces, etc. were created for a given instance:
$ ovsdb-tool show-log | grep ${INSTANCE_ID}
# example output:
record 467: 2015-10-06 23:37:05.589
"ovs-vsctl: /bin/ovs-vsctl --timeout=120 \
-- --if-exists del-port qvo55bfa719-2b \
-- add-port br-int qvo55bfa719-2b \
-- set Interface qvo55bfa719-2b \
external-ids:iface-id=55bfa719-2b39-4793-823e-4a79405d9256 \
external-ids:iface-status=active \
external-ids:attached-mac=fa:16:3e:b8:dc:e2 \
external-ids:vm-uuid=0e6c8821-22f3-4c6a-95b9-0fbf855e82e0"
$ brctl show $ brctl showmacs qbrc648c3ca-76 $ ps afux|grep [d]nsmasq $ cat /proc/$(pidof dnsmasq)/cmdline | tr '\0' '\n' $ ps afux|grep qemu
$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
$ ip netns exec ${QROUTER} ip a
$ ip netns exec ${QROUTER} router
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
$ ip netns exec ${QROUTER} iptables -S -t nat
- Work inside the DHCP network namespace:
$ ip netns identify $(pidof dnsmasq)
#~OR~
$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
$ ip netns exec ${QDHCP} ip r
- Log into instance via the DHCP network namespace (using the private {non-floating IP} network IP assigned to the instance):
$ ip netns exec ${QDHCP} ssh cirros@10.10.1.x # use actual IP; password = "cubswin:)"
- Check host's iptables rules related to our instance:
$ NEUTRON_COMPUTE_PORT_ID=$(neutron port-list -c id -c device_owner -- \
--device_id ${INSTANCE_ID} | \grep compute | awk '{print $2}')
$ iptables -L neutron-openvswi-i$(echo ${NEUTRON_COMPUTE_PORT_ID} |awk '{print substr($0,1,10)}')
# example output (trimmed): RETURN tcp -- anywhere anywhere tcp dpt:http RETURN tcp -- anywhere anywhere tcp dpt:ssh RETURN icmp -- anywhere anywhere
The above rules should match our Nova secgroup "base" rules we defined previously. I.e.,
$ nova secgroup-list-rules base +-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | tcp | 80 | 80 | 0.0.0.0/0 | | | tcp | 22 | 22 | 0.0.0.0/0 | | | icmp | -1 | -1 | 0.0.0.0/0 | | +-------------+-----------+---------+-----------+--------------+
- Capture traffic to/from host and/or instance:
$ tcpdump -nni eth0 \( dst host 10.0.0.19 and port 22 \) $ tcpdump -nni eth0 icmp $ tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
- See what virsh knows about our instance:
$ VIRSH_INSTANCE_ID=$(nova show ${INSTANCE_ID} |\grep OS-EXT-SRV-ATTR:instance_name|awk '{print $4}')
$ virsh dumpxml instance-00000003 | grep -A3 bridge
$ grep -A4 'type="bridge"' /var/lib/nova/instances/${INSTANCE_ID}/libvirt.xml
$ xmllint --xpath '//devices/interface[@type="bridge"]/source[@bridge]' libvirt.xml
Open vSwitch (OVS)
+--------------------------------------+
| cirros |
| +------+ |
| | eth0 | (10.10.1.102) |
+---------------+------+---------------+
|
|
+-----------------+
Virtual NIC (tap) | tap-55bfa719-2b |
+-----------------+
|
|
+-----------------+
Linux bridge | qbr-55bfa719-2b |
+-----------------+
|
|
+-----------------+
| qvb-55bfa719-2b |
+-----------------+
veth pair |
+-----------------+
| qvo-55bfa719-2b |
+---------------+-----------------+--------------+
OVS Bridge | 10.10.1.0/24 br-int |
+--+----------------+-----------+----------------+
/ | qr-b24d8155-69 | \ / | tap0d8a8773-84 | \
/ +----------------+ \ / +----------------+ \
/ \ / \
| qrouter- | | Namespace: |
| 9f36.... | | qdhcp- |
| | | |
| 10.1.100.16 | \______________________/
\ +----------------+ /
\ | qg-20ffa8ce-2f | /
+--+----------------+--+
| 10.1.100.0/23 |
OVS Bridge | br-ex |
+----------------------+
The names for the OVS ports and interfaces are made from a prefix (qr-, qg-, or tap) followed by the first 11 chars of the Neutron port they represent.
- Inspect the ethernet bridge ("Linux bridge") configuration in the Linux kernel:
$ brctl show
bridge name bridge id STP enabled interfaces
qbr55bfa719-2b 8000.fe163e32aa55 no qvb55bfa719-2b
tap55bfa719-2b
$ brctl showmacs qbr55bfa719-2b port no mac addr is local? ageing timer 2 fe:16:3e:32:aa:55 yes 0.00 2 fe:16:3e:32:aa:55 yes 0.00 1 fe:be:6e:86:cc:9f yes 0.00 1 fe:be:6e:86:cc:9f yes 0.00
The "bridge id" ("8000.fe163e32aa55") should match one of the MAC addresses listed above. That is,
8000.fe163e32aa55 => fe:16:3e:32:aa:55 (on port #2)
Every bridge has an id associated with it; this is an eight-byte number, the first two bytes being the bridge priority, which we can set manually, and the next six bytes are the MAC address of the bridge. Under Linux, the default bridge priority is 32768 (`printf '%x\n' 32768` #=> 8000). The bridge's MAC address is that of the lowest numbered MAC address of all the bridge's ports. We generally represent the bridge ID as a two part hexadecimal number, the bridge ID followed by the MAC address as the fractional part. For example, 8000.fe163e32aa55 is the ID of a bridge with a priority of 32768 (8000 hex) and a MAC address of fe:16:3e:32:aa:55.
- Get a list of network namespaces:
$ ip netns list qrouter-9f3646b2-cfa0-435b-9708-5854656b797f # router1 qdhcp-7f8821c4-7f8a-4a48-ac17-15f27a32a60c # private_network
$ QROUTER=qrouter-$(neutron router-list|awk '/router1/{print $2}')
$ ip netns exec ${QROUTER} ip a
$ ip netns exec ${QROUTER} router
$ ip netns exec ${QROUTER} ping -c3 10.10.1.1 # gateway
$ ip netns exec ${QROUTER} iptables -S -t nat
$ ip netns exec ${QROUTER} ip r
default via 10.1.100.1 dev qg-20ffa8ce-2f
10.1.100.0/23 dev qg-20ffa8ce-2f proto kernel scope link src 10.1.100.16
10.10.1.0/24 dev qr-b24d8155-69 proto kernel scope link src 10.10.1.1
- Check interfaces configured within the network namespace:
$ ip netns exec ${QROUTER} ip a|grep -E 'state|inet'
...
32: qr-b24d8155-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 10.10.1.1/24 brd 10.10.1.255 scope global qr-b24d8155-69
36: qg-20ffa8ce-2f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 10.1.100.16/23 brd 10.1.101.255 scope global qg-20ffa8ce-2f
inet 10.1.100.19/32 brd 10.1.100.19 scope global qg-20ffa8ce-2f
The agent has bound 10.10.1.1 (the gateway_ip of our private_subnet) to qr-b24d8155-69 (router1's interface for private_subnet). DHCP pushes this IP to instances connected to private_subnet for their default gateway.
The agent has also bound IP addresses 10.1.100.16 (for gateway SNAT) and 10.1.100.19 (the floating IP) to the gateway interface qg-20ffa8ce-2f.
- Verify agent has also enabled IP forwarding within the namespace:
$ ip netns exec ${QROUTER} sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
- Look at the Network Address Translation (NAT) rules within the network namespace:
$ ip netns exec ${QROUTER} iptables-save|grep -i nat
...
-A neutron-l3-agent-OUTPUT -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-20ffa8ce-2f -j SNAT --to-source 10.1.100.16
Thus, a connection initiated from within instance to the outside will hit this rule in the NAT table (post routing):
-A neutron-l3-agent-float-snat -s 10.10.1.102/32 -j SNAT --to-source 10.1.100.19
And connection initiated from the outside to the instance's floating IP will hit this rule:
-A neutron-l3-agent-PREROUTING -d 10.1.100.19/32 -j DNAT --to-destination 10.10.1.102
$ QDHCP=qdhcp-$(neutron net-list|awk '/private_network/{print $2}')
$ ip netns exec ${QDHCP} ip r
$ grep tap /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/interface
tap0d8a8773-84
$ cat /var/lib/neutron/dhcp/${PRIVATE_NET_ID}/opts
tag:tag0,option:router,10.10.1.1
tag:tag0,option:dns-server,10.10.1.100
cirros$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.1.1 0.0.0.0 UG 0 0 0 eth0 10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
cirros$ cat /etc/resolv.conf search openstacklocal nameserver 8.8.8.8
Metadata
- Direct access to Nova metadata:
- see: for details
Note: You need to have the "hmac" and "hashlib" Python modules installed (they are by default on most distros). You also need to have crudini installed (or you could use regex with grep, sed, awk, etc.).
$ SHARED_SECRET=$(crudini --get /etc/nova/nova.conf neutron metadata_proxy_shared_secret)
$ META_SIGNATURE=$(python -c 'import hmac,hashlib;print hmac.new("'${SHARED_SECRET}'",\
"'${INSTANCE_ID}'",hashlib.sha256).hexdigest()')
$ ADMIN_TENANT_ID=$(keystone tenant-list | awk '/admin/{print $2}')
$ ENDPOINT=http://10.1.100.15:8775
$ curl -s -H "x-instance-id:${INSTANCE_ID}" \
-H "x-tenant-id:${ADMIN_TENANT_ID}" \
-H "x-instance-id-signature:${META_SIGNATURE}" \
${ENDPOINT}/latest/meta-data
# RESPONSE: ami-id ami-launch-index ami-manifest-path block-device-mapping/ hostname instance-action instance-id instance-type kernel-id local-hostname local-ipv4 placement/ public-hostname public-ipv4 public-keys/ ramdisk-id reservation-id security-groups
See also
External links
- xtof-openstack-rdo-packstack on GitHub
