HA-Proxy setup for Openstack Services

HA-PROXY SETUP FOR OPENSTACK SERVICES


INTRODUCTION:
To obtain maximum uptime in openstack services we are in need of high availability (aka HA ) in this document we are using ha-proxy for openstack HA and keepalived for Ha-proxy HA.
Software topology
As mentioned, we’ll use the following set of software:
  • the services themselves;
  • HAProxy for making the services HA; and
  • Keepalived for making HAProxy HA.


Getting hands-on

Suppose we have two machines, from which we want to make an HA OpenStack controller pair, installing both the API services and Keepalived + HAProxy.
Suppose machine 1 has address 192.168.56.200, machine 2 has address 192.168.56.201, and we want the services to be accessible through virtual IP 192.168.56.210. And suppose all these IPs are on eth1.










Installing necessary packages$ sudo apt-get install haproxy keepalived

Configuration of haproxy

This configuration is identical on both nodes and resides in /etc/haproxy/haproxy.cfg
$ cat /etc/haproxy/haproxy.cfg
global
chroot /var/lib/haproxy
daemon
group haproxy
log 192.168.56.200 local0
maxconn 4000
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
user haproxy


defaults
log global
maxconn 8000
mode http
option redispatch
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s


listen keystone-1 192.168.56.210:5000
balance roundrobin
option tcplog
server controller-1 192.168.56.200:5000 check
server controller-2 192.168.56.201:5000 check


listen keystone-2 192.168.56.210:35357
balance roundrobin
option tcplog
server controller-1 192.168.56.200:35357 check
server controller-2 192.168.56.201:35357 check


listen nova-api-1 192.168.56.210:8773
balance roundrobin
option tcplog
server controller-1 192.168.56.200:8773 check
server controller-2 192.168.56.201:8773 check


listen nova-api-2 192.168.56.210:8774
balance roundrobin
option tcplog
server controller-1 192.168.56.200:8774 check
server controller-2 192.168.56.201:8774 check


listen nova-api-3 192.168.56.210:8775
balance roundrobin
option tcplog
server controller-1 192.168.56.200:8775 check
server controller-2 192.168.56.201:8775 check


listen nova-api-4 192.168.56.210:8776
balance roundrobin
option tcplog
server controller-1 192.168.56.200:8776 check
server controller-2 192.168.56.201:8776 check


listen glance-api 192.168.56.210:9292
balance roundrobin
option tcplog
server controller-1 192.168.56.200:9292 check
server controller-2 192.168.56.201:9292 check






This configuration encompasses the four nova-api services (EC2, volume, compute, metadata), glance-api, and the two keystone-api services (regular and admin API).
Now restart HAProxy on both nodes:
$ sudo service haproxy restart
It might not start, due to two reasons.
1. Haproxy is not enabled. You have to enable the haproxy by editing the file /etc/default/haproxy and change the line that says 'enabled=0' to 'enabled=1'
2. Stats socket file is not created. To resolve this, create a directory like below
# mkdir /var/lib/haproxy
# touch /var/lib/haproxy/stats
Once done, restart the haproxy.

Configuration of Keepalived

This configuration is almost, but not quite identical on both nodes as well, and resides in /etc/keepalived/keepalived.conf.
vrrp_script chk_haproxy { # Requires keepalived-1.1.13
script "killall -0 haproxy" # cheaper than pidof
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}


vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.56.210
}
track_script {
chk_haproxy
}
}


vrrp_script chk_haproxy { # Requires keepalived-1.1.13
script "killall -0 haproxy" # cheaper than pidof
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}


vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 100 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.56.210


}
track_script {
chk_haproxy
}
}


The difference is that one node has its priority defined as 101, and the other as 100. Whichever of the available nodes has highest priority at any given moment, wins (that is, claims the virtual IP).


And now let us check that Keepalived + HAProxy work by poking glance
telnet 192.168.56.210 9292
192.168.56.210...
Connected to 192.168.56.210.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
Also, we can see that just one of the controllers—the one with higher “priority”—claimed the virtual IP
openstack@controller-1:~$ ip addr show dev eth1
eth1: mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 08:00:27:9d:c4:b0 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.200/24 brd 192.168.56.255 scope global eth1
inet 192.168.56.210/32 scope global eth1
inet6 fe80::a00:27ff:fe9d:c4b0/64 scope link
valid_lft forever preferred_lft forever


openstack@controller-2:~$ ip addr show dev eth1
2: eth1: mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 08:00:27:bd:7f:14 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.201/24 brd 192.168.56.255 scope global eth1
inet6 fe80::a00:27ff:febd:7f14/64 scope link
valid_lft forever preferred_lft forever





Configuration of Open Stack services

Now for the service wiring. We need two things:
1) to listen on the proper local IP address and
2) address others by the virtual IP address.
NOVA
openstack@controller-1:~$ cat /etc/nova/nova.conf
--metadata_listen=192.168.56.200
--glance_api_servers=192.168.56.210:9292
--osapi_volume_listen=192.168.56.200
--ec2_listen=192.168.56.200
--sql_connection=mysql://nova:nova@192.168.56.210/nova
--osapi_compute_listen=192.168.56.200
--novncproxy_host=192.168.56.210


openstack@controller-1~$ cat /etc/nova/api-paste.ini
[filter:authtoken]
auth_host = 192.168.56.210








KEYSTONE
openstack@controller-1~$ cat /etc/keystone/keystone.conf
[DEFAULT]
bind_host = 192.168.56.200
[sql]
connection = mysql://keystone_admin:nova@192.168.56.210/keystone


GLANCE
openstack@controller-1:~$ cat /etc/glance/glance-scrubber.conf
[DEFAULT]
registry_host = 192.168.56.210
openstack@controller-1:~$ cat /etc/glance/glance-api-paste.ini
[filter:authtoken]
auth_host = 192.168.56.210
auth_uri = http://192.168.56.210:5000/
openstack@controller-1:~$ cat /etc/glance/glance-api.conf
[DEFAULT]
registry_host = 192.168.56.210
bind_host = 192.168.56.200
openstack@controller-1:~$ cat /etc/glance/glance-api-paste.ini
[filter:authtoken]
auth_host = 192.168.56.200
auth_uri = http://192.168.56.200:5000/
openstack@controller-1:~$ cat /etc/glance/glance-cache.conf
[DEFAULT]
registry_host = 192.168.56.210
auth_url = http://192.168.56.210:5000/
openstack@controller-1:~$ cat /etc/glance/glance-registry.conf
[DEFAULT]
bind_host = 192.168.56.200
sql_connection = mysql://glance:nova@192.168.56.210/glance
openstack@controller-1:~$ cat /etc/glance/glance-registry-paste.ini
[filter:authtoken]
auth_host = 192.168.56.210
auth_uri = http://192.168.56.210:5000/


OPENRC - FILE (ENV FILE)
openstack@controller-1:~$ cat /root/openrc
export OS_AUTH_URL="http://192.168.56.210:5000/v2.0/"
export SERVICE_ENDPOINT=http://192.168.56.210:35357/v2.0/


Keypoint while using HA-Proxy service to load balance services.
1) Keystone need to configured in one place.
2) My sql should be common and it should be pointed to common sql in Keystone,glance,nova API files.


3) Also in /etc/openstack-dashboard/local_settings.py
CACHE_BACKEND = 'memcached://192.168.3.227:11211/'
(IMP point common memcached server in all server else horizon dashboard will logout frequently or it will throw access denied or authorization error.)
&
OPENSTACK_HOST = "192.168.3.220"
4)novaconsole-auth is to be common for all controllers.
https://bugzilla.redhat.com/show_bug.cgi?id=910790




5) Api conf file bind configuration for HA-proxy
A) Glance service:
Glance-api.conf
Address to bind the API server
bind_host = 192.168.3.227
bind_port = 9292
Glance-registry.conf
bind_host = 192.168.3.227
bind_port = 9191


B)/etc/keystone/ keystone.conf:
bind_host = 192.168.3.227


C)Note: Rabbitmq should be point their own server as of now.
D) Auth file should export -should be common on all servers using .bashrc or stackrc.


E)To make Rabbitmq server listen specific port and Ip-----> /usr/lib/rabbitmq/bin/ rabbitmq-env
export RABBITMQ_NODENAME=rabbit@openstack1
export RABBITMQ_NODE_IP_ADDRESS=192.168.3.230 -->Specific host.
export ERL_EPMD_ADDRESS=192.168.3.230




F)/etc/nova/api-paste.ini-----> below setting (need to be done)
service_protocol = http
service_host = 192.168.3.220
service_port = 5000
auth_host = 192.168.3.220
auth_port = 35357
auth_protocol = http
auth_uri = http://192.168.3.220:5000/
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = nova
admin_password = nova


G)Mysql listen to its IP.
MY-sql tuning -->handling sql queries,loadbalance,etc...


H)/etc/keystone//keystone.conf
for cookies increasing or decreasing
# Amount of time a token should remain valid (in seconds)
expiration = 86400
[sql]
#connection = sqlite:////var/lib/keystone/keystone.db
connection = mysql://keystonedbadmin:root@192.168.3.227/keystone
idle_timeout = 200









Comments

Post a Comment

Popular posts from this blog

Openstack to Cloudstack Template Migration

Image
This blog covers the image/template migration between Openstack to Cloudstack platform. To be precise, converting the image from QCOW2 format to VHD format and upload it to Cloudstack platform.   Requirements Root access privileges to openstack and cloudstack with Xen server platform Enough free space to do the conversion python 2.7 to run a http server Image Download This part is already covered in one of my existing blogs. Please refer  https://benyraja.blogspot.in/2017/03/templateimage-migration-with-reduced.html 'Image Download' section. Image Conversion Once we have downloaded the image to local disk, we have to convert it into VHD as it is the supported version of Cloudstack Xen hypervisor. If you are using KVM hypervisor, then we can skip to next step. Use the below command to convert the image from QCOW2 to VHD. $ qemu-img convert -O vpc Now you will have the converted image in VHD format. Image Upload/Register Template To register te...
Read more

Openstack Swift HA with HAproxy

Introduction We used five machines for swift setup . In that, 3 machines will act as swift-proxy servers and all 5 machines will act as swift-storage nodes. Swift with HA proxy setup We have installed HA proxy service in all three proxy servers. The KeepAlived service will monitor the HA proxy health and assign the floating virtual IP address(192.168.1.xxx) to the high priority HA proxy server. (102 is master, 101 is slave1, 100 is slave2) The HA Proxy will check all three proxy servers and send the request in round robin method. If one proxy server fails, it will automatically sends the request to the other two proxy servers only, until the failed proxy server is back online. Swift Proxy Node setup We have configured three swift proxy nodes in the public network(192.168.1.xxx) and it is connected to all the storage nodes through a private network(172.0.0.xxx). Rings The ring builder has created an account ring, container ring and ob...
Read more