用 GCP 搭建 LVS/TUN 失败,求教

2018-09-05 11:30:46 +08:00
 qqqasdwx

想搭个 LVS 测试,但怎么也不成功,求大神们给点思路。 我是跟着以下文章做的

LVS 负载均衡之 LVS-TUN 实例部署(案例篇)

相关环境:

Director Server:google 的 GCP,公网 VPS,处于 NAT 环境下,eth0 分配的是内网地址
Real Server:公司宽带,有公网地址,多层 NAT,映射 8086 端口到服务器的 8086 端口,服务器 eth0 分配的是 192.168.0.XX
客户端:手机,用流量访问

DS 和 RS 全是新装的系统,centos6.10

以下是操作流程

Director Server

先安装软件包及依赖

yum install openssl-devel popt-devel libnl-devel ipvsadm  -y

然后新建一个 shell 脚本lvs.sh,如下:

#!/bin/sh
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise the Linux Virtual Server for TUN
#
### BEGIN INIT INFO
# Provides: ipvsadm
# Required-Start: $local_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Short-Description: Initialise the Linux Virtual Server
# Description: The Linux Virtual Server is a highly scalable and highly
#   available server built on a cluster of real servers, with the load
#   balancer running on Linux.
# description: start LVS of TUN
LOCK=/var/lock/lvs-tun.lock
VIP=GCP 的公网 IP
RIP1=公司的公网 IP
#RIP2=192.168.1.11
. /etc/rc.d/init.d/functions

start()    {
     PID=`ipvsadm -Ln | grep ${VIP} | wc -l`
     if    [ $PID -gt 0 ];

     then
           echo "The LVS-TUN Server is already running !"
     else
           #Load the tun mod
           /sbin/modprobe tun
           /sbin/modprobe ipip
           #Set the tun Virtual IP Address
           /sbin/ifconfig tunl0 $VIP broadcast $VIP netmask 255.255.255.255 up
           /sbin/route add -host $VIP dev tunl0
           #Clear IPVS Table
           /sbin/ipvsadm -C
           #The icmp recruit setting
           echo "0" >/proc/sys/net/ipv4/ip_forward
           echo "0" >/proc/sys/net/ipv4/conf/all/send_redirects
           echo "0" >/proc/sys/net/ipv4/conf/default/send_redirects
           echo "0" >/proc/sys/net/ipv4/conf/eth0/send_redirects
           #echo "0" >/proc/sys/net/ipv4/conf/eth1/send_redirects
           #Set Lvs
           /sbin/ipvsadm -At $VIP:8086 -s rr
           /sbin/ipvsadm -at $VIP:8086 -r $RIP1:8086 -i  -w 1
           #/sbin/ipvsadm -at $VIP:80 -r $RIP2:80 -i  -w 1
           /bin/touch $LOCK
           #Run Lvs
           echo "starting LVS-TUN-DIR Server is ok !"       
     fi
}

stop()    {
           #stop  Lvs server
           /sbin/ipvsadm -C
           /sbin/ifconfig tunl0 down >/dev/null
           #Remove the tun mod
           /sbin/modprobe -r tun
           /sbin/modprobe -r ipip
           rm -rf $LOCK
           echo "stopping LVS-TUN-DIR server is ok !"
}

status()  {
     if [ -e $LOCK ];
     then
         echo "The LVS-TUN Server is already running !"
     else
         echo "The LVS-TUN Server is not running !"
     fi
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        stop
        sleep 1
        start
        ;;
  status)
        status
        ;;
  *)
        echo "Usage: $1 {start|stop|restart|status}"
        exit 1
esac
exit 0

加权限,运行

chmod 777 lvs.sh
./lvs.sh start

显示starting LVS-TUN-DIR Server is ok ! 配置完成

Real Server

新建一个 shell 脚本lvs.sh,如下:

#!/bin/sh
#
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise the Linux Virtual Server for TUN
#
### BEGIN INIT INFO
# Provides: ipvsadm
# Required-Start: $local_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Short-Description: Initialise the Linux Virtual Server
# Description: The Linux Virtual Server is a highly scalable and highly
#   available server built on a cluster of real servers, with the load
#   balancer running on Linux.
# description: start LVS of TUN-RIP
LOCK=/var/lock/ipvsadm.lock
VIP=GCP 的公网 IP
. /etc/rc.d/init.d/functions
start() {
     PID=`ifconfig | grep tunl0 | wc -l`
     if [ $PID -ne 0 ];
     then
         echo "The LVS-TUN-RIP Server is already running !"
     else
         #Load the tun mod
         /sbin/modprobe tun
         /sbin/modprobe ipip
         #Set the tun Virtual IP Address
         /sbin/ifconfig tunl0 $VIP netmask 255.255.255.255 broadcast $VIP up
         /sbin/route add -host $VIP dev tunl0
         echo "1" >/proc/sys/net/ipv4/conf/tunl0/arp_ignore
         echo "2" >/proc/sys/net/ipv4/conf/tunl0/arp_announce
         echo "1" >/proc/sys/net/ipv4/conf/eth0/arp_ignore
         echo "2" >/proc/sys/net/ipv4/conf/eth0/arp_announce
         echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
         echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
         echo "0" > /proc/sys/net/ipv4/conf/tunl0/rp_filter
         echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter
         /bin/touch $LOCK
         echo "starting LVS-TUN-RIP server is ok !"
     fi
}

stop() {
         /sbin/ifconfig tunl0 down
         echo "0" >/proc/sys/net/ipv4/conf/tunl0/arp_ignore
         echo "0" >/proc/sys/net/ipv4/conf/tunl0/arp_announce
         echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_ignore
         echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_announce
         echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
         echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
         #Remove the tun mod
         /sbin/modprobe -r tun
         /sbin/modprobe -r ipip
         rm -rf $LOCK
         echo "stopping LVS-TUN-RIP server is ok !"
}

status() {
     if [ -e $LOCK ];
     then
        echo "The LVS-TUN-RIP Server is already running !"
     else
        echo "The LVS-TUN-RIP Server is not running !"
     fi
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        stop
        start
        ;;
  status)
        status
        ;;
  *)
        echo "Usage: $1 {start|stop|restart|status}"
        exit 1
esac
exit 0

加权,运行

chmod 777 lvs.sh
./lvs.sh start

然后在服务器上放了个监听 8086 端口的静态页

公司路由器

映射 8086 端口到测试服务器的 8086 端口。

客户端

手机访问 http://GCP 公网 IP:8086

然后没有成功 T。T

抓包

望各位大佬提点几句,感激不尽!

2071 次点击
所在节点    程序员
5 条回复
defunct9
2018-09-05 11:41:42 +08:00
ipip 的 tunnel 不太对
1423
2018-09-05 11:58:47 +08:00
gcp 只允许 TCP 和 UDP 流量
tunnel 不行
yexm0
2018-09-05 12:04:33 +08:00
gcp 的防火墙一如既往的垃圾
qqqasdwx
2018-09-05 12:11:48 +08:00
@defunct9 #1 应该怎么设置呢,求教!

@1423 #2 哇,是这样的么,我换个 VPS 试一下!

@yexm0 有赠金快到期了,就顺手用 GCP 测了,不出所料,果然很垃圾
ConDuseW
2020-10-08 15:45:10 +08:00
对不起挖坟了。恕我直言,这脚本作者就一 XX,DS 脚本里面有这么一行“echo "0" >/proc/sys/net/ipv4/ip_forward”,直接把转发给关了,能成功才有鬼了,网络上关于 lvs 的配置良莠不齐,我也碰壁好久

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/486334

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX