求助: 没有到主机的路由(no route to host)

2021-10-14 16:57:44 +08:00
 Mrkon

系统:ubuntu 16.04 (已经关闭防火墙,sudo ufw disable, telnet 可以联通) ES 版本 : 7.13.1

目的: 使用两个节点通过外网组成集群

问题描述: 已经开启一台服务器作为 master 节点,将第二个节点加入时报错: 没有到主机的路由

关键配置:

network.host: 0.0.0.0
network.publish_host: 192.168.3.134
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: ["61.15.284.58:10091", "1.65.184.8:10081"]
cluster.initial_master_nodes: ["node-1", "node-2"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

我的猜想: DNS 原因,我通过路由器将内网端口映射到固定 IP 导致的问题,报错是找不到内网 192.168.3.86:19300

映射关系: node-1: 61.15.284.58:10091 -> 192.168.3.86:19300

node-2: 1.65.184.8:10081 -> 192.168.3.134:29300

错误 log:

[2021-09-24T09:29:32,786][WARN ][o.e.d.HandshakingTransportAddressConnector] [
node-2] [connectToRemoteMasterNode[61.15.284.58:10091]] completed handshake with [{node-1}{oapoq8khQ9GtiWj63kfbCQ}{VwXYChZEQdKceJYNrCRPmw}{192.168.3.86}{192.168.3.86:19300}{cdfhilmrstw}{ml.machine_memory=33599610880, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=12884901888, transform.node=true}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [node-1][192.168.3.86:19300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:971) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.13.1.jar:7.13.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
        at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:57) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: 没有到主机的路由: 192.168.3.86/192.168.3.86:19300
Caused by: java.net.NoRouteToHostException: 没有到主机的路由
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        ... 7 more

1549 次点击
所在节点    Elasticsearch
8 条回复
wujiezero
2021-10-14 17:01:09 +08:00
我上次遇到这个问题改了下 /etc/hosts 好了
Mrkon
2021-10-14 17:23:53 +08:00
@wujiezero 感谢回复,该如何修改呢?
julyclyde
2021-10-14 17:30:22 +08:00
iptables 的问题。对方使用了 ICMP host unreachable 来拒绝连接
Mrkon
2021-10-14 17:49:55 +08:00
@julyclyde 感觉不太像我的 iptables 表分别是

node1

```
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:9600

Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (11 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:tproxy
ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:http-alt
ACCEPT tcp -- anywhere 192.168.32.2 tcp dpt:3000
ACCEPT tcp -- anywhere 172.20.0.2 tcp dpt:6379
ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7687
ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7474
ACCEPT tcp -- anywhere 172.24.0.3 tcp dpt:27017
ACCEPT tcp -- anywhere 172.24.0.4 tcp dpt:6379
ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:5000
ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:3000
ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:ssh
ACCEPT tcp -- anywhere 172.20.0.4 tcp dpt:3000
ACCEPT tcp -- anywhere 172.20.0.3 tcp dpt:5000

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (11 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
```

node2

```
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
```
julyclyde
2021-10-15 11:06:22 +08:00
看 192.168.3.86 那边的 iptables
@Mrkon
Mrkon
2021-10-17 14:46:55 +08:00
@julyclyde 192.168.3.86 的 iptables 是:上面 node1 的,就是 ACCEPT 。我实在是找不到问题点了,能帮我看下吗?

还有如果安装了 docker,当 es network.host 设置为:0.0.0.0 时,好像默认地址就会绑定为 docker 的 IP 段时怎么回事呢?
julyclyde
2021-10-18 11:24:14 +08:00
@Mrkon 那你抓包看看吧。看看是不是 ICMP host unreacable 、是谁发过来的
Mrkon
2021-11-12 16:00:58 +08:00
看了下官方文档,问题解决了,需要将所有节点 network.publish_host 设置映射前的 ip 地址 例如:61.15.284.58 ,transport.publish_port 同样需要设置 例如:10091

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/807846

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX