HBase NotServingRegionException

线上最近通过日志系统发现某BU的服务里HBase某时间段内有几十次如下这样的报错，error发生在同一个表的查询上，而业务对该表的查询仅限于点查询：

根因分析：其实是因为region split造成的抖动，trip:rt_vehicle_history_5m这张表最近一些region到10G大小开始split，相当于那region 短暂close了然后再访问它就报错。解决方案： hbase.client.retries.number 稍微调整大些，默认是3-->10, pause time 100ms~1s左右

split所需时间与表的文件数有关。

附上日志

Fri Sep 25 08:38:03 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Fri Sep 25 08:38:04 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
atorg.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)