Readiness probe failed: Client.Timeout exceeded while awaiting headers)
Pods restart frequently causing periodic timeout errors
After you complete your installation, you might encounter an issue that causes some pods to become not ready every few minutes. In addition, this issue can cause login difficulty.
Symptoms
One or more pods experience multiple restarts that result in the pod or pods being frequently in a not ready state. In addition, attempts to log in result in periodic 502 Bad Gateway or 504 Gateway Timeout errors.
You can view the events for a pod that is frequently restarting by running the following commands:
Your output can include the following error message:
Readiness probe failed: Get http://<host>:<port>/readinessProbe: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
The logs might show errors similar to the following sample log messages:
[2020-01-23T19:59:23.036] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T19:59:29.064] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T19:59:38.087] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:01:53.096] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:01:59.111] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:02:08.137] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:11:39.951] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:11:50.184] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:11:59.207] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:12:08.232] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
[2020-01-23T20:13:53.051] [ERROR] [mcm-ui] [status] GET /readinessProbe 500
Causes
This issue can occur due to frequent failing readiness probes for a pod. When the pod becomes 'not ready', you might not be able to log in or use the console.
Resolving the problem
To reduce the frequency of timeout errors from this issue, you can configure a workaround or apply a DNS config patch to help resolve this issue.
Apply DNS config patch
To help address the periodic timeout errors, you can apply a OpenShift DNS config patch for IBM Cloud Pak foundational services clusters. This patch addresses an issue that occurs when accessing services from pods that results in requests that have a response delay of up to 5 seconds. Normal response times for requests typically require only a millisecond delay.
This patch is available as an interim fix for IBM Cloud Pak foundational services. For more information about this patch, and to obtain this patch, go to IBM® Fix Central. IBM® Fix Central contains fixes and updates for IBM® products. To access this website, see IBM Fix Central .
To directly access the interim fix for this patch, see CS=3.2.4-fix-37137 .
To apply the patch, follow the README instructions that are included with the interim fix.
Workaround
To resolve the issue, complete manual adjustments to all liveness and readiness probes in the pod daemonset, for example the following steps adjusts the auth-idp
daemonset as an example.
-
Install kubectl. See Installing the Kubernetes CLI (kubectl).
-
Edit the
auth-idp
daemonset. -
Locate the settings for each liveness and readiness probe for all containers in the daemonset. For example, the following section shows the
readinessProbe
settings for theplatform-auth-service
container:name: platform-auth-service ports: - containerPort: 8443 hostPort: 8443 name: http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: / port: 8443 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "1" memory: 1Gi requests: cpu: 100m memory: 256Mi
-
Set values for each probe setting for all containers in the daemonset to increase the
initialDelaySeconds
,periodSeconds
, andtimeoutSeconds
settings. If missing, add theinitialDelaySeconds
setting. The following example shows the placement of these settings for a readiness probe:readinessProbe: failureThreshold: 3 httpGet: path: / port: 8443 scheme: HTTPS initialDelaySeconds: 420 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 10
-
Save the file and wait until all the
auth-idp
pods restart. The pods might take a few minutes to restart.
原文地址:https://www.cnblogs.com/cheyunhua/p/15246305.html
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- 刷新/关闭页面之前发送请求
- Web Beacon 刷新/关闭页面之前发送请求
- 解决django无法访问本地static文件(js,css,img)网页里js,cs都加载不了
- python3 logging日志封装实例
- 解决Python中报错TypeError: must be str, not bytes问题
- H5 notification浏览器桌面通知
- Android线程池控制并发数多线程下载
- Android progressbar实现带底部指示器和文字的进度条
- js 调用栈机制与ES6尾调用优化介绍
- Android Fragment实现列表和内容联动
- 前端中等算法-无重复字符的最长子串
- Android自定义动态壁纸开发(时钟)
- 手摸手教你写个ESLint 插件以及了解ESLint的运行原理
- 填满Github的绿色格子用我做的VSCode插件-Auto Commit
- Android多国语言转换Excel及Excel转换为string详解