Kafka集群搭建过程(kafka2.5+eagle)
1.环境准备
1.1 硬件环境
为了安装kafka最近环境,向运维申请了3台测试虚拟机,后续很多学习过程都将在此3台服务器上进行,内存为8G,虚拟磁盘 500G. IP如下:
IP HOST
192.168.162.201 m162p201
192.168.162.202 m162p202
192.168.162.203 m162p203
操作系统版本如下:
[root@m162p201 software]# lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.7.1908 (Core)
Release: 7.7.1908
Codename: Core
1.2软件环境
下载1.8版本的jdk和kafka2.5 以及 jdk https://download.oracle.com/otn/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz kafka https://kafka.apache.org/downloads#2.5.0 scalca版本选择了2.12 在apache的下载链接上下载 https://www.apache.org/dyn/closer.cgi?path=/kafka/2.5.0/kafka_2.12-2.5.0.tgz kafka-eagle http://download.kafka-eagle.org/ 将上述文件都上传到上述三台服务器的/opt/software目录
cd /opt/software
ls
jdk-8u251-linux-x64.tar.gz kafka_2.12-2.5.0.tgz kafka-eagle-bin-1.4.8.tar.gz
apache-zookeeper-3.5.6-bin.tar.gz
1.3 jdk安装
jdk为基础软件,通常情况下不使用系统自带的jdk,一般还是通过自行解压之后,在需要使用的用户中配置环境变量即可.
mkdir /opt/jdk
cd /opt/jdk
tar -zxvf /opt/software/jdk-8u251-linux-x64.tar.gz
1.4 zookeeper安装
安装过程参见 https://www.jianshu.com/p/0e813f6a6049 此处就不赘述详细流程,在每台服务器都新建一个zookeeper用户,之后配置环境变量和各服务器的zk参数即可。
useradd -d /opt/zookeeper zookeeper
在l /etc/security/limit.conf中增加如下内容:
zookeeper soft nproc 16384
zookeeper hard nproc 16384
zookeeper soft nofile 65535
zookeeper hard nofile 65536
切换到zookeeper用户 su - zookeeper 建立软链接,便于以后切换版本:
ln -s /opt/zookeeper/zookeeper-3.4.10 /opt/zookeeper/zookeeper
环境变量 .bash_profile文件配置如下:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
JAVA_HOME=/opt/jdk1.8.0_251
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
PATH=$JAVA_HOME/bin:$PATH
export PATH
alias acdconf='cd /opt/zookeeper/apache-zookeeper/conf/'
之后配置好zookeeper启动 zookeeper配置:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=192.168.162.201:2888:3888
server.2=192.168.162.202:2888:3888
server.3=192.168.162.203:2888:3888
之后建立data目录:
mkdir /opt/zookeeper/data
echo 1 > /opt/zookeeper/data/myid
同样在另外两台服务器将myid配置为 2和3. 启动zookeeper
/opt/zookeeper/zookeeper/bin/zkServer.sh start
2 安装kafka和eagle
2.1安装kafka
useradd -d /opt/kafka kafka
在l /etc/security/limit.conf中增加如下内容:
kafka soft nproc 16384
kafka hard nproc 16384
kafka soft nofile 65535
kafka hard nofile 65536
切换到kafka 用户 su - kafka 建立软链接,便于以后切换版本:
tar -zxvf /opt/soft/kafka_2.12-2.5.0.tar.gz
ln -s /opt/kafka/kafka_2.12-2.5.0 /opt/kafka/apache-kafka
环境变量 .bash_profile文件配置如下:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
JAVA_HOME=/opt/jdk1.8.0_251
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
PATH=$JAVA_HOME/bin:$PATH
export PATH
alias acdconf='cd /opt/kafka/apache-kafka/conf/'
配置/opt/kafka/apache-kafka/config/server.properties 需要修改的参数如下:
broker.id=1
listeners=PLAINTEXT://192.168.162.201:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.162.201:2181,192.168.162.202:2181,192.168.162.203:2181
另外两台服务器配置的birker.id 分别为2和3,其他配置参数只需要修改IP m162p202
broker.id=2
listeners=PLAINTEXT://192.168.162.202:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.162.201:2181,192.168.162.202:2181,192.168.162.203:2181
m162p203
broker.id=3
listeners=PLAINTEXT://192.168.162.203:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.162.201:2181,192.168.162.202:2181,192.168.162.203:2181
之后启动kafka
cd /opt/kafka/apache-kafka
bin/kafka-server-start.sh -daemon config/server.properties
2.2安装eagle
eagle安装在kafka用户下,因此不用重新创建目录。此外eagle只需要在一个节点安装即可。 切换到kafka用户:
tar -zxvf kafka-eagle-bin-1.4.8.tar.gz
建立软链接,便于后续切换版本
ln -s /opt/kafka/kafka-eagle-web-1.4.8 /opt/kafka/kafka-eagle
在环境变量中增加eagle的相关内容:
KE_HOME=/opt/kafka/kafka-eagle
export KE_HOME
修改配置文件: vim /opt/kafka/kafka-eagle/conf/system-config.properties
######################################
# multi zookeeper & kafka cluster list
######################################
kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=192.168.162.201:2181,192.168.162.202:2181,192.168.162.203:2181
######################################
# broker size online list
######################################
cluster1.kafka.eagle.broker.size=20
######################################
# zk client thread limit
######################################
kafka.zk.limit.size=25
######################################
# kafka eagle webui port
######################################
kafka.eagle.webui.port=8048
######################################
# kafka offset storage
######################################
cluster1.kafka.eagle.offset.storage=kafka
cluster2.kafka.eagle.offset.storage=zk
######################################
# kafka metrics, 15 days by default
######################################
kafka.eagle.metrics.charts=false
kafka.eagle.metrics.retain=15
######################################
# kafka sql topic records max
######################################
kafka.eagle.sql.topic.records.max=5000
kafka.eagle.sql.fix.error=false
######################################
# delete kafka topic token
######################################
kafka.eagle.topic.token=keadmin
######################################
# kafka sasl authenticate
######################################
cluster1.kafka.eagle.sasl.enable=false
cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster1.kafka.eagle.sasl.mechanism=SCRAM-SHA-256
cluster1.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka" password="kafka-eagle";
cluster1.kafka.eagle.sasl.client.id=
cluster1.kafka.eagle.sasl.cgroup.enable=false
cluster1.kafka.eagle.sasl.cgroup.topics=
cluster2.kafka.eagle.sasl.enable=false
cluster2.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster2.kafka.eagle.sasl.mechanism=PLAIN
cluster2.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-eagle";
cluster2.kafka.eagle.sasl.client.id=
cluster2.kafka.eagle.sasl.cgroup.enable=false
cluster2.kafka.eagle.sasl.cgroup.topics=
######################################
# kafka ssl authenticate
######################################
cluster3.kafka.eagle.ssl.enable=false
cluster3.kafka.eagle.ssl.protocol=SSL
cluster3.kafka.eagle.ssl.truststore.location=
cluster3.kafka.eagle.ssl.truststore.password=
cluster3.kafka.eagle.ssl.keystore.location=
cluster3.kafka.eagle.ssl.keystore.password=
cluster3.kafka.eagle.ssl.key.password=
cluster3.kafka.eagle.ssl.cgroup.enable=false
cluster3.kafka.eagle.ssl.cgroup.topics=
######################################
# kafka sqlite jdbc driver address
######################################
kafka.eagle.driver=org.sqlite.JDBC
kafka.eagle.url=jdbc:sqlite:/opt/kafka/kafka-eagle/db/ke.db
kafka.eagle.username=admin
kafka.eagle.password=123456
######################################
# kafka mysql jdbc driver address
######################################
#kafka.eagle.driver=com.mysql.jdbc.Driver
#kafka.eagle.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
#kafka.eagle.username=root
#kafka.eagle.password=123456
由于服务器上没有安装mysql因此,用sqlite即可 kafka.eagle.username=admin kafka.eagle.password=123456 则指定了eagle登陆的用户名和密码。 监听8048端口。此外,为了eagle采集更多的kafka运行信息,需要开启kafka的jmx服务,修改每个节点kafka的启动脚本 vim /opt/kafka/apache-kafka/bin/kafka-server-start.sh 增加30行内容
28 if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
29 export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
30 export JMX_PORT="7788"
31 fi
之后重启即可。注意每个节点都需要修改。 现在启动eagle cd /opt/kafka/kafka-eagle/bin 执行 ke.sh 即可 初次启动,需要初始化数据库,比较慢,耐心等待即可。 eagel界面访问: http://192.168.162.201:8048/ke/account/signin?/ke/
eagle的各项监控指标都比较齐全,对kafka的架构比较了解之后,很方便就能看懂。因此不再做过多介绍。
- 通俗理解决策树中的熵&条件熵&信息增益
- KNN实现手写数字识别
- softmax分类算法原理(用python实现)
- DeepLearning.ai学习笔记(一)神经网络和深度学习--Week4深层神经网络
- 博客园自定义皮肤扁平化设计
- DeepLearning.ai学习笔记(四)卷积神经网络 -- week3 目标检测
- DeepLearning.ai学习笔记(四)卷积神经网络 -- week2深度卷积神经网络 实例探究
- DeepLearning.ai学习笔记(四)卷积神经网络 -- week1 卷积神经网络基础知识介绍
- Udacity并行计算课程笔记-The GPU Hardware and Parallel Communication Patterns
- DeepLearning.ai学习笔记(三)结构化机器学习项目--week2机器学习策略(2)
- DeepLearning.ai学习笔记(三)结构化机器学习项目--week1 机器学习策略
- coursera 视频总是缓冲或者无法观看的解决办法
- DeepLearning.ai学习笔记(二)改善深层神经网络:超参数调试、正则化以及优化--week3 超参数调试、Batch正则化和程序框架
- DeepLearning.ai学习笔记(一)神经网络和深度学习--Week3浅层神经网络
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- PHP实现一个限制实例化次数的类示例
- AndroidStudio中重载方法@Override的使用详解
- mysqli扩展无法在PHP7下升级问题的解决
- PHP通过文件保存和更新信息的方法分析
- PHP中将一个字符串部分字符用星号*替代隐藏的实现代码
- PHP创建XML的方法示例【基于DOMDocument类及SimpleXMLElement类】
- android之json数据过长打印不全问题的解决
- PHP实现时间日期友好显示实现代码
- AndroidStudio插件GsonFormat之Json快速转换JavaBean教程
- android studio错误: 常量字符串过长的解决方式
- Android Shader着色器/渲染器的用法解析
- PHP实现的文件浏览器功能简单示例
- Laravel中正确地返回HTTP状态码方法示例
- Android 实现抖音头像底部弹框效果的实例代码
- Android Studio修改Log信息颜色的实现