首页    技术博客    PHP教程    数据库技术    前端开发   HTML5    Nginx    php论坛
新用户注册 | 会员登录
加入会员，解锁专属内容
取消
热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Kafka：ZK+Kafka+SparkStreaming集群环境搭建（三）安装spark2.2.1

作者：手机用户2502861455 | 来源：互联网 | 2023-09-09 13:21
如何搭建配置centos虚拟机请参考《Kafka：ZK+Kafka+SparkStreaming集群环境搭建（一）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。》如

如何搭建配置centos虚拟机请参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（一）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。》

如何安装hadoop2.9.0请参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二）安装hadoop2.9.0》

如何配置hadoop2.9.0 HA 请参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》

安装spark的服务器：

192.168.0.120      master
192.168.0.121      slave1
192.168.0.122      slave2
192.168.0.123      slave3

从spark官网下载spark安装包：

官网地址：http://spark.apache.org/downloads.html

注意：上一篇文章中我们安装了hadoop2.9.0，但是这里没有发现待下载spark对应的hadoop版本可选项中发现hadoop2.9.0，因此也只能选择“Pre-built for Apache Hadoop 2.7 and later”。

 

这spark可选版本比较多，就选择“2.2.1(Dec 01 2017)”。

选中后，此时带下来的spark安装包版本信息为：

下载“spark-2.2.1-bin-hadoop2.7.tgz”，上传到master的/opt目录下，并解压：

[root@master opt]# tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz 
[root@master opt]# ls
hadoop-2.9.0  hadoop-2.9.0.tar.gz  jdk1.8.0_171  jdk-8u171-linux-x64.tar.gz  scala-2.11.0  scala-2.11.0.tgz  spark-2.2.1-bin-hadoop2.7  spark-2.2.1-bin-hadoop2.7.tgz
[root@master opt]#

配置Spark

[root@master opt]# ls
hadoop-2.9.0  hadoop-2.9.0.tar.gz  jdk1.8.0_171  jdk-8u171-linux-x64.tar.gz  scala-2.11.0  scala-2.11.0.tgz  spark-2.2.1-bin-hadoop2.7  spark-2.2.1-bin-hadoop2.7.tgz
[root@master opt]# cd spark-2.2.1-bin-hadoop2.7/conf/
[root@master conf]# ls
docker.properties.template  metrics.properties.template   spark-env.sh.template
fairscheduler.xml.template  slaves.template
log4j.properties.template   spark-defaults.conf.template
[root@master conf]# scp spark-env.sh.template spark-env.sh
[root@master conf]# ls
docker.properties.template  metrics.properties.template   spark-env.sh
fairscheduler.xml.template  slaves.template               spark-env.sh.template
log4j.properties.template   spark-defaults.conf.template
[root@master conf]# vi spark-env.sh

在spark-env.sh末尾添加以下内容（这是我的配置，你需要根据自己安装的环境情况自行修改）：

export SCALA_HOME=/opt/scala-2.11.0
export JAVA_HOME=/opt/jdk1.8.0_171
export HADOOP_HOME=/opt/hadoop-2.9.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=master
SPARK_LOCAL_DIRS=/opt/spark-2.2.1-bin-hadoop2.7
SPARK_DRIVER_MEMORY=1G

注：在设置Worker进程的CPU个数和内存大小，要注意机器的实际硬件条件，如果配置的超过当前Worker节点的硬件条件，Worker进程会启动失败。

vi slaves在slaves文件下填上slave主机名：

[root@master conf]# scp slaves.template slaves
[root@master conf]# vi slaves

配置内容为：

#localhost
slave1
slave2
slave3

将配置好的spark-2.2.1-bin-hadoop2.7文件夹分发给所有slaves吧

scp -r /opt/spark-2.2.1-bin-hadoop2.7 spark@slave1:/opt/
scp -r /opt/spark-2.2.1-bin-hadoop2.7 spark@slave2:/opt/
scp -r /opt/spark-2.2.1-bin-hadoop2.7 spark@slave3:/opt/

 注意：此时默认slave1,slave2,slave3上是没有/opt/spark-2.2.1-bin-hadoop2.7，因此直接拷贝可能会出现无权限操作 。

            解决方案，分别在slave1,slave2,slave3的/opt下创建spark-2.2.1-bin-hadoop2.7，并分配777权限。

[root@slave1 opt]# mkdir spark-2.2.1-bin-hadoop2.7
[root@slave1 opt]# chmod 777 spark-2.2.1-bin-hadoop2.7
[root@slave1 opt]#

之后，再次操作拷贝就有权限操作了。

启动Spark

在spark安装目录下执行下面命令才行 , 目前的master安装目录在/opt/spark-2.2.1-bin-hadoop2.7

sbin/start-all.sh

此时，我使用非root账户（spark用户名的用户）启动spark，出现master上spark无权限写日志的问题：

[spark@master opt]$ cd /opt/spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ sbin/start-all.sh
mkdir: cannot create directory ‘/opt/spark-2.2.1-bin-hadoop2.7/logs’: Permission denied
chown: cannot access ‘/opt/spark-2.2.1-bin-hadoop2.7/logs’: No such file or directory
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
/opt/spark-2.2.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out: No such file or directory
failed to launch: nice -n 0 /opt/spark-2.2.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host master --port 7077 --webui-port 8080
tail: cannot open ‘/opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out’ for reading: No such file or directory
full log in /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave1.out
slave3: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave3.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave2.out
[spark@master spark-2.2.1-bin-hadoop2.7]$ cd ..
[spark@master opt]$ su root
Password: 
[root@master opt]# chmod 777 spark-2.2.1-bin-hadoop2.7
[root@master opt]# su spark
[spark@master opt]$ cd spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ sbin/start-all.sh           
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
slave2: org.apache.spark.deploy.worker.Worker running as process 3153.  Stop it first.
slave3: org.apache.spark.deploy.worker.Worker running as process 3076.  Stop it first.
slave1: org.apache.spark.deploy.worker.Worker running as process 3241.  Stop it first.
[spark@master spark-2.2.1-bin-hadoop2.7]$ sbin/stop-all.sh 
slave1: stopping org.apache.spark.deploy.worker.Worker
slave3: stopping org.apache.spark.deploy.worker.Worker
slave2: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
[spark@master spark-2.2.1-bin-hadoop2.7]$ sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave1.out
slave3: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave3.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.2.1-bin-hadoop2.7/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-slave2.out

解决方案：给master的spark安装目录也分配777操作权限。

验证 Spark 是否安装成功

启动过程发现问题：

1）以spark on yarn方式运行spark-shell抛出异常：ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful：解决方案参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（六）针对spark2.2.1以yarn方式启动spark-shell抛出异常：ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful》

用jps检查，在 master 上正常启动包含以下几个进程：

$ jps
7949 Jps
7328 SecondaryNameNode
7805 Master
7137 NameNode
7475 ResourceManager

在 slave 上正常启动包含以下几个进程：

$jps
3132 DataNode
3759 Worker
3858 Jps
3231 NodeManager

进入Spark的Web管理页面： http://192.168.0.120:8080

运行示例

本地方式两线程运行测试：

[spark@master spark-2.2.1-bin-hadoop2.7]$ cd /opt/spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ ./bin/run-example SparkPi 10 --master local[2]

Spark Standalone 集群模式运行

[spark@master spark-2.2.1-bin-hadoop2.7]$ cd /opt/spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ ./bin/spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master spark://master:7077 \
> examples/jars/spark-examples_2.11-2.2.1.jar \
> 100

此时是可以从spark监控界面查看到运行状况：

Spark on YARN 集群上 yarn-cluster 模式运行

[spark@master spark-2.2.1-bin-hadoop2.7]$ cd /opt/spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ ./bin/spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn-cluster \
> /opt/spark-2.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.1.jar \
> 10

执行日志信息：

[spark@master hadoop-2.9.0]$ cd /opt/spark-2.2.1-bin-hadoop2.7
[spark@master spark-2.2.1-bin-hadoop2.7]$ ./bin/spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn-cluster \
> /opt/spark-2.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.1.jar \
> 10
Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
18/06/30 22:55:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/30 22:55:37 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.120:8032
18/06/30 22:55:38 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
18/06/30 22:55:38 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container)
18/06/30 22:55:38 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/06/30 22:55:38 INFO yarn.Client: Setting up container launch context for our AM
18/06/30 22:55:38 INFO yarn.Client: Setting up the launch environment for our AM container
18/06/30 22:55:38 INFO yarn.Client: Preparing resources for our AM container
18/06/30 22:55:40 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/06/30 22:55:47 INFO yarn.Client: Uploading resource file:/opt/spark-2.2.1-bin-hadoop2.7/spark-f46b4dc7-8074-4bb3-babd-c3124d1a7e07/__spark_libs__1523582418834894726.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1530369937777_0001/__spark_libs__1523582418834894726.zip
18/06/30 22:56:02 INFO yarn.Client: Uploading resource file:/opt/spark-2.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.1.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1530369937777_0001/spark-examples_2.11-2.2.1.jar
18/06/30 22:56:02 INFO yarn.Client: Uploading resource file:/opt/spark-2.2.1-bin-hadoop2.7/spark-f46b4dc7-8074-4bb3-babd-c3124d1a7e07/__spark_conf__4967231916988729566.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1530369937777_0001/__spark_conf__.zip
18/06/30 22:56:02 INFO spark.SecurityManager: Changing view acls to: spark
18/06/30 22:56:02 INFO spark.SecurityManager: Changing modify acls to: spark
18/06/30 22:56:02 INFO spark.SecurityManager: Changing view acls groups to: 
18/06/30 22:56:02 INFO spark.SecurityManager: Changing modify acls groups to: 
18/06/30 22:56:02 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
18/06/30 22:56:02 INFO yarn.Client: Submitting application application_1530369937777_0001 to ResourceManager
18/06/30 22:56:02 INFO impl.YarnClientImpl: Submitted application application_1530369937777_0001
18/06/30 22:56:03 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:03 INFO yarn.Client: 
         client token: N/A
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1530370563128
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1530369937777_0001/
         user: spark
18/06/30 22:56:04 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:05 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:06 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:07 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:08 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:09 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:10 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:11 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:12 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:13 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:14 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:15 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:16 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:17 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:18 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:19 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:20 INFO yarn.Client: Application report for application_1530369937777_0001 (state: ACCEPTED)
18/06/30 22:56:22 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:22 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.0.121
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1530370563128
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1530369937777_0001/
         user: spark
18/06/30 22:56:23 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:24 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:25 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:26 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:27 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:29 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:30 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:31 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:32 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:33 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:34 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:35 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:36 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:37 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:38 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:39 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:40 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:41 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:42 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:43 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:45 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:46 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:47 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:48 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:49 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:50 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:51 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:52 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:53 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:54 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:55 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:56 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:57 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:58 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:56:59 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:57:00 INFO yarn.Client: Application report for application_1530369937777_0001 (state: RUNNING)
18/06/30 22:57:01 INFO yarn.Client: Application report for application_1530369937777_0001 (state: FINISHED)
18/06/30 22:57:01 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.0.121
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1530370563128
         final status: SUCCEEDED
         tracking URL: http://master:8088/proxy/application_1530369937777_0001/
         user: spark
18/06/30 22:57:01 INFO util.ShutdownHookManager: Shutdown hook called
18/06/30 22:57:01 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-f46b4dc7-8074-4bb3-babd-c3124d1a7e07

从hadoop yarn监控界面查看执行任务：

另外也可以进入http://slave1:8042查看slave1的信息：

 

注意：Spark on YARN 支持两种运行模式，分别为yarn-cluster和yarn-client，具体的区别可以看这篇博文，从广义上讲，yarn-cluster适用于生产环境；而yarn-client适用于交互和调试，也就是希望快速地看到application的输出。

 


推荐阅读
  • php

    在CentOS系统下详细解析阿里云ECS服务器上搭建LAMP环境的配置步骤

    本文详细介绍了在CentOS 6.5 64位系统上使用阿里云ECS服务器搭建LAMP环境的具体步骤。首先，通过PuTTY工具实现远程连接至服务器。接着，检查当前系统的磁盘空间使用情况，确保有足够的空间进行后续操作，可使用 `df` 命令进行查看。此外，文章还涵盖了安装和配置Apache、MySQL和PHP的相关步骤，以及常见问题的解决方法，帮助用户顺利完成LAMP环境的搭建。 ... [详细]
  • utf-8

    一个Tomcat配置多个端口

    一、Tomcat安装后本身提供了一个server，端口配置默认是8080，对应目录为：..\Tomcat8.0\webapps二、Tomcat8.0配置多个端口，其实也就是给T ... [详细]
  • php

    秒建一个后台管理系统？用这5个开源免费的Java项目就够了

    秒建一个后台管理系统？用这5个开源免费的Java项目就够了
    秒建一个后台管理系统？用这5个开源免费的Java项目就够了 ... [详细]
  • sum

    Kafka安装与配置深度解析：程序员必备指南

    Kafka 是由 Apache 软件基金会开发的高性能分布式消息系统，支持高吞吐量的发布和订阅功能，主要使用 Scala 和 Java 编写。本文将深入解析 Kafka 的安装与配置过程，为程序员提供详尽的操作指南，涵盖从环境准备到集群搭建的每一个关键步骤。 ... [详细]
  • main

    Hadoop 2.6 日志文件解析与MapReduce日志管理深入探讨

    Hadoop 2.6 主要由 HDFS 和 YARN 两大部分组成，其中 YARN 包含了运行在 ResourceManager 的 JVM 中的组件以及在 NodeManager 中运行的部分。本文深入探讨了 Hadoop 2.6 日志文件的解析方法，并详细介绍了 MapReduce 日志管理的最佳实践，旨在帮助用户更好地理解和优化日志处理流程，提高系统运维效率。 ... [详细]
  • utf-8

    短视频app源码，Android开发底部滑出菜单

    短视频app源码，Android开发底部滑出菜单首先依赖三方库implementationandroidx.appcompat:appcompat:1.2.0im ... [详细]
  • php

    如何查看PHP网站及其源码

    本文介绍了如何查看PHP网站及其源码的方法，包括环境搭建、本地测试、源码查看和在线查找等步骤。 ... [详细]
  • utf-8

    使用Python爬取妙笔阁小说信息并保存为TXT和CSV格式

    本文介绍了如何使用Python爬取妙笔阁小说网仙侠系列中所有小说的信息，并将其保存为TXT和CSV格式。主要内容包括如何构造请求头以避免被网站封禁，以及如何利用XPath解析HTML并提取所需信息。 ... [详细]
  • utf-8

    CentOS 7 中 MySQL 5.6 启动时遇到权限问题

    在 CentOS 7 环境中使用 MySQL 5.6 镜像启动数据库时遇到权限问题，本文将详细探讨并提供解决方案。 ... [详细]
  • sum

    在Linux系统上彻底卸载Zimbra邮件系统

    本文详细介绍了如何在Linux系统（以CentOS为例）上彻底卸载Zimbra邮件系统，包括停止服务、删除文件和用户等步骤。 ... [详细]
  • utf-8

    php更新数据库字段的函数是,php更新数据库字段的函数是

    php更新数据库字段的函数是,php更新数据库字段的函数是 ... [详细]
  • ip

    在CentOS系统中部署与配置ZooKeeper详解

    在CentOS系统中部署与配置ZooKeeper详解 ... [详细]
  • main

    深入解析OpenCV中的人脸检测算法实现

    本文详细探讨了OpenCV中人脸检测算法的实现原理与代码结构。通过分析核心函数和关键步骤，揭示了OpenCV如何高效地进行人脸检测。文章不仅提供了代码示例，还深入解释了算法背后的数学模型和优化技巧，为开发者提供了全面的理解和实用的参考。 ... [详细]
  • ip

    Twitter架构深度解析与学习心得

    作为140字符的开创者，Twitter看似简单却异常复杂。其简洁之处在于仅用140个字符就能实现信息的高效传播，甚至在多次全球性事件中超越传统媒体的速度。然而，为了支持2亿用户的高效使用，其背后的技术架构和系统设计则极为复杂，涉及高并发处理、数据存储和实时传输等多个技术挑战。 ... [详细]
  • ip

    如何正确配置与使用日志组件：Log4j、SLF4J及Logback的连接与整合方法

    如何正确配置与使用日志组件：Log4j、SLF4J及Logback的连接与整合方法
    在当前的软件开发实践中，无论是开源项目还是日常工作中，日志框架都是不可或缺的工具之一。本文详细探讨了如何正确配置与使用Log4j、SLF4J及Logback这三个流行的日志组件，并深入解析了它们之间的连接与整合方法，旨在帮助开发者高效地管理和优化日志记录流程。 ... [详细]
author-avatar
手机用户2502861455
这个家伙很懒，什么也没留下！
Tags | 热门标签
RankList | 热门文章
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有