ambari2.8+ambari-metrics3.0+bigtop3.2编译、打包、安装
- bigtop编译
- centos 真机编译branch-3.2
- docker-centos 编译
- bigtop 源码分析
bigtop编译
资源说明:
软件及代码镜像
- apache 代码及软件发布版本资源下载
- 历史包(几乎所有版本):https://archive.apache.org/dist/
- 历史包(最近版本):https://apache.osuosl.org/
- 国内历史包(几乎所有版本):https://repo.huaweicloud.com/apache/
- 国内历史包(最近版本):https://mirrors.aliyun.com/apache/
- 腾讯云:https://mirrors.cloud.tencent.com/
- 清华大学镜像站: https://mirrors.tuna.tsinghua.edu.cn/
- 开发语言及相关软件安装包国内镜像
- gradle
- nodejs:
- npm
开发包镜像
-
jar包[maven镜像]:
- node包[npm、yarn镜像]
- 国内镜像:https://registry.npmmirror.com
- 使用:
- npm config set registry https://registry.npmmirror.com
- yarn config set registry https://registry.npmmirror.com
- bower config set registry https://registry.npmmirror.com
- bower:
- 用途:1)可以直接从npm镜像下载包并进行安装 2) 可以git clone github发布分支,进行安装
- 了解参考:https://www.oschina.net/p/bower?hmsr=aladdin1e1
- 国外地址:https://bower.herokuapp.com
- 国内可访问镜像地址:https://registry.bower.io
- bower内置命令:
- 查看git分支: git ls-remote --tags --heads https://github.com/components/ember.git
github访问
修改hosts文件,以便访问github; https://www.ip138.com/ 通过查询域名,找到对应ip,将域名和ip填写到hosts
140.82.112.4 github.com
199.232.69.194 github.global.ssl.fastly.net
185.199.108.133 raw.githubusercontent.com
185.199.109.133 raw.githubusercontent.com
185.199.110.133 raw.githubusercontent.com
185.199.111.133 raw.githubusercontent.com
编译相关知识
技术知识
- maven命令参数须知:
- 各种不同的跳过
- -Drat.skip=true :RAT 插件用于检查源代码中的许可证问题,确保所有的文件都有适当的许可证声明;此参数跳过许可检查
- -Dmaven.test.skip=true : 跳过测试编译和测试执行阶段。即它不仅会跳过测试的执行,还会跳过测试代码的编译。
- -DskipTests:Maven 跳过测试的执行,但会编译测试代码。
- 有的组件(如flink大部分子项目)需要编译测试代码,有的编译不需要编译测试代码(io.confluent相关)。
- maven日志打印时间:
-Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS
- mavrn编译失败,处理完失败原因(如网络不通),继续从上次失败的模块编译。
mvn <args> -rf: xxxx
,起哄xxx为上次失败的模块名称
- 各种不同的跳过
bigtop编译流程及经验总结
- 目录说明
- bigtop.bom: 定义了各个组件的版本
- package.gradle: 定义了gradle编译组件的整体流程
- 组件配置目录(简称packages):bigtop\bigtop-packages\src\common\XXX\,其中xxx为组件名称,
- do-component-build:一般在此将maven命令包装为gradle命令
- 组件源码包下载目录(简称dl):bigtop\dl\xxx,其中xxx组件名称
- 组件编译目录(简称build):bigtop/build/xxx其中xxx组件名称
- rpm/SOURCES: 组件配置目录下的文件都将被copy到此,另外组件源码下载目录下的源码包也被copy到此,
- rpm/BUILD:组件源码将被解压在此,并进行编译;(注意每次编译时,此目录会重新生成,故编译前更改此目录文件无效)
- rpm/RPMS:组件打包后的rpm包存放位置;若此位置有文件,gradle将跳过编译
- rpm/SRPMS:srpm包存放位置
- tar:存放源码tar包
- 组件下载编译流程:建议详细阅读package.gradle
- 文件流转:dl/spark-xxx [tar.gz|zip|tgz] -> build/spark/rpm/SOURCES/ [tar.gz&sourcecode] -> build/spark/rpm/BUILD/spark/spark-xxx[sourcecode] -> build/spark/rpm/RPMS/[src.rpm] -> output/spark/ [src.rpm]
-
- 下载组件源码包到dl目录,若源码包或者build/xxx/.download已存在,则不再重新下载(可以利用此规则事先下载,节省时间,或者更改源码后,重新打包)
-
- 将dl目录下的源码压缩包解压到build目录;将packages目录的配置信息应用到build目录。(只要没有编译成功,会重复此步骤,故修改解压后的代码无效,会被解压覆盖)
-
- 在build目录进行编译;若发现配置没有生效,则需要删除build目录,重新执行xxx-rpm gradle命令;如npm、yarn、bower镜像配置没有生效的时候
- 编译经验
- package目录中,编译前进行相关配置,尤其是更改为国内镜像。
- 源码包事先下载后放到dl中,部分压缩包需要更改代码,重新打包
- 若是前端项目编译失败,如tez-ui,可以定位到目录,执行node包安装命令,如 npm install,yarn install,bower install
- 注意:请勿使用全局npm、yarn、install,定位到局部的,然后跟上全目录执行
- 若是maven编译失败,处理失败原因后,优先使用 -rf: xxx,进行本模块及后续模块的编译,等maven在整体编译成功后,再使用gradle编译整个组件
- 注意maven具体命令可以通过查看gradle编译日志中的获取
- gradle 编译时,将日志保留下来,以便失败后,查看原因及搜索maven、和前端编译命令及局部npm、yarn、bower具体位置
- 并行编译:bigtop 3.2.0 貌似不支持并行编译,3.3.0 才支持
--parallel --parallel-threads=N
- 记录日志,以方便获取mvn命令及问题排查:
- gradle编译:
./gradlew tez-rpm -PparentDir=/usr/bigtop >> tez-rpm.log 2>> tez-rpm.log
- maven编译(请以gradle日志中的mvn命令为准):
mvn package install -DskipTests -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS" >> /soft/code/bigtop/spark-rpm.log 2>> /soft/code/bigtop/spark-rpm.log
- 前端编译:
npm install
yarn install
bower install --allow-root
- gradle编译:
各模块编译难度及大概耗时(纯编译耗时,不包含下载文件和排错时间)
- 组件名称 编译难度(5星) 纯编译耗时
- bigtop-ambari-mpack * (16秒)[成功]
- bigtop-groovy * (5秒)[成功]
- bigtop-jsvc * (5秒)[成功]
- bigtop-select * (5秒)[成功]
- bigtop-utils * (5秒)[成功]
- hadoop *** (1小时)[成功、无日志]
- zookeeper * (6分钟)[成功、无日志]
- hive * (1小时) [成功]
- hbase * (30分钟)[成功]
- phoenix * (14分钟)[成功]
- tez *** (4分钟)[成功]
- ranger (无此项)
- solr *** (5分钟)[成功]
- kafka *** (5分钟)[成功]
- spark ***** (2小时)[成功,去掉本地R环境,但是sparkR不去掉]
- flink ***** (42分钟)[]
- zeppelin ()[]
centos 真机编译branch-3.2
- 参考(强烈建议先通读以下文章,以便了解)
- BigTop3.2.0 大数据组件编译--基础环境准备:https://blog.csdn.net/m0_48319997/article/details/128037302
- BigTop3.2.0 大数据组件编译--组件编译 https://blog.csdn.net/m0_48319997/article/details/128101667
- ambari2.8.0+bigtop3.2.0发行版大数据平台编译指南:https://blog.csdn.net/m0_48319997/article/details/130046296
- Ambari+Bigtop大数据平台安装部署指南(Centos7)一:https://blog.csdn.net/m0_48319997/article/details/130069050
- Ambari+Bigtop大数据平台安装部署指南(Centos7)二:https://blog.csdn.net/m0_48319997/article/details/130233526
硬件说明:
- 联想笔记本W540:内存32G CPU4核8线程 固态硬盘1T
- 编译环境:vmware centos 7.9 2核4线程(占整体物理机的一半) ,内存 12G,硬盘500G(建议至少100G,详见编译后磁盘占用)
- 说明:centos 的虚拟机文件放在固态硬盘分区中,以加快响应速度
编译步骤
下载代码并切换分支
git clone https://gitee.com/piaolingzxh/bigtop.git
cd bigtop/
git checkout -b branch-3.2 origin/branch-3.2
- 说明:
- 本此编译时间:2024-7-18~29日
- branch-3.2 分支会不断提交代码,你获取的代码可能比我的新;编译时,可能已经修复部分问题,但也可能出现新的问题。
- 当前branch-3.2最新提交为2024-06-02。git hashcode 为3ffe75e05e8428f353a018aafc9c003be72ca6ff
- branch3.2的代码其实已经包含了3.2.0和3.2.1release的代码,也就是比这两个tag的代码更新。有部分commit比master分支还要更新。
国内镜像配置
#修改bigtop/bigtop.bom配置 有两处要修改(两处全改或者改version=bigtop3.2.x对应的)
#1.修改镜像源为国内镜像源 103、104行
APACHE_MIRROR = "https://repo.huaweicloud.com/apache"
APACHE_ARCHIVE = "https://mirrors.aliyun.com/apache"
#2.解开bigtop-select组件的注释 删除273、281行
注意:部分材料可能在aliyun上照不到,需要换回原始地址
APACHE_MIRROR = "https://apache.osuosl.org"
APACHE_ARCHIVE = "https://archive.apache.org/dist"
基础环境准备
依赖环境安装(yum)
#安装组件编译所需的依赖
#1.hadoop依赖
yum -y install fuse-devel cmake cmake3 lzo-devel openssl-devel protobuf* cyrus-*
#cmake默认版本改为cmake3
mv /usr/bin/cmake /usr/bin/cmake.bak
ln -s /usr/bin/cmake3 /usr/bin/cmake
#2.zookeeper依赖
yum -y install cppunit-devel
#3.spark依赖
yum -y install R* harfbuzz-devel fribidi-devel libcurl-devel libxml2-devel freetype-devel libpng-devel libtiff-devel libjpeg-turbo-devel pandoc* libgit2-devel
# 我的Rscript没有安装,且spark编译时跳过了本地R语言的make
Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival'), repos='http://mirrors.tuna.tsinghua.edu.cn/CRAN/')"
Rscript -e "install.packages(c('devtools'), repos='http://mirrors.tuna.tsinghua.edu.cn/CRAN/')"
Rscript -e "install.packages(c('evaluate'), repos='https://mirrors.bfsu.edu.cn/CRAN/')"
# 本地makeR环境报错如下
package 'evaluate' is not available (for R version 3.6.0)
dependency 'evaluate' is not available for package 'knitr'
Rscript -e "install.packages(c('knitr'), repos='http://mirrors.tuna.tsinghua.edu.cn/CRAN/')"
Rscript -e "install.packages(c('evaluate'), repos='http://mirrors.tuna.tsinghua.edu.cn/CRAN/')"
依赖环境配置
- github相关hosts
140.82.112.4 github.com
199.232.69.194 github.global.ssl.fastly.net
185.199.108.133 raw.githubusercontent.com
185.199.109.133 raw.githubusercontent.com
185.199.110.133 raw.githubusercontent.com
185.199.111.133 raw.githubusercontent.com
国内镜像配置|软件全局配置
- maven镜像配置v 注意:~/.m2/settings.xml优先级比程序目录配置优先级高
- node镜像配置: ~/.npmrc , npm config set registry https://registry.npmmirror.com
- yarn镜像配置: ~/.yarnrc
- bower镜像配置: ~/.bowerrc
~/.m2/settings.xml中mirrors节点下配置以下信息,
<mirror>
<id>aliyunmaven</id>
<mirrorOf>*</mirrorOf>
<name>阿里云公共仓库</name>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
<mirror>
<id>aliyunspring</id>
<mirrorOf>*</mirrorOf>
<name>阿里云公共仓库</name>
<url>https://maven.aliyun.com/repository/spring</url>
</mirror>
# ~/.npmrc
registry=https://registry.npmmirror.com/
strict-ssl=false
~/.yarnrc
registry "https://registry.npmmirror.com/"
sass_binary_site "https://registry.npmmirror.com/node-sass/"
phantomjs_cdnurl "https://registry.npmmirror.com/phantomjs"
electron_mirror "https://registry.npmmirror.com/electron"
#sqlite3_binary_host_mirror "https://foxgis.oss-cn-shanghai.aliyuncs.com/"
#profiler_binary_host_mirror "https://npm.taobao.org/mirrors/node-inspector/"
#chromedriver_cdnurl "https://cdn.npm.taobao.org/dist/chromedriver"
# ~/.bowerrc
{
"directory": "bower_components",
"registry": "https://registry.bower.io",
"analytics": false,
"resolvers": [
"bower-shrinkwrap-resolver-ext"
],
"strict-ssl": false
}
修改部分组件源代码
下载组件源码
#1.先下载
./gradlew tez-download zeppelin-download flink-download spark-download
#2.进入下载目录
cd dl
#3.解压这几个tar
tar -zxvf hadoop-3.3.6.tar.gz
tar -zxvf apache-tez-0.10.2-src.tar.gz
tar -zxvf spark-3.2.3.tar.gz
tar -zxvf flink-1.15.3.tar.gz
tar -zxvf zeppelin-0.10.1.tar.gz
修改代码
- 说明:修改代码请再转到对应组件编译节点,编译节点可能会有细节补充
修改hadoop代码
-- dl/hadoop-3.3.6-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml,在yarn install和bower innstall 之间添加一个节点如下
<execution>
<phase>generate-resources</phase>
<id>bower install moment-timezone</id>
<configuration>
<arguments>install moment-timezone=https://github.com/moment/moment-timezone.git#=0.5.1 --allow-root</arguments>
</configuration>
<goals>
<goal>bower</goal>
</goals>
</execution>
修改flink代码
1)node、npm版本及相关设置
vi flink-1.15.0/flink-runtime-web/pom.xml
在275行 nodeVersion改为v12.22.1
在276行 npmVersion改为6.14.12
bigtop/dl/flink-1.15.3/flink-runtime-web/pom.xml
<arguments>ci --cache-max=0 --no-save ${npm.proxy}</arguments> 此行删除,更改为下一行,注意没了ci
<arguments> install -g --registry=https://registry.npmmirror.com --cache-max=0 --no-save</arguments>
2)注释掉报错的测试代码
cd dl/flink-1.15.3/flink-formats/flink-avro-confluent-registry/src/test/java/org/apache/flink/formats/avro/registry/confluent/
mv CachedSchemaCoderProviderTest.java CachedSchemaCoderProviderTest.java1
mv RegistryAvroFormatFactoryTest.java RegistryAvroFormatFactoryTest.java1
cd dl/flink-1.15.3/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/
mv SQLClientSchemaRegistryITCase.java SQLClientSchemaRegistryITCase.java1
3)io.confluent包无法下载的问题,手动下载,并安装到本地仓库()
wget https://packages.confluent.io/maven/io/confluent/common-config/6.2.2/common-config-6.2.2.jar ./
wget https://packages.confluent.io/maven/io/confluent/common-utils/6.2.2/common-utils-6.2.2.jar ./
wget https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/6.2.2/kafka-avro-serializer-6.2.2.jar ./
wget http://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/6.2.2/kafka-schema-registry-client-6.2.2.jar ./
# 安装jar包到本地仓库
mvn install:install-file -Dfile=/soft/ambari-develop/common-config-6.2.2.jar -DgroupId=io.confluent -DartifactId=common-config -Dversion=6.2.2 -Dpackaging=jar
mvn install:install-file -Dfile=/soft/ambari-develop/common-utils-6.2.2.jar -DgroupId=io.confluent -DartifactId=common-utils -Dversion=6.2.2 -Dpackaging=jar
mvn install:install-file -Dfile=/soft/ambari-develop/kafka-avro-serializer-6.2.2.jar -DgroupId=io.confluent -DartifactId=kafka-avro-serializer -Dversion=6.2.2 -Dpackaging=jar
mvn install:install-file -Dfile=/soft/ambari-develop/kafka-schema-registry-client-6.2.2.jar -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=6.2.2 -Dpackaging=jar
npm install -g @angular/[email protected]
修改tez代码
vi apache-tez-0.10.2-src/tez-ui/pom.xml
在37行 allow-root-build改为--allow-root=true
bower国内镜像配置:https://bower.herokuapp.com 改为https://registry.bower.io
- 涉及到的文件:
- bigtop\bigtop-packages\src\common\ambari\patch13-AMBARI-25946.diff
- bigtop\bigtop-packages\src\common\tez\patch6-TEZ-4492.diff
修改zeppelin代码
vi zeppelin-0.10.1/pom.xml
在209行plugin.gitcommitid.useNativeGit改为true
vi zeppelin-0.10.1/spark/pom.xml
在50行spark.src.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}.tgz
在53行spark.bin.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}-bin-without-hadoop.tgz
vi zeppelin-0.10.1/rlang/pom.xml
在41行spark.src.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}.tgz
在44行spark.bin.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}-bin-without-hadoop.tgz
vi zeppelin-0.10.1/flink/flink-scala-parent/pom.xml
在45行flink.bin.download.url改为https://repo.huaweicloud.com/apache/flink/flink-${flink.version}/flink-${flink.version}-bin-scala_${flink.scala.binary.version}.tgz
修改spark源码
cd bigtop/dl
vim spark-3.2.3/dev/make-distribution.sh
#BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
BUILD_COMMAND=("$MVN" clean package -DskipTests -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS" $@) #此行中添加打印时间参数
tar zcvf spark-3.2.3.tar.gz spark-3.2.3
重新打包组件源码
- 注意:打包后的压缩包,应与解压前的压缩包同名(含后缀)
tar zcvf hadoop-3.3.6.tar.gz hadoop-3.3.6-src
tar zcvf apache-tez-0.10.2-src.tar.gz apache-tez-0.10.2-src
tar zcvf spark-3.2.3.tar.gz spark-3.2.3
tar zcvf zeppelin-0.10.1.tar.gz zeppelin-0.10.1
tar zcvf flink-1.15.3.tar.gz flink-1.15.3
整体编译[不建议]
- 预计需要N小时以上,且中间可能报各种错误,建议逐个组件进行编译,等所有组件编译成功之后,再整体编译
- 编译参数
- allclean
- -PparentDir=/usr/bigtop :打出来的rpm包的默认安装位置,相当于ambari默认安装路径
/usr/hdp
- -Dbuildwithdeps=true: 编译时,若依赖组件没有编译,先编译依赖组件
- -PpkgSuffix:
- 编译时的bigtop版本,类似于hdp版本号,如3.2.0,3.2.1,3.3.0;
- 此版本号将体现在打包后的文件名重,编译时务必带上此参数,否则编译后,ambari找不到对应的包。
- 示例包名称:hadoop_3_2_1-hdfs-namenode-3.3.6-1.el7.x86_64.rpm,其中3_2_1代表bigtop版本号
- $component-rpm: component 代表组件名称,如spark,hive,hbase等大数据组件
- allclean: 删除build/*、output/、/dist,请慎重操作,重新编译会极其耗时,且由于被墙,可能出现以前没出现的错误。建议使用此命令前,先将此三个目录手动备份
特别强调:
编译时一定要添加-PpkgSuffix,否则打出的包不带bigtop版本号,后期安装大数据组件的时候,ambari无法识别,具体报错如下:
Error getting repository data for BIGTOP-3.2.0, repository not found
No package found for hadoop_${stack_version}(expected name: hadoop_3_2_0)
整体编译命令(在bigtop根目录执行)
#
./gradlew bigtop-groovy-rpm bigtop-jsvc-rpm bigtop-select-rpm bigtop-utils-rpm flink-rpm hadoop-rpm hbase-rpm hive-rpm kafka-rpm solr-rpm spark-rpm tez-rpm zeppelin-rpm zookeeper-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-full.log 2>> logs/bigtop-full.log
逐步逐组件编译
先编译一些周边依赖组件:
- bigtop-ambari-mpack-rpm
- bigtop-groovy-rpm
- bigtop-jsvc-rpm
- bigtop-select-rpm
- bigtop-utils-rpm
./gradlew bigtop-ambari-mpack-rpm bigtop-groovy-rpm bigtop-jsvc-rpm bigtop-select-rpm bigtop-utils-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-dep.log 2>> logs/bigtop-dep.log
zookeeper(子项目21个-6分钟)
./gradlew zookeeper-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-zookeeper.log 2>> logs/bigtop-zookeeper.log
hadoop(子项目111个-48分+12分钟)
- gradle编译命令:
./gradlew hadoop-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-hadoop.log 2>> logs/bigtop-hadoop.log
- maven编译命令示例:
cd /soft/code/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.6-src
mvn -Pdist -Pnative -Psrc -Pyarn-ui -Dtar -Dzookeeper.version=3.6.4 -Dhbase.profile=2.0 -DskipTests -DskipITs install -rf :hadoop-yarn-ui >> /soft/code/bigtop/logs/bigtop-hadoop-mvn.log 2>> /soft/code/bigtop/logs/bigtop-hadoop-mvn.log
问题点-解决方法:
- 1)cmake3:见上边基础环境配置
- 2)hadoop-yarn-ui编译
- hadoop-yarn-ui:pom.xml -> 添加:bower install moment-timezone=https://hub.nuaa.cf/moment/moment-timezone.git#=0.5.1 --allow-root
- 3)bower要访问github.com,注意确认网络通畅
具体操作:
- 1)hadoop-yarn-ui编译时,测试是否联通github
git ls-remote --tags --heads https://github.com/moment/moment-timezone.git
可以使用此命令测试是否能够正常访问github分支信息- 2)编译时,若moment-timezone#0.5.1或者ember#2.8.0无法安装,报ENORES component No version found that was able to satisfy
- 则在hadoop-yarn-ui:pom.xml -> 在bow install命令之前添加:
bower install moment-timezone=https://hub.nuaa.cf/moment/moment-timezone.git#=0.5.1 --allow-root
,详见hadoop代码修改 - 前端命令位置:
- node、yarn、bower命令执行的位置(以日志中为准):bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.6-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/component/webapp
- 局部node位置:webapp/node/yarn/dist/bin/yarn install
- 局部bower位置:webapp/node_modules/bower/bin/bower,若在root账户下编译执行,需要添加 --allow-root
hbase(子项目44个-11分+5分钟+10分)
./gradlew hbase-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-hbase.log 2>> logs/bigtop-hbase.log
hive(子项目42个+9分钟)
./gradlew hive-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-hive.log 2>> logs/bigtop-hive.log
phoenix(子项目15个-14分钟)
./gradlew bigtop-ambari-mpack-rpm phoenix-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-phoenix.log 2>> logs/bigtop-phoenix.log
tez(子项目29个-4分钟)
./gradlew tez-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-tez.log 2>> logs/bigtop-tez.log
问题点|解决方法:
- 1)root编译,apache-tez-0.10.2-src/tez-ui/pom.xml ,在37行 allow-root-build改为--allow-root=true
- 2)bower镜像:https://bower.herokuapp.com 改为https://registry.bower.io
- 涉及到的文件:
- bigtop\bigtop-packages\src\common\ambari\patch13-AMBARI-25946.diff
- bigtop\bigtop-packages\src\common\tez\patch6-TEZ-4492.diff
- 涉及到的文件:
- 3)证书过期:codemirror#5.11.0 certificate has expired
- export BOWER_STRCT_SSL=false
- bower全局配置:"strct_ssl" = false ,详细见上
报错信息:
- 1)bower ember-cli-shims#0.0.6 ECONNREFUSED Request to https://bower.herokuapp.com/packages/ember-cli-shims failed: connect ECONNREFUSED 108.160.169.186:443
- 解决方法:bower国内镜像设置,见上
- 2)bower codemirror#5.11.0 CERT_HAS_EXPIRED Request to https://registry.bower.io/packages/codemirror failed: certificate has expired
- 解决方法:证书过期设置,见上
spark(子项目29个-49分+62分)
- gradle 命令
./gradlew spark-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-spark.log 2>> logs/bigtop-spark.log
- maven命令(请以日志中命令为准):
mvn clean package -DskipTests -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.6 -Dyarn.version=3.3.6 -Pyarn -Phadoop-3.2 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS" >> /soft/code/bigtop/spark-rpm-mvn.log 2>> /soft/code/bigtop/spark-rpm-mvn.log
- 说明:gradle先调用spark源码解压目录下dev/make-distribution.sh进行mvn clean package;然后再执行maven install,花费时间为单次的2倍。纯编译时长总共大约在150分钟左右
问题点|解决方法
- 1)SparkR能够正常打包,但是本地MAKE_R一直失败,可以考虑取消本地make_r
- 解决方法:vim bigtop\bigtop-packages\src\common\spark\do-component-build 中./dev/make-distribution.sh --mvn mvn --r $BUILD_OPTS -DskipTests # 此行中去掉 --r
- 2)scala、java混合编译耗时较长,maven编译日志,显示时间
- vim dl/spark-3.2.3/dev/make-distribution.sh
-
BUILD_COMMAND=("$MVN" clean package -DskipTests $@) 注释掉本行
- BUILD_COMMAND=("$MVN" clean package -DskipTests -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS" $@) #此行中添加打印时间参数
- vim bigtop\bigtop-packages\src\common\spark\do-component-build
- mvn $BUILD_OPTS install -DskipTests=$SPARK_SKIP_TESTS -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat="yyyy-MM-dd HH:mm:ss.SSS" #此行中添加打印时间参数
报错信息
- 1)Error in loadVignetteBuilder(pkgdir, TRUE) : vignette builder 'knitr' not found
- 解决方法:去掉本地make_r
kafka(5分钟)
./gradlew kafka-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-kafka.log 2>> logs/bigtop-kafka.log
注意点:
- 1)访问:raw.githubusercontent.com
- 解决方法:hosts文件配置(见上)
flink(子项目207个-42分钟)
-
./gradlew flink-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-flink.log 2>> logs/bigtop-flink.log
-
注意:此项目不能跳过测试代码编译
问题点|解决方法: -
1)io.confluent相关jar包无法下载,
-
解决方法:手动下载,并注册到本地仓库,详见上
-
2)测试代码出错:
- CachedSchemaCoderProviderTest.java
- RegistryAvroFormatFactoryTest.java
- SQLClientSchemaRegistryITCase.java
-
- ndoe、npm 下载失败,
- Downloading https://nodejs.org/dist/v16.13.2/node-v16.13.2-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
- Downloading https://registry.npmjs.org/npm/-/npm-8.1.2.tgz to /root/.m2/repository/com/github/eirslett/npm/8.1.2/npm-8.1.2.tar.gz
- 解决方案:手动下载,并放到下载目录(注意后缀名保持一致)
- mv /home/zxh/soft/ambari-develop/node-v16.13.2-linux-x64.tar.gz /root/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
- mv /home/zxh/soft/ambari-develop/npm-8.1.2.tgz /root/.m2/repository/com/github/eirslett/npm/8.1.2/npm-8.1.2.tar.gz
-
solr
./gradlew solr-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-solr.log 2>> logs/bigtop-solr.log
- 注意点:
- 1)此项目使用ivy编译,强烈建议配置国内源
zeppelin
./gradlew zeppelin-rpm -Dbuildwithdeps=true -PparentDir=/usr/bigtop -PpkgSuffix -x test >> logs/bigtop-zeppelin.log 2>> logs/bigtop-zeppelin.log
#下载zeppelin源码包
./gradlew zeppelin-download
#解压zeppelin源码
cd dl
tar -zxvf zeppelin-0.10.1.tar.gz
#修改pom文件
vi zeppelin-0.10.1/pom.xml
在209行plugin.gitcommitid.useNativeGit改为true
vi zeppelin-0.10.1/spark/pom.xml
在50行spark.src.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}.tgz
在53行spark.bin.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}-bin-without-hadoop.tgz
vi zeppelin-0.10.1/rlang/pom.xml
在41行spark.src.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}.tgz
在44行spark.bin.download.url改为https://repo.huaweicloud.com/apache/spark/${spark.archive}/${spark.archive}-bin-without-hadoop.tgz
vi zeppelin-0.10.1/flink/flink-scala-parent/pom.xml
在45行flink.bin.download.url改为https://repo.huaweicloud.com/apache/flink/flink-${flink.version}/flink-${flink.version}-bin-scala_${flink.scala.binary.version}.tgz
#重新打包zeppelin源码
tar -zcvf zeppelin-0.10.1.tar.gz zeppelin-0.10.1
#编译
./gradlew zeppelin-rpm -PparentDir=/usr/bigtop
问题:
[INFO] Downloading https://registry.npmjs.org/npm/-/npm-6.9.0.tgz to /root/.m2/repository/com/github/eirslett/npm/6.9.0/npm-6.9.0.tar.gz
I/O exception (java.net.SocketException) caught when processing request to {s}->https://registry.npmjs.org:443: Connection reset
解决:去阿里云、华为云下载,放到上述目标位置,注意文件重命名
https://mirrors.aliyun.com/macports/distfiles/npm6/npm-6.9.0.tgz
mvn -Dhadoop3.2.version=3.3.6 -Dlivy.version=0.7.1-incubating -Pscala-2.11 -Phadoop3 -Pbuild-distr -DskipTests clean package -pl '!beam,!hbase,!pig,!jdbc,!flink,!ignite,!kylin,!lens,!cassandra,!elasticsearch,!bigquery,!alluxio,!scio,!groovy,!sap,!java,!geode,!neo4j,!hazelcastjet,!submarine,!sparql,!mongodb,!ksql,!scalding' -am -rf :zeppelin-client-examples >> /soft/code/bigtop/logs/bigtop-zeppelin.log 2>> /soft/code/bigtop/logs/bigtop-zeppelin.log
问题:Bower resolver not found: bower-shrinkwrap-resolver-ext
https://www.cnblogs.com/jameBo/p/10615444.html
编译后磁盘占用
- 磁盘占用:
- bigtop 16G
- ambari 2.1G
- ambari 6G
- 公共部分磁盘占用
- .m2: 16G
- .ant 2M
- .ivy2 200M
- .gradle 1.2g
- .nvm 75m
- .npm 400M
- .cache 400M
- 总计磁盘占用:50G+
branch-3.2 各组件编译耗时[纯编译时间,不含下载相关包时间]
- 注:不出错的情况下
zookeeper(21个子项目-6分钟-ok)
mvn clean install -DskipTests -Pfull-build
[INFO] Reactor Summary for Apache ZooKeeper 3.6.4:
[INFO]
[INFO] Apache ZooKeeper ................................... SUCCESS [ 19.276 s]
[INFO] Apache ZooKeeper - Documentation ................... SUCCESS [ 13.275 s]
[INFO] Apache ZooKeeper - Jute ............................ SUCCESS [ 24.500 s]
[INFO] Apache ZooKeeper - Server .......................... SUCCESS [ 40.075 s]
[INFO] Apache ZooKeeper - Metrics Providers ............... SUCCESS [ 9.652 s]
[INFO] Apache ZooKeeper - Prometheus.io Metrics Provider .. SUCCESS [ 11.991 s]
[INFO] Apache ZooKeeper - Client .......................... SUCCESS [ 9.944 s]
[INFO] Apache ZooKeeper - Client - C ...................... SUCCESS [ 43.346 s]
[INFO] Apache ZooKeeper - Recipes ......................... SUCCESS [ 9.404 s]
[INFO] Apache ZooKeeper - Recipes - Election .............. SUCCESS [ 9.763 s]
[INFO] Apache ZooKeeper - Recipes - Lock .................. SUCCESS [ 9.288 s]
[INFO] Apache ZooKeeper - Recipes - Queue ................. SUCCESS [ 9.637 s]
[INFO] Apache ZooKeeper - Assembly ........................ SUCCESS [ 19.397 s]
[INFO] Apache ZooKeeper - Compatibility Tests ............. SUCCESS [ 9.336 s]
[INFO] Apache ZooKeeper - Compatibility Tests - Curator ... SUCCESS [ 8.593 s]
[INFO] Apache ZooKeeper - Tests ........................... SUCCESS [ 13.297 s]
[INFO] Apache ZooKeeper - Contrib ......................... SUCCESS [ 9.006 s]
[INFO] Apache ZooKeeper - Contrib - Fatjar ................ SUCCESS [ 16.370 s]
[INFO] Apache ZooKeeper - Contrib - Loggraph .............. SUCCESS [ 16.161 s]
[INFO] Apache ZooKeeper - Contrib - Rest .................. SUCCESS [ 16.152 s]
[INFO] Apache ZooKeeper - Contrib - ZooInspector .......... SUCCESS [ 19.938 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:39 min
[INFO] Finished at: 2024-07-27T01:06:56+08:00
[INFO] ------------------------------------------------------------------------
hadoop(111个子项目 60分钟-ok)
mvn -Pdist -Pnative -Psrc -Pyarn-ui -Dtar -Dzookeeper.version=3.6.4 -Dhbase.profile=2.0 -DskipTests -DskipITs install
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.6:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 29.205 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.466 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.582 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 6.002 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.466 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.587 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 8.592 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 3.445 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 15.306 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 5.702 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:23 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 17.947 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 16.682 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [ 13.522 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.180 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 52.640 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:12 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [03:42 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 17.544 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 12.443 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 54.272 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.151 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.142 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 30.541 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:04 min]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.151 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 24.234 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [01:04 min]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 11.881 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 14.836 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 12.752 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 46.104 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 9.362 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 16.279 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 11.508 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 11.695 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.116 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 12.285 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 20.129 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.181 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 2.0 SUCCESS [ 15.300 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 12.552 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 13.316 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [ 11.596 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.131 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 13.113 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 9.545 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 6.801 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 14.716 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 18.491 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 11.763 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 22.129 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 16.263 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 17.511 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 10.385 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.100 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 12.803 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 10.791 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [ 0.148 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 42.258 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [ 3.458 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [ 0.125 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [ 9.859 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.161 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 4.323 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [05:15 min]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 26.939 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 31.039 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 9.021 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [01:08 min]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 8.951 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 15.563 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 10.572 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 15.810 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 13.630 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 8.084 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [ 11.176 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [ 12.493 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [ 10.631 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 15.383 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [ 0.080 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 10.879 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 11.712 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 15.125 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 13.488 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 11.425 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 12.758 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 5.375 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 40.051 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 7.605 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 21.646 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 11.321 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 12.938 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 13.073 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 10.130 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 14.060 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 44.517 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 0.169 s]
[INFO] Apache Hadoop Common Benchmark ..................... SUCCESS [ 20.616 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.124 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [02:44 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:33 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 3.377 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [04:11 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 4.710 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 4.934 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:15 min]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.072 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 12.562 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 18.067 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.094 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47:53 min
[INFO] Finished at: 2024-07-27T20:11:32+08:00
[INFO] ------------------------------------------------------------------------
mvn site site:stage -Dzookeeper.version=3.6.4 -Dhbase.profile=2.0 -DskipTests -DskipITs
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.6:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [06:14 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.007 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 2.179 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 0.543 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.190 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.203 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 1.732 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.409 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 2.345 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 0.579 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 55.981 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.518 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.182 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [ 2.197 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.178 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 19.055 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 32.371 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 0.991 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 9.106 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.081 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 5.866 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.104 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.161 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 11.008 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 11.283 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.146 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 5.983 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 8.138 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.694 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 3.020 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.306 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 15.166 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 2.273 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.446 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.712 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.716 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.106 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 1.774 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 2.472 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.123 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 2.0 SUCCESS [ 1.879 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 3.484 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.241 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [ 1.670 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.117 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.072 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.382 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.562 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 7.220 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 3.748 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.503 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 5.192 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 3.178 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 5.272 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.769 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.098 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 3.884 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 2.252 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [ 0.096 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 4.489 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [ 0.142 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [ 0.111 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [ 1.289 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 2.222 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 0.751 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.107 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 5.498 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 2.222 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 2.176 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 2.437 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 1.411 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.802 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 0.645 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.216 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.968 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.283 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [ 1.399 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [ 1.998 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [ 1.356 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 1.456 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [ 0.166 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.999 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.946 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.649 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.565 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.736 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.759 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.097 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 7.156 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.221 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.531 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.532 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 1.862 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 1.758 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.407 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 1.865 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 1.174 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 0.080 s]
[INFO] Apache Hadoop Common Benchmark ..................... SUCCESS [ 1.178 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.083 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [ 1.051 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 1.091 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.127 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [ 1.269 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.141 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.125 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 0.697 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.177 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 1.327 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.698 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.070 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:05 min
[INFO] Finished at: 2024-07-27T20:23:39+08:00
[INFO] ------------------------------------------------------------------------
hbase(44个子项目-30分钟-ok)
mvn -Phadoop-3.0 -Dhadoop-three.version=3.3.6 -Dhadoop.guava.version=27.0-jre -Djetty.version=9.3.29.v20201019 -Dzookeeper.version=3.6.4 -DskipTests -Dcheckstyle.skip=true -Dadditionalparam=-Xdoclint:none clean install
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.4.17:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [ 9.339 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 1.467 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 3.171 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.350 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 1.649 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 41.400 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 18.882 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 7.207 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 7.854 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 6.949 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 9.814 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 14.866 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 19.238 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 8.541 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 8.806 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.763 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 13.134 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 11.537 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 8.098 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 51.306 s]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [ 15.222 s]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [ 12.845 s]
[INFO] Apache HBase - Thrift .............................. SUCCESS [ 21.466 s]
[INFO] Apache HBase - RSGroup ............................. SUCCESS [ 12.444 s]
[INFO] Apache HBase - Shell ............................... SUCCESS [ 14.736 s]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ 13.530 s]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 15.545 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [ 11.912 s]
[INFO] Apache HBase - Examples ............................ SUCCESS [ 15.016 s]
[INFO] Apache HBase - Shaded .............................. SUCCESS [ 0.453 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [ 34.695 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [ 19.313 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [ 30.992 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [ 9.254 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 9.288 s]
[INFO] Apache HBase - Assembly ............................ SUCCESS [ 34.417 s]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [01:19 min]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 12.668 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 13.913 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [ 9.286 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [ 0.113 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 9.375 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 11.940 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [ 0.774 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:57 min
[INFO] Finished at: 2024-07-27T21:08:15+08:00
[INFO] ------------------------------------------------------------------------
mvn -Phadoop-3.0 -Dhadoop-three.version=3.3.6 -Dhadoop.guava.version=27.0-jre -Djetty.version=9.3.29.v20201019 -Dzookeeper.version=3.6.4 -DskipTests -Dcheckstyle.skip=true -Dadditionalparam=-Xdoclint:none site
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.4.17:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [04:46 min]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 0.008 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 0.012 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.007 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 0.010 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 0.006 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 0.023 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 0.019 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 0.018 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 0.015 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 0.020 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 0.005 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 0.028 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 0.027 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 0.029 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.005 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 0.057 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 0.057 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 0.023 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 0.099 s]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [ 0.106 s]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [ 0.105 s]
[INFO] Apache HBase - Thrift .............................. SUCCESS [ 0.094 s]
[INFO] Apache HBase - RSGroup ............................. SUCCESS [ 0.119 s]
[INFO] Apache HBase - Shell ............................... SUCCESS [ 0.116 s]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ 0.076 s]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 0.130 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [ 0.101 s]
[INFO] Apache HBase - Examples ............................ SUCCESS [ 0.140 s]
[INFO] Apache HBase - Shaded .............................. SUCCESS [ 0.004 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [ 0.021 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [ 0.020 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [ 0.050 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [ 0.041 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 0.025 s]
[INFO] Apache HBase - Assembly ............................ SUCCESS [ 0.229 s]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [ 0.088 s]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 0.105 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 0.055 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [ 0.017 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [ 0.003 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 0.094 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 0.090 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [ 0.005 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:51 min
[INFO] Finished at: 2024-07-27T21:13:08+08:00
[INFO] ------------------------------------------------------------------------
mvn -Phadoop-3.0 -Dhadoop-three.version=3.3.6 -Dhadoop.guava.version=27.0-jre -Djetty.version=9.3.29.v20201019 -Dzookeeper.version=3.6.4 -DskipTests -Dcheckstyle.skip=true -Dadditionalparam=-Xdoclint:none package assembly:single
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.4.17:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [ 5.820 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 1.286 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 1.009 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.382 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 1.215 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 42.784 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 11.707 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 6.980 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 7.306 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 8.228 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 8.730 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 9.967 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 10.097 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 7.798 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 7.468 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.290 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 9.759 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 10.864 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 6.501 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 20.587 s]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [ 11.919 s]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [ 13.675 s]
[INFO] Apache HBase - Thrift .............................. SUCCESS [ 14.494 s]
[INFO] Apache HBase - RSGroup ............................. SUCCESS [ 11.399 s]
[INFO] Apache HBase - Shell ............................... SUCCESS [ 13.935 s]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ 12.549 s]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 14.051 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [ 10.351 s]
[INFO] Apache HBase - Examples ............................ SUCCESS [ 14.691 s]
[INFO] Apache HBase - Shaded .............................. SUCCESS [ 0.431 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [ 33.310 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [ 18.827 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [ 30.286 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [ 9.316 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 8.675 s]
[INFO] Apache HBase - Assembly ............................ SUCCESS [02:01 min]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [01:11 min]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 10.106 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 11.550 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [ 8.374 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [ 0.099 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 9.118 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 11.979 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [ 0.576 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:44 min
[INFO] Finished at: 2024-07-27T21:23:53+08:00
[INFO] ------------------------------------------------------------------------
tez(29个子项目-4分钟)
mvn clean package -Dtar -Dhadoop.version=3.3.6 -Phadoop28 -DskipTests
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for tez 0.10.2:
[INFO]
[INFO] tez ................................................ SUCCESS [ 2.515 s]
[INFO] hadoop-shim ........................................ SUCCESS [ 3.211 s]
[INFO] tez-api ............................................ SUCCESS [ 10.015 s]
[INFO] tez-build-tools .................................... SUCCESS [ 0.142 s]
[INFO] tez-common ......................................... SUCCESS [ 2.432 s]
[INFO] tez-runtime-internals .............................. SUCCESS [ 3.497 s]
[INFO] tez-runtime-library ................................ SUCCESS [ 6.568 s]
[INFO] tez-mapreduce ...................................... SUCCESS [ 4.433 s]
[INFO] tez-examples ....................................... SUCCESS [ 2.743 s]
[INFO] tez-dag ............................................ SUCCESS [ 9.771 s]
[INFO] tez-tests .......................................... SUCCESS [ 4.260 s]
[INFO] tez-ext-service-tests .............................. SUCCESS [ 3.605 s]
[INFO] tez-ui ............................................. SUCCESS [01:08 min]
[INFO] tez-plugins ........................................ SUCCESS [ 0.064 s]
[INFO] tez-protobuf-history-plugin ........................ SUCCESS [ 5.118 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [ 5.678 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [ 5.366 s]
[INFO] tez-yarn-timeline-cache-plugin ..................... SUCCESS [ 20.169 s]
[INFO] tez-yarn-timeline-history-with-fs .................. SUCCESS [ 4.842 s]
[INFO] tez-history-parser ................................. SUCCESS [ 20.014 s]
[INFO] tez-aux-services ................................... SUCCESS [ 13.834 s]
[INFO] tez-tools .......................................... SUCCESS [ 0.075 s]
[INFO] tez-perf-analyzer .................................. SUCCESS [ 0.062 s]
[INFO] tez-job-analyzer ................................... SUCCESS [ 3.651 s]
[INFO] tez-javadoc-tools .................................. SUCCESS [ 1.221 s]
[INFO] hadoop-shim-impls .................................. SUCCESS [ 0.048 s]
[INFO] hadoop-shim-2.8 .................................... SUCCESS [ 0.988 s]
[INFO] tez-dist ........................................... SUCCESS [ 41.472 s]
[INFO] Tez ................................................ SUCCESS [ 0.316 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:05 min
[INFO] Finished at: 2024-07-27T22:25:28+08:00
[INFO] ------------------------------------------------------------------------
Phoenix(15个子项目-14分钟)
mvn -DskipTests -Dhadoop.version=3.3.6 -Dhbase.version=2.4.17 -Dhbase.profile=2.4 clean install
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Phoenix 5.1.3:
[INFO]
[INFO] Apache Phoenix ..................................... SUCCESS [ 4.768 s]
[INFO] Phoenix Hbase 2.5.0 compatibility .................. SUCCESS [ 10.642 s]
[INFO] Phoenix Hbase 2.4.1 compatibility .................. SUCCESS [ 4.776 s]
[INFO] Phoenix Hbase 2.4.0 compatibility .................. SUCCESS [ 4.079 s]
[INFO] Phoenix Hbase 2.3.0 compatibility .................. SUCCESS [ 4.855 s]
[INFO] Phoenix Hbase 2.2.5 compatibility .................. SUCCESS [ 3.527 s]
[INFO] Phoenix Hbase 2.1.6 compatibility .................. SUCCESS [ 3.922 s]
[INFO] Phoenix Core ....................................... SUCCESS [01:00 min]
[INFO] Phoenix - Pherf .................................... SUCCESS [ 13.966 s]
[INFO] Phoenix - Tracing Web Application .................. SUCCESS [ 13.353 s]
[INFO] Phoenix Client Parent .............................. SUCCESS [ 0.034 s]
[INFO] Phoenix Client ..................................... SUCCESS [05:16 min]
[INFO] Phoenix Client Embedded ............................ SUCCESS [05:08 min]
[INFO] Phoenix Server JAR ................................. SUCCESS [01:00 min]
[INFO] Phoenix Assembly ................................... SUCCESS [ 26.774 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:58 min
[INFO] Finished at: 2024-07-28T13:05:51+08:00
[INFO] ------------------------------------------------------------------------
spark(子项目29个-2小时-ok)
./dev/make-distribution.sh --mvn mvn -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.6 -Dyarn.version=3.3.6 -Pyarn -Phadoop-3.2 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
->
mvn package -DskipTests -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat=yyyy-MM-dd HH:mm:ss.SSS -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.6 -Dyarn.version=3.3.6 -Pyarn -Phadoop-3.2 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
2024-07-27 23:18:30.330 [INFO] ------------------------------------------------------------------------
2024-07-27 23:18:30.330 [INFO] Reactor Summary for Spark Project Parent POM 3.2.3:
2024-07-27 23:18:30.330 [INFO]
2024-07-27 23:18:30.331 [INFO] Spark Project Parent POM ........................... SUCCESS [ 5.161 s]
2024-07-27 23:18:30.332 [INFO] Spark Project Tags ................................. SUCCESS [ 9.465 s]
2024-07-27 23:18:30.332 [INFO] Spark Project Sketch ............................... SUCCESS [ 10.156 s]
2024-07-27 23:18:30.332 [INFO] Spark Project Local DB ............................. SUCCESS [ 2.971 s]
2024-07-27 23:18:30.333 [INFO] Spark Project Networking ........................... SUCCESS [ 8.322 s]
2024-07-27 23:18:30.333 [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.003 s]
2024-07-27 23:18:30.333 [INFO] Spark Project Unsafe ............................... SUCCESS [ 14.102 s]
2024-07-27 23:18:30.333 [INFO] Spark Project Launcher ............................. SUCCESS [ 2.603 s]
2024-07-27 23:18:30.334 [INFO] Spark Project Core ................................. SUCCESS [04:39 min]
2024-07-27 23:18:30.334 [INFO] Spark Project ML Local Library ..................... SUCCESS [ 52.463 s]
2024-07-27 23:18:30.335 [INFO] Spark Project GraphX ............................... SUCCESS [01:04 min]
2024-07-27 23:18:30.335 [INFO] Spark Project Streaming ............................ SUCCESS [01:57 min]
2024-07-27 23:18:30.336 [INFO] Spark Project Catalyst ............................. SUCCESS [05:23 min]
2024-07-27 23:18:30.336 [INFO] Spark Project SQL .................................. SUCCESS [08:20 min]
2024-07-27 23:18:30.336 [INFO] Spark Project ML Library ........................... SUCCESS [06:02 min]
2024-07-27 23:18:30.336 [INFO] Spark Project Tools ................................ SUCCESS [ 17.453 s]
2024-07-27 23:18:30.336 [INFO] Spark Project Hive ................................. SUCCESS [03:53 min]
2024-07-27 23:18:30.337 [INFO] Spark Project REPL ................................. SUCCESS [ 58.167 s]
2024-07-27 23:18:30.337 [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 14.265 s]
2024-07-27 23:18:30.337 [INFO] Spark Project YARN ................................. SUCCESS [02:02 min]
2024-07-27 23:18:30.338 [INFO] Spark Project Kubernetes ........................... SUCCESS [01:57 min]
2024-07-27 23:18:30.339 [INFO] Spark Project Hive Thrift Server ................... SUCCESS [01:58 min]
2024-07-27 23:18:30.339 [INFO] Spark Project Assembly ............................. SUCCESS [ 6.100 s]
2024-07-27 23:18:30.339 [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 54.621 s]
2024-07-27 23:18:30.340 [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:15 min]
2024-07-27 23:18:30.341 [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [02:21 min]
2024-07-27 23:18:30.341 [INFO] Spark Project Examples ............................. SUCCESS [01:34 min]
2024-07-27 23:18:30.341 [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 14.814 s]
2024-07-27 23:18:30.341 [INFO] Spark Avro ......................................... SUCCESS [01:54 min]
2024-07-27 23:18:30.341 [INFO] ------------------------------------------------------------------------
2024-07-27 23:18:30.342 [INFO] BUILD SUCCESS
2024-07-27 23:18:30.342 [INFO] ------------------------------------------------------------------------
2024-07-27 23:18:30.342 [INFO] Total time: 49:01 min
2024-07-27 23:18:30.343 [INFO] Finished at: 2024-07-27T23:18:30+08:00
2024-07-27 23:18:30.343 [INFO] ------------------------------------------------------------------------
mvn -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.6 -Dyarn.version=3.3.6 -Pyarn -Phadoop-3.2 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre install -DskipTests=true -Dorg.slf4j.simpleLogger.showDateTime=true '-Dorg.slf4j.simpleLogger.dateTimeFormat=yyyy-MM-dd HH:mm:ss.SSS'
2024-07-28 00:20:33.546 [INFO] ------------------------------------------------------------------------
2024-07-28 00:20:33.546 [INFO] Reactor Summary for Spark Project Parent POM 3.2.3:
2024-07-28 00:20:33.546 [INFO]
2024-07-28 00:20:33.547 [INFO] Spark Project Parent POM ........................... SUCCESS [ 8.573 s]
2024-07-28 00:20:33.547 [INFO] Spark Project Tags ................................. SUCCESS [ 10.006 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Sketch ............................... SUCCESS [ 7.142 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Local DB ............................. SUCCESS [ 8.655 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Networking ........................... SUCCESS [ 18.733 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 18.477 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Unsafe ............................... SUCCESS [ 8.548 s]
2024-07-28 00:20:33.548 [INFO] Spark Project Launcher ............................. SUCCESS [ 7.839 s]
2024-07-28 00:20:33.549 [INFO] Spark Project Core ................................. SUCCESS [05:28 min]
2024-07-28 00:20:33.549 [INFO] Spark Project ML Local Library ..................... SUCCESS [01:07 min]
2024-07-28 00:20:33.549 [INFO] Spark Project GraphX ............................... SUCCESS [01:22 min]
2024-07-28 00:20:33.549 [INFO] Spark Project Streaming ............................ SUCCESS [02:28 min]
2024-07-28 00:20:33.549 [INFO] Spark Project Catalyst ............................. SUCCESS [07:59 min]
2024-07-28 00:20:33.549 [INFO] Spark Project SQL .................................. SUCCESS [10:54 min]
2024-07-28 00:20:33.550 [INFO] Spark Project ML Library ........................... SUCCESS [07:34 min]
2024-07-28 00:20:33.550 [INFO] Spark Project Tools ................................ SUCCESS [ 7.285 s]
2024-07-28 00:20:33.550 [INFO] Spark Project Hive ................................. SUCCESS [04:07 min]
2024-07-28 00:20:33.551 [INFO] Spark Project REPL ................................. SUCCESS [01:07 min]
2024-07-28 00:20:33.551 [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 23.160 s]
2024-07-28 00:20:33.551 [INFO] Spark Project YARN ................................. SUCCESS [02:25 min]
2024-07-28 00:20:33.551 [INFO] Spark Project Kubernetes ........................... SUCCESS [02:23 min]
2024-07-28 00:20:33.551 [INFO] Spark Project Hive Thrift Server ................... SUCCESS [02:25 min]
2024-07-28 00:20:33.551 [INFO] Spark Project Assembly ............................. SUCCESS [ 5.738 s]
2024-07-28 00:20:33.551 [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [01:07 min]
2024-07-28 00:20:33.552 [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:31 min]
2024-07-28 00:20:33.552 [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [02:46 min]
2024-07-28 00:20:33.552 [INFO] Spark Project Examples ............................. SUCCESS [02:29 min]
2024-07-28 00:20:33.552 [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 16.961 s]
2024-07-28 00:20:33.552 [INFO] Spark Avro ......................................... SUCCESS [02:18 min]
2024-07-28 00:20:33.552 [INFO] ------------------------------------------------------------------------
2024-07-28 00:20:33.553 [INFO] BUILD SUCCESS
2024-07-28 00:20:33.553 [INFO] ------------------------------------------------------------------------
2024-07-28 00:20:33.553 [INFO] Total time: 01:02 h
2024-07-28 00:20:33.554 [INFO] Finished at: 2024-07-28T00:20:33+08:00
2024-07-28 00:20:33.554 [INFO] ------------------------------------------------------------------------
flink(207个子项目-42分钟-ok)
vn install -Drat.skip=true -DskipTests -Dhadoop.version=3.3.6
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Flink : 1.15.3:
[INFO]
[INFO] Flink : ............................................ SUCCESS [ 7.746 s]
[INFO] Flink : Annotations ................................ SUCCESS [ 9.409 s]
[INFO] Flink : Architecture Tests ......................... SUCCESS [ 0.393 s]
[INFO] Flink : Architecture Tests : Base .................. SUCCESS [ 1.935 s]
[INFO] Flink : Test utils : ............................... SUCCESS [ 0.334 s]
[INFO] Flink : Test utils : Junit ......................... SUCCESS [ 10.334 s]
[INFO] Flink : Metrics : .................................. SUCCESS [ 0.805 s]
[INFO] Flink : Metrics : Core ............................. SUCCESS [ 4.101 s]
[INFO] Flink : Core ....................................... SUCCESS [01:12 min]
[INFO] Flink : Table : .................................... SUCCESS [ 0.385 s]
[INFO] Flink : Table : Common ............................. SUCCESS [ 22.496 s]
[INFO] Flink : Table : API Java ........................... SUCCESS [ 10.875 s]
[INFO] Flink : Java ....................................... SUCCESS [ 14.422 s]
[INFO] Flink : Connectors : ............................... SUCCESS [ 0.227 s]
[INFO] Flink : Connectors : File Sink Common .............. SUCCESS [ 1.395 s]
[INFO] Flink : RPC : ...................................... SUCCESS [ 0.299 s]
[INFO] Flink : RPC : Core ................................. SUCCESS [ 1.355 s]
[INFO] Flink : RPC : Akka ................................. SUCCESS [ 19.480 s]
[INFO] Flink : RPC : Akka-Loader .......................... SUCCESS [ 7.192 s]
[INFO] Flink : Queryable state : .......................... SUCCESS [ 0.329 s]
[INFO] Flink : Queryable state : Client Java .............. SUCCESS [ 2.441 s]
[INFO] Flink : FileSystems : .............................. SUCCESS [ 0.308 s]
[INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 7.778 s]
[INFO] Flink : Runtime .................................... SUCCESS [02:13 min]
[INFO] Flink : Streaming Java ............................. SUCCESS [ 32.967 s]
[INFO] Flink : Table : API bridge base .................... SUCCESS [ 1.123 s]
[INFO] Flink : Table : API Java bridge .................... SUCCESS [ 2.388 s]
[INFO] Flink : Table : Code Splitter ...................... SUCCESS [ 4.890 s]
[INFO] Flink : Optimizer .................................. SUCCESS [ 8.934 s]
[INFO] Flink : Clients .................................... SUCCESS [ 6.233 s]
[INFO] Flink : DSTL ....................................... SUCCESS [ 0.231 s]
[INFO] Flink : DSTL : DFS ................................. SUCCESS [ 1.950 s]
[INFO] Flink : State backends : ........................... SUCCESS [ 0.222 s]
[INFO] Flink : State backends : RocksDB ................... SUCCESS [ 4.026 s]
[INFO] Flink : State backends : Changelog ................. SUCCESS [ 1.786 s]
[INFO] Flink : Test utils : Utils ......................... SUCCESS [ 4.068 s]
[INFO] Flink : Libraries : ................................ SUCCESS [ 0.225 s]
[INFO] Flink : Libraries : CEP ............................ SUCCESS [ 7.600 s]
[INFO] Flink : Table : Runtime ............................ SUCCESS [ 18.623 s]
[INFO] Flink : Scala ...................................... SUCCESS [01:45 min]
[INFO] Flink : Table : SQL Parser ......................... SUCCESS [ 12.453 s]
[INFO] Flink : Table : SQL Parser Hive .................... SUCCESS [ 6.941 s]
[INFO] Flink : Table : API Scala .......................... SUCCESS [ 26.600 s]
[INFO] Flink : Test utils : Connectors .................... SUCCESS [ 3.125 s]
[INFO] Flink : Architecture Tests : Test .................. SUCCESS [ 1.333 s]
[INFO] Flink : Connectors : Base .......................... SUCCESS [ 3.567 s]
[INFO] Flink : Connectors : Files ......................... SUCCESS [ 7.590 s]
[INFO] Flink : Examples : ................................. SUCCESS [ 0.403 s]
[INFO] Flink : Examples : Batch ........................... SUCCESS [ 28.380 s]
[INFO] Flink : Connectors : Hadoop compatibility .......... SUCCESS [ 14.366 s]
[INFO] Flink : Tests ...................................... SUCCESS [01:20 min]
[INFO] Flink : Streaming Scala ............................ SUCCESS [01:04 min]
[INFO] Flink : Table : API Scala bridge ................... SUCCESS [ 21.147 s]
[INFO] Flink : Table : Planner ............................ SUCCESS [06:23 min]
[INFO] Flink : Formats : .................................. SUCCESS [ 0.183 s]
[INFO] Flink : Format : Common ............................ SUCCESS [ 0.461 s]
[INFO] Flink : Formats : Csv .............................. SUCCESS [ 2.454 s]
[INFO] Flink : Formats : Hadoop bulk ...................... SUCCESS [ 2.788 s]
[INFO] Flink : Formats : Orc .............................. SUCCESS [ 3.154 s]
[INFO] Flink : Formats : Orc nohive ....................... SUCCESS [ 2.782 s]
[INFO] Flink : Formats : Avro ............................. SUCCESS [ 8.346 s]
[INFO] Flink : Formats : Parquet .......................... SUCCESS [ 17.089 s]
[INFO] Flink : Connectors : Hive .......................... SUCCESS [ 35.185 s]
[INFO] Flink : Python ..................................... SUCCESS [01:06 min]
[INFO] Flink : Table : SQL Client ......................... SUCCESS [ 4.754 s]
[INFO] Flink : Connectors : AWS Base ...................... SUCCESS [ 2.323 s]
[INFO] Flink : Connectors : Cassandra ..................... SUCCESS [ 7.680 s]
[INFO] Flink : Formats : Json ............................. SUCCESS [ 2.482 s]
[INFO] Flink : Connectors : Elasticsearch base ............ SUCCESS [ 3.874 s]
[INFO] Flink : Connectors : Elasticsearch 6 ............... SUCCESS [ 2.679 s]
[INFO] Flink : Connectors : Elasticsearch 7 ............... SUCCESS [ 2.081 s]
[INFO] Flink : Connectors : Google PubSub ................. SUCCESS [ 1.840 s]
[INFO] Flink : Connectors : HBase base .................... SUCCESS [ 3.534 s]
[INFO] Flink : Connectors : HBase 1.4 ..................... SUCCESS [ 7.560 s]
[INFO] Flink : Connectors : HBase 2.2 ..................... SUCCESS [ 6.232 s]
[INFO] Flink : Connectors : JDBC .......................... SUCCESS [ 5.505 s]
[INFO] Flink : Metrics : JMX .............................. SUCCESS [ 0.838 s]
[INFO] Flink : Formats : Avro confluent registry .......... SUCCESS [ 0.941 s]
[INFO] Flink : Connectors : Kafka ......................... SUCCESS [ 9.958 s]
[INFO] Flink : Connectors : Amazon Kinesis Data Streams ... SUCCESS [ 2.815 s]
[INFO] Flink : Connectors : Kinesis ....................... SUCCESS [ 32.955 s]
[INFO] Flink : Connectors : Nifi .......................... SUCCESS [ 1.290 s]
[INFO] Flink : Connectors : Pulsar ........................ SUCCESS [ 22.619 s]
[INFO] Flink : Connectors : RabbitMQ ...................... SUCCESS [ 1.688 s]
[INFO] Flink : Architecture Tests : Production ............ SUCCESS [ 3.075 s]
[INFO] Flink : FileSystems : Hadoop FS shaded ............. SUCCESS [ 7.840 s]
[INFO] Flink : FileSystems : S3 FS Base ................... SUCCESS [ 2.241 s]
[INFO] Flink : FileSystems : S3 FS Hadoop ................. SUCCESS [ 14.848 s]
[INFO] Flink : FileSystems : S3 FS Presto ................. SUCCESS [01:40 min]
[INFO] Flink : FileSystems : OSS FS ....................... SUCCESS [ 24.308 s]
[INFO] Flink : FileSystems : Azure FS Hadoop .............. SUCCESS [ 29.502 s]
[INFO] Flink : FileSystems : Google Storage FS Hadoop ..... SUCCESS [ 35.837 s]
[INFO] Flink : Runtime web ................................ SUCCESS [04:48 min]
[INFO] Flink : Connectors : HCatalog ...................... SUCCESS [ 12.331 s]
[INFO] Flink : Connectors : Amazon Kinesis Data Firehose .. SUCCESS [ 1.719 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 6 ......... SUCCESS [ 9.443 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 7 ......... SUCCESS [ 12.243 s]
[INFO] Flink : Connectors : SQL : HBase 1.4 ............... SUCCESS [ 8.191 s]
[INFO] Flink : Connectors : SQL : HBase 2.2 ............... SUCCESS [ 18.302 s]
[INFO] Flink : Connectors : SQL : Hive 1.2.2 .............. SUCCESS [ 14.127 s]
[INFO] Flink : Connectors : SQL : Hive 2.2.0 .............. SUCCESS [ 14.409 s]
[INFO] Flink : Connectors : SQL : Hive 2.3.6 .............. SUCCESS [ 13.488 s]
[INFO] Flink : Connectors : SQL : Hive 3.1.2 .............. SUCCESS [ 19.408 s]
[INFO] Flink : Connectors : SQL : Kafka ................... SUCCESS [ 2.206 s]
[INFO] Flink : Connectors : SQL : Amazon Kinesis Data Streams SUCCESS [ 3.958 s]
[INFO] Flink : Connectors : SQL : Amazon Kinesis Data Firehose SUCCESS [ 3.819 s]
[INFO] Flink : Connectors : SQL : Kinesis ................. SUCCESS [ 12.975 s]
[INFO] Flink : Connectors : SQL : Pulsar .................. SUCCESS [ 6.255 s]
[INFO] Flink : Connectors : SQL : RabbitMQ ................ SUCCESS [ 0.450 s]
[INFO] Flink : Formats : Sequence file .................... SUCCESS [ 1.351 s]
[INFO] Flink : Formats : Compress ......................... SUCCESS [ 1.467 s]
[INFO] Flink : Formats : Avro AWS Glue Schema Registry .... SUCCESS [ 3.423 s]
[INFO] Flink : Formats : JSON AWS Glue Schema Registry .... SUCCESS [ 3.054 s]
[INFO] Flink : Formats : SQL Orc .......................... SUCCESS [ 0.695 s]
[INFO] Flink : Formats : SQL Parquet ...................... SUCCESS [ 1.107 s]
[INFO] Flink : Formats : SQL Avro ......................... SUCCESS [ 1.702 s]
[INFO] Flink : Formats : SQL Avro Confluent Registry ...... SUCCESS [ 1.819 s]
[INFO] Flink : Examples : Streaming ....................... SUCCESS [ 28.345 s]
[INFO] Flink : Examples : Table ........................... SUCCESS [ 17.437 s]
[INFO] Flink : Examples : Build Helper : .................. SUCCESS [ 0.260 s]
[INFO] Flink : Examples : Build Helper : Streaming State machine SUCCESS [ 1.097 s]
[INFO] Flink : Examples : Build Helper : Streaming Google PubSub SUCCESS [ 6.774 s]
[INFO] Flink : Container .................................. SUCCESS [ 0.685 s]
[INFO] Flink : Queryable state : Runtime .................. SUCCESS [ 2.076 s]
[INFO] Flink : Dist-Scala ................................. SUCCESS [ 3.018 s]
[INFO] Flink : Kubernetes ................................. SUCCESS [ 15.094 s]
[INFO] Flink : Yarn ....................................... SUCCESS [ 4.876 s]
[INFO] Flink : Table : API Java Uber ...................... SUCCESS [ 5.322 s]
[INFO] Flink : Table : Planner Loader Bundle .............. SUCCESS [ 6.538 s]
[INFO] Flink : Table : Planner Loader ..................... SUCCESS [ 4.900 s]
[INFO] Flink : Libraries : Gelly .......................... SUCCESS [ 7.432 s]
[INFO] Flink : Libraries : Gelly scala .................... SUCCESS [ 38.454 s]
[INFO] Flink : Libraries : Gelly Examples ................. SUCCESS [ 21.924 s]
[INFO] Flink : External resources : ....................... SUCCESS [ 0.153 s]
[INFO] Flink : External resources : GPU ................... SUCCESS [ 0.445 s]
[INFO] Flink : Metrics : Dropwizard ....................... SUCCESS [ 0.717 s]
[INFO] Flink : Metrics : Graphite ......................... SUCCESS [ 0.533 s]
[INFO] Flink : Metrics : InfluxDB ......................... SUCCESS [ 1.629 s]
[INFO] Flink : Metrics : Prometheus ....................... SUCCESS [ 1.235 s]
[INFO] Flink : Metrics : StatsD ........................... SUCCESS [ 0.576 s]
[INFO] Flink : Metrics : Datadog .......................... SUCCESS [ 1.279 s]
[INFO] Flink : Metrics : Slf4j ............................ SUCCESS [ 0.642 s]
[INFO] Flink : Libraries : CEP Scala ...................... SUCCESS [ 26.066 s]
[INFO] Flink : Libraries : State processor API ............ SUCCESS [ 3.937 s]
[INFO] Flink : Dist ....................................... SUCCESS [ 26.987 s]
[INFO] Flink : Yarn Tests ................................. SUCCESS [ 11.118 s]
[INFO] Flink : E2E Tests : ................................ SUCCESS [ 0.195 s]
[INFO] Flink : E2E Tests : CLI ............................ SUCCESS [ 0.386 s]
[INFO] Flink : E2E Tests : Parent Child classloading program SUCCESS [ 0.363 s]
[INFO] Flink : E2E Tests : Parent Child classloading lib-package SUCCESS [ 0.243 s]
[INFO] Flink : E2E Tests : Dataset allround ............... SUCCESS [ 0.454 s]
[INFO] Flink : E2E Tests : Dataset Fine-grained recovery .. SUCCESS [ 0.532 s]
[INFO] Flink : E2E Tests : Datastream allround ............ SUCCESS [ 1.913 s]
[INFO] Flink : E2E Tests : Batch SQL ...................... SUCCESS [ 0.288 s]
[INFO] Flink : E2E Tests : Stream SQL ..................... SUCCESS [ 0.352 s]
[INFO] Flink : E2E Tests : Distributed cache via blob ..... SUCCESS [ 0.311 s]
[INFO] Flink : E2E Tests : High parallelism iterations .... SUCCESS [ 9.859 s]
[INFO] Flink : E2E Tests : Stream stateful job upgrade .... SUCCESS [ 0.861 s]
[INFO] Flink : E2E Tests : Queryable state ................ SUCCESS [ 2.495 s]
[INFO] Flink : E2E Tests : Local recovery and allocation .. SUCCESS [ 0.338 s]
[INFO] Flink : E2E Tests : Elasticsearch 6 ................ SUCCESS [ 3.868 s]
[INFO] Flink : Quickstart : ............................... SUCCESS [ 1.611 s]
[INFO] Flink : Quickstart : Java .......................... SUCCESS [ 0.927 s]
[INFO] Flink : Quickstart : Scala ......................... SUCCESS [ 0.390 s]
[INFO] Flink : E2E Tests : Quickstart ..................... SUCCESS [ 0.698 s]
[INFO] Flink : E2E Tests : Confluent schema registry ...... SUCCESS [ 3.283 s]
[INFO] Flink : E2E Tests : Stream state TTL ............... SUCCESS [ 8.303 s]
[INFO] Flink : E2E Tests : SQL client ..................... SUCCESS [ 1.151 s]
[INFO] Flink : E2E Tests : File sink ...................... SUCCESS [ 0.492 s]
[INFO] Flink : E2E Tests : State evolution ................ SUCCESS [ 1.045 s]
[INFO] Flink : E2E Tests : RocksDB state memory control ... SUCCESS [ 0.867 s]
[INFO] Flink : E2E Tests : Common ......................... SUCCESS [ 1.695 s]
[INFO] Flink : E2E Tests : Metrics availability ........... SUCCESS [ 0.452 s]
[INFO] Flink : E2E Tests : Metrics reporter prometheus .... SUCCESS [ 0.549 s]
[INFO] Flink : E2E Tests : Heavy deployment ............... SUCCESS [ 10.037 s]
[INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SUCCESS [ 1.107 s]
[INFO] Flink : E2E Tests : Streaming Kafka base ........... SUCCESS [ 0.475 s]
[INFO] Flink : E2E Tests : Streaming Kafka ................ SUCCESS [ 9.267 s]
[INFO] Flink : E2E Tests : Plugins : ...................... SUCCESS [ 0.138 s]
[INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SUCCESS [ 0.198 s]
[INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SUCCESS [ 0.216 s]
[INFO] Flink : E2E Tests : TPCH ........................... SUCCESS [ 0.979 s]
[INFO] Flink : E2E Tests : Streaming Kinesis .............. SUCCESS [ 27.913 s]
[INFO] Flink : E2E Tests : Elasticsearch 7 ................ SUCCESS [ 5.258 s]
[INFO] Flink : E2E Tests : Common Kafka ................... SUCCESS [ 1.509 s]
[INFO] Flink : E2E Tests : TPCDS .......................... SUCCESS [ 1.184 s]
[INFO] Flink : E2E Tests : Netty shuffle memory control ... SUCCESS [ 0.310 s]
[INFO] Flink : E2E Tests : Python ......................... SUCCESS [ 9.581 s]
[INFO] Flink : E2E Tests : HBase .......................... SUCCESS [ 3.361 s]
[INFO] Flink : E2E Tests : Pulsar ......................... SUCCESS [ 1.123 s]
[INFO] Flink : E2E Tests : Avro AWS Glue Schema Registry .. SUCCESS [ 2.192 s]
[INFO] Flink : E2E Tests : JSON AWS Glue Schema Registry .. SUCCESS [ 2.127 s]
[INFO] Flink : E2E Tests : Scala .......................... SUCCESS [ 11.629 s]
[INFO] Flink : E2E Tests : Kinesis SQL tests .............. SUCCESS [ 0.717 s]
[INFO] Flink : E2E Tests : Kinesis Firehose SQL tests ..... SUCCESS [ 0.666 s]
[INFO] Flink : E2E Tests : SQL ............................ SUCCESS [ 0.558 s]
[INFO] Flink : State backends : Heap spillable ............ SUCCESS [ 1.245 s]
[INFO] Flink : Table : Test Utils ......................... SUCCESS [ 1.048 s]
[INFO] Flink : Contrib : .................................. SUCCESS [ 0.134 s]
[INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [ 0.674 s]
[INFO] Flink : FileSystems : Tests ........................ SUCCESS [ 1.941 s]
[INFO] Flink : Docs ....................................... SUCCESS [ 3.384 s]
[INFO] Flink : Walkthrough : .............................. SUCCESS [ 0.285 s]
[INFO] Flink : Walkthrough : Common ....................... SUCCESS [ 2.847 s]
[INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [ 0.316 s]
[INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [ 0.424 s]
[INFO] Flink : Tools : CI : Java .......................... SUCCESS [ 0.584 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41:53 min
[INFO] Finished at: 2024-07-27T00:58:18+08:00
[INFO] ------------------------------------------------------------------------
docker-centos 编译
- Bigtop 系列 - 如何使用 Bigtop 构建 RPM/DEB 包:https://www.bilibili.com/video/BV1DL411X7xZ
- 下载编译环境:
docker pull docker.fxxk.dedyn.io/bigtop/slaves:3.2.0-centos-7
# 将本地代码目录G:\OpenSource\Data\platform\bigtop映射到ws目录
# 将maven本地仓库文件目录映射到/root,方便之后maven、grandle、ant使用
docker run -d -it -p 8000:8000 --network ambari -v G:\OpenSource\Data\platform\bigtop:/ws -v F:\docker\data\bigtop:/root --workdir /ws --name repo bigtop/slaves:3.2.0-centos-7
docker pull docker.fxxk.dedyn.io/bigtop/slaves:3.2.0-centos-7
docker pull docker.fxxk.dedyn.io/bigtop/slaves:trunk-centos-7
docker pull docker.fxxk.dedyn.io/bigtop/puppet:trunk-centos-7
docker pull mariadb:10.2
docker pull centos:7
docker pull mysql:5.7
bigtop 源码分析
packages.gradle
- task 列表
- packages-help: All package build related tasks information
- bom-json: List the components of the stack in json format
- all-components: List the components of the stack
- 单个组件:如zookeeper
- ${component}-download: Download $component artifacts 压缩包存在或者build/xxx/.dowwnload存在,则不重新下载
- ${component}-tar: Preparing a tarball for $component artifacts,若build/xxx/.tar存在,则跳过,否则解压到build/spark/tar/xxxx-xxx
- $component-deb:Building DEB for $component artifacts
- $component-sdeb:Building SDEB for $component artifacts
- $component-rpm:Building RPM for $component artifacts 若/build/xxx/.rpm存在,则跳过;否则调用rpmbuild编译,然后将build/spark/rpm/RPMS下的文件复制到 output/zookeeper
- $component-srpm:Building SRPM for $component artifacts;
- $component-pkg:Invoking a native binary packaging component $ptype
- $component-spkg:Invoking a native binary packaging component s$ptype
- $component-pkg-ind: Invoking a native binary packaging for $component in Docker
- $component-version: Show version of $component component
- ${component}_vardefines: 变量定义
- $component-info: Info about $component component build
- $component-relnotes: Preparing release notes for $component. No yet implemented!!!
- $component-clean: Removing $component component build and output directories
- $component-help: List of available tasks for $component
- 所有组件
- srpm:Build all SRPM packages for the stack components
- rpm: Build all RPM packages for the stack
- sdeb:Build all SDEB packages for the stack components
- deb: Build all DEB packages for the stack components
- pkgs:Build all native packages for the stack components
- pkgs-ind: Build all native packages for the stack components inside Docker
- allclean:Removing $BUILD_DIR, $OUTPUT_DIR, and $DIST_DIR,Cleaning all components' build and output directories
- realclean:Removing $DL_DIR
- apt:Creating APT repository
- yum: Creating YUM repository
- repo:Invoking a native repository target $
- repo-ind: Invoking a native repository in Docker