博客
关于我
Spark安装部署
阅读量:179 次
发布时间:2019-02-28

本文共 2801 字,大约阅读时间需要 9 分钟。

一 下载Scala和Spark
[root@master opt]# wget http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz[root@master opt]# wget http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz
二 安装Scala
1 解压
[root@master opt]# tar -zxvf scala-2.11.8.tgz
2 配置环境变量
export SCALA_HOME=/opt/scala-2.11.8export PATH=$PATH:$SCALA_HOME/bin
3 测试
[root@master opt]# scalaWelcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152).Type in expressions for evaluation. Or try :help.scala>
三 安装Spark
1 解压
[root@master opt]# tar -zxvf spark-2.0.0-bin-hadoop2.7.tgz
2 配置环境变量
export SPARK_HOME=/opt/spark-2.0.0-bin-hadoop2.7export PATH=$PATH:$SPARK_HOME/bin
3 配置spark-env.sh
export JAVA_HOME=/opt/jdk1.8export PATH=$PATH:$JAVA_HOME/binexport SCALA_HOME=/opt/scala-2.11.8export PATH=$PATH:$SCALA_HOME/binexport SPARK_HOME=/opt/spark-2.0.0-bin-hadoop2.7export PATH=$PATH:$SPARK_HOME/bin
四 启动
[root@master sbin]# ./start-all.shstarting org.apache.spark.deploy.master.Master, logging to /opt/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.outlocalhost: \Slocalhost: Kernel \r on an \mlocalhost: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out[root@master sbin]# jps4128 Jps4049 Worker3992 Master
五 测试
[root@master ~]# cat test.loghello gojavac mysql[root@master sbin]# spark-shellUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel).18/02/03 22:25:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable18/02/03 22:25:08 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.Spark context Web UI available at http://192.168.0.110:4040Spark context available as 'sc' (master = local[*], app id = local-1517667907847).Spark session available as 'spark'.Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /___/ .__/\_,_/_/ /_/\_\   version 2.0.0      /_/         Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)Type in expressions to have them evaluated.Type :help for more information.scala> var file = sc.textFile("/root/test.log");file: org.apache.spark.rdd.RDD[String] = /root/test.log MapPartitionsRDD[1] at textFile at 
:24scala> file.collectres1: Array[String] = Array(hello go, java, c mysql, "", "")scala> var file = sc.textFile("hdfs://master/test.log");file: org.apache.spark.rdd.RDD[String] = hdfs://master/test.log MapPartitionsRDD[3] at textFile at
:24scala> file.collectres2: Array[String] = Array(hello go, java, c mysql, "", "")
你可能感兴趣的文章
Mysql InnoDB存储引擎 —— 数据页
查看>>
Mysql InnoDB存储引擎中的checkpoint技术
查看>>
Mysql InnoDB存储引擎中缓冲池Buffer Pool、Redo Log、Bin Log、Undo Log、Channge Buffer
查看>>
MySQL InnoDB引擎的锁机制详解
查看>>
Mysql INNODB引擎行锁的3种算法 Record Lock Next-Key Lock Grap Lock
查看>>
mysql InnoDB数据存储引擎 的B+树索引原理
查看>>
mysql innodb通过使用mvcc来实现可重复读
查看>>
mysql interval显示条件值_MySQL INTERVAL关键字可以使用哪些不同的单位值?
查看>>
Mysql join原理
查看>>
mysql order by多个字段排序
查看>>
MySQL Order By实现原理分析和Filesort优化
查看>>
mysql problems
查看>>
mysql replace first,MySQL中处理各种重复的一些方法
查看>>
MySQL replace函数替换字符串语句的用法(mysql字符串替换)
查看>>
Mysql Row_Format 参数讲解
查看>>
mysql select, from ,join ,on ,where groupby,having ,order by limit的执行顺序和书写顺序
查看>>
MySQL Server 5.5安装记录
查看>>
mysql slave 停了_slave 停止。求解决方法
查看>>
MySQL SQL 优化指南:主键、ORDER BY、GROUP BY 和 UPDATE 优化详解
查看>>
mysql sum 没返回,如果没有找到任何值,我如何在MySQL中获得SUM函数以返回'0'?
查看>>