pysparkRDD数据的读取与保存

数据读取

hadoopFile

Parameters:

• path – path to Hadoop file
• inputFormatClass – fully qualified classname of Hadoop InputFormat (e.g. “org.apache.hadoop.mapred.TextInputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.Text”)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.LongWritable”)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – Hadoop configuration, passed in as a dict (None by default)
• batchSize – The number of Python objects represented as a single Java object. (default 0, choose batchSize automatically)

# hadoopFile&＃xff1a;返回键值对&＃xff0c;键为为行的偏移量&＃xff0c;值为行的内容
# log.txt:
# http://www.baidu.com
# http://www.google.com
# http://www.google.com
# ... ... ...rdd &＃61; sc.hadoopFile("hdfs://centos03:9000/datas/log.txt",
inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.LongWritable",
valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd.collect()) #1
rdd1 &＃61; rdd.map(lambda x: x[1].split(":"))
print(rdd1.collect()) #2


#1 [(0, ‘http://www.baidu.com’), (22, ‘http://www.google.com’), (45, ‘http://www.google.com’), (68, ‘http://cn.bing.com’), (88, ‘http://cn.bing.com’), (108, ‘http://www.baidu.com’), (130, ‘http://www.sohu.com’), (151, ‘http://www.sina.com’), (172, ‘http://www.sin2a.com’), (194, ‘http://www.sin2desa.com’), (219, ‘http://www.sindsafa.com’)]

#2 [[‘http’, ‘//www.baidu.com’], [‘http’, ‘//www.google.com’], [‘http’, ‘//www.google.com’], [‘http’, ‘//cn.bing.com’], [‘http’, ‘//cn.bing.com’], [‘http’, ‘//www.baidu.com’], [‘http’, ‘//www.sohu.com’], [‘http’, ‘//www.sina.com’], [‘http’, ‘//www.sin2a.com’], [‘http’, ‘//www.sin2desa.com’], [‘http’, ‘//www.sindsafa.com’]]

newAPIHadoopFile

Parameters:

• path – path to Hadoop file
• inputFormatClass – fully qualified classname of Hadoop InputFormat (e.g. “org.apache.hadoop.mapreduce.lib.input.TextInputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.Text”)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.LongWritable”)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – Hadoop configuration, passed in as a dict (None by default)
• batchSize – The number of Python objects represented as a single Java object. (default 0, choose batchSize automatically)

# newAPIHadoopFile&＃xff1a;返回键值对&＃xff0c;键为为行的偏移量&＃xff0c;值为行的内容
rdd &＃61; sc.newAPIHadoopFile("hdfs://centos03:9000/datas/log.txt",
# inputFormatClass与旧的API不同
inputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.input.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.LongWritable",
valueClass&＃61;"org.apache.hadoop.io.Text"
)
print(rdd.collect()) #1
rdd1 &＃61; rdd.map(lambda x: x[1].split(":"))
print(rdd1.collect()) #2


#1 [(0, ‘http://www.baidu.com’), (22, ‘http://www.google.com’), (45, ‘http://www.google.com’), (68, ‘http://cn.bing.com’), (88, ‘http://cn.bing.com’), (108, ‘http://www.baidu.com’), (130, ‘http://www.sohu.com’), (151, ‘http://www.sina.com’), (172, ‘http://www.sin2a.com’), (194, ‘http://www.sin2desa.com’), (219, ‘http://www.sindsafa.com’)]

#2 [[‘http’, ‘//www.baidu.com’], [‘http’, ‘//www.google.com’], [‘http’, ‘//www.google.com’], [‘http’, ‘//cn.bing.com’], [‘http’, ‘//cn.bing.com’], [‘http’, ‘//www.baidu.com’], [‘http’, ‘//www.sohu.com’], [‘http’, ‘//www.sina.com’], [‘http’, ‘//www.sin2a.com’], [‘http’, ‘//www.sin2desa.com’], [‘http’, ‘//www.sindsafa.com’]]

hadoopRDD

Parameters:

• inputFormatClass – fully qualified classname of Hadoop InputFormat (e.g. “org.apache.hadoop.mapred.TextInputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.Text”)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.LongWritable”)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – Hadoop configuration, passed in as a dict (None by default)
• batchSize – The number of Python objects represented as a single Java object. (default 0, choose batchSize automatically)

confs &＃61; {"mapred.input.dir": "hdfs://centos03:9000/datas/log.txt"}
rdd &＃61; sc.hadoopRDD(inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat",keyClass&＃61;"org.apache.hadoop.io.LongWritable",valueClass&＃61;"org.apache.hadoop.io.Text",conf&＃61;confs)
print(rdd.collect()) #1


#1&＃96; [(0, ‘http://www.baidu.com’), (22, ‘http://www.google.com’), (45, ‘http://www.google.com’), (68, ‘http://cn.bing.com’), (88, ‘http://cn.bing.com’), (108, ‘http://www.baidu.com’), (130, ‘http://www.sohu.com’), (151, ‘http://www.sina.com’), (172, ‘http://www.sin2a.com’), (194, ‘http://www.sin2desa.com’), (219, ‘http://www.sindsafa.com’)]

newAPIHadoopRDD

Parameters:

• inputFormatClass – fully qualified classname of Hadoop InputFormat (e.g. “org.apache.hadoop.mapreduce.lib.input.TextInputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.Text”)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.LongWritable”)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – Hadoop configuration, passed in as a dict (None by default)
• batchSize – The number of Python objects represented as a single Java object. (default 0, choose batchSize automatically)

confs &＃61; {"mapreduce.input.fileinputformat.inputdir":"hdfs://centos03:9000/datas/log.txt"}
rdd &＃61; sc.newAPIHadoopRDD(
inputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.input.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.LongWritable",
valueClass&＃61;"org.apache.hadoop.io.Text",
conf&＃61;confs)
print(rdd.collect()) #1


#1 [(0, ‘http://www.baidu.com’), (22, ‘http://www.google.com’), (45, ‘http://www.google.com’), (68, ‘http://cn.bing.com’), (88, ‘http://cn.bing.com’), (108, ‘http://www.baidu.com’), (130, ‘http://www.sohu.com’), (151, ‘http://www.sina.com’), (172, ‘http://www.sin2a.com’), (194, ‘http://www.sin2desa.com’), (219, ‘http://www.sindsafa.com’)]

pickleFile

Parameter:

• name – 加载数据的地址
• minPartitions&＃61;None

读取由saveAsPickleFile保存的RDD

# pickleFile读取由saveAsPickleFile保存的数据&＃xff0c;数据形式与原来保存的数据形式一样
rdd &＃61; sc.newAPIHadoopFile("hdfs://centos03:9000/datas/log.txt",
inputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.input.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.LongWritable",
valueClass&＃61;"org.apache.hadoop.io.Text"
)
print(rdd.collect()) #1
rdd1 &＃61; rdd.map(lambda x: x[1].split(":")).map(lambda x: (x[0], x[1]))
print(rdd1.collect()) #2rdd1.saveAsPickleFile("hdfs://centos03:9000/datas/logp.txt")
print(sc.pickleFile("hdfs://centos03:9000/datas/logp.txt").collect()) #3


#1[(0, ‘http://www.baidu.com’), (22, ‘http://www.google.com’), (45, ‘http://www.google.com’), (68, ‘http://cn.bing.com’), (88, ‘http://cn.bing.com’), (108, ‘http://www.baidu.com’), (130, ‘http://www.sohu.com’), (151, ‘http://www.sina.com’), (172, ‘http://www.sin2a.com’), (194, ‘http://www.sin2desa.com’), (219, ‘http://www.sindsafa.com’)]

#2 [(‘http’, ‘//www.baidu.com’), (‘http’, ‘//www.google.com’), (‘http’, ‘//www.google.com’), (‘http’, ‘//cn.bing.com’), (‘http’, ‘//cn.bing.com’), (‘http’, ‘//www.baidu.com’), (‘http’, ‘//www.sohu.com’), (‘http’, ‘//www.sina.com’), (‘http’, ‘//www.sin2a.com’), (‘http’, ‘//www.sin2desa.com’), (‘http’, ‘//www.sindsafa.com’)]

#3 [(‘http’, ‘//www.baidu.com’), (‘http’, ‘//www.google.com’), (‘http’, ‘//www.google.com’), (‘http’, ‘//cn.bing.com’), (‘http’, ‘//cn.bing.com’), (‘http’, ‘//www.baidu.com’), (‘http’, ‘//www.sohu.com’), (‘http’, ‘//www.sina.com’), (‘http’, ‘//www.sin2a.com’), (‘http’, ‘//www.sin2desa.com’), (‘http’, ‘//www.sindsafa.com’)]

sequenceFile

Parameters:

• path – path to sequncefile
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.Text”)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.LongWritable”)
• keyConverter –
• valueConverter –
• minSplits – minimum splits in dataset (default min(2, sc.defaultParallelism))
• batchSize – The number of Python objects represented as a single Java object. (default 0, choose batchSize automatically)

# 读取hadoop序列化的文件&＃xff0c;其中keyClass和valueClass可以不用指定
rdd &＃61; sc.sequenceFile(path&＃61;"hdfs://centos03:9000/datas/seqFile",
keyClass&＃61;"org.apache.hadoop.io.LongWritable",
valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd.collect()) #1


#1 [(‘Pandas’, 3), (‘Key’, 6), (‘Sanil’, 2)]

textFile

Parameter:

• name – 文件名称
• minPartitions&＃61;None
• use_unicode&＃61;True

# textFile&＃xff0c;如果use_unicode&＃61;False, 字符串为str类型&＃xff0c;会比unicode更快更小
rdd &＃61; sc.textFile(name&＃61;"hdfs://centos03:9000/datas/log.txt")
print(rdd.collect()) #1


#1 [‘http://www.baidu.com’, ‘http://www.google.com’, ‘http://www.google.com’, ‘http://cn.bing.com’, ‘http://cn.bing.com’, ‘http://www.baidu.com’, ‘http://www.sohu.com’, ‘http://www.sina.com’, ‘http://www.sin2a.com’, ‘http://www.sin2desa.com’, ‘http://www.sindsafa.com’]

wholeTextFiles

从HDFS&＃xff0c;本地文件系统或其他hadoop支持的文件系统中读取文件路径&＃xff0c;每个文件作为一个record被读取&＃xff0c;并返回一个key-value pair&＃xff0c; key为每个文件的路径&＃xff0c;value为文件的内容

Parameters:

• path
• minPartitions&＃61;None
• use_unicode&＃61;True

# wholeTextFiles&＃xff0c;比较适合小文件多的情况
rdd &＃61; sc.wholeTextFiles(path&＃61;"hdfs://centos03:9000/table")
print(rdd.collect()) #1
rdd1 &＃61; rdd.map(lambda x: x[1].split("\t"))
print(rdd1.collect()) #2


#1 [(‘hdfs://centos03:9000/table/order.txt’, ‘1001\t01\t1\r\n1002\t02\t2\r\n1003\t03\t3\r\n1004\t01\t4\r\n1005\t02\t5\r\n1006\t03\t6’), (‘hdfs://centos03:9000/table/pd.txt’, ‘01\t小米\r\n02\t华为\r\n03\t格力\r\n’)]

#2 [[‘1001’, ‘01’, ‘1\r\n1002’, ‘02’, ‘2\r\n1003’, ‘03’, ‘3\r\n1004’, ‘01’, ‘4\r\n1005’, ‘02’, ‘5\r\n1006’, ‘03’, ‘6’], [‘01’, ‘小米\r\n02’, ‘华为\r\n03’, ‘格力\r\n’]]

数据保存

saveAsHadoopFile

Output a Python RDD of key-value pairs(of form RDD[(K, V)])

Parameters:

• path – path to Hadoop file
• outputFormatClass – fully qualified classname of Hadoop OutputFormat (e.g. “org.apache.hadoop.mapred.SequenceFileOutputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.IntWritable”, None by default)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.Text”, None by default)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – (None by default)
• compressionCodecClass – (None by default)

# saveAsHadoopFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
print(rdd.collect())
rdd.saveAsHadoopFile(
path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapred.SequenceFileOutputFormat"
)
print(sc.sequenceFile("hdfs://centos03:9000/datas/rdd_seq").collect()) #1


#1 [(‘good’, 1), (“spark”, 4), (“beats”, 3)]

或&＃xff1a;

# saveAsHadoopFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
print(rdd.collect())
rdd.saveAsHadoopFile(
path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapred.TextOutputFormat")rdd1 &＃61; sc.hadoopFile(
"hdfs://centos03:9000/datas/rdd_seq",
inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.IntWritable",
valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd1.collect()) #1


#1 [(0, ‘good\t1’), (0, ‘spark\t4’), (0, ‘beats\t3’)]

从上面两段代码来看&＃xff0c;序列化形式保存数据比较好。

但是当数据为sc.parallelize([{&＃39;good&＃39;: 1}, {&＃39;spark&＃39;: 4}, {&＃39;beats&＃39;: 3}])时会出现org.apache.spark.SparkException: RDD element of type java.util.HashMap cannot be used的错误&＃xff0c;即使rdd中的数据使用json.dumps后仍然出错(org.apache.spark.SparkException: RDD element of type java.lang.String cannot be used)&＃xff0c;在网上找到一句话: To use String and Map objects you will need to use the more extensive native support available in Scala and Java.

其实在官方API文档也解释了输出的是键值对的PythonRDD

saveAsNewAPIHadoopFile

Output a Python RDD of key-value pairs(of form RDD[(K, V)])

Parameters:

• path – path to Hadoop file
• outputFormatClass – fully qualified classname of Hadoop OutputFormat (e.g. “org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat”)
• keyClass – fully qualified classname of key Writable class (e.g. “org.apache.hadoop.io.IntWritable”, None by default)
• valueClass – fully qualified classname of value Writable class (e.g. “org.apache.hadoop.io.Text”, None by default)
• keyConverter – (None by default)
• valueConverter – (None by default)
• conf – Hadoop job configuration, passed in as a dict (None by default)

# saveAsNewAPIHadoopFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
print(rdd.collect())
rdd.saveAsNewAPIHadoopFile(path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat")
print(sc.sequenceFile("hdfs://centos03:9000/datas/rdd_seq").collect()) #1


#1 [(‘good’, 1), (‘spark’, 4), (‘beats’, 3)]

# saveAsNewAPIHadoopFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
print(rdd.collect())
rdd.saveAsNewAPIHadoopFile(
path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.output.TextOutputFormat"
)rdd2 &＃61; sc.hadoopFile("hdfs://centos03:9000/datas/rdd_seq", inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat", keyClass&＃61;"org.apache.hadoop.io.IntWritable", valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd2.collect()) #1


#1 [(0, ‘good\t1’), (0, ‘spark\t4’), (0, ‘beats\t3’)]

如果改变数据存储形式呢&＃xff1a;

rdd &＃61; sc.parallelize([(1, {&＃39;good&＃39;: 1}), (2, {&＃39;spark&＃39;: 4}), (3, {&＃39;beats&＃39;: 3})])
print(rdd.collect())
rdd.saveAsNewAPIHadoopFile(
path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat")
print(sc.sequenceFile("hdfs://centos03:9000/datas/rdd_seq").collect()) #1


#1 [(1, {‘good’: 1}), (2, {‘spark’: 4}), (3, {‘beats’: 3})]

rdd &＃61; sc.parallelize([(1, {&＃39;good&＃39;: 1}), (2, {&＃39;spark&＃39;: 4}), (3, {&＃39;beats&＃39;: 3})])
print(rdd.collect())
rdd.saveAsNewAPIHadoopFile(
path&＃61;"hdfs://centos03:9000/datas/rdd_seq",
outputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.output.TextOutputFormat")
rdd2 &＃61; sc.hadoopFile(
"hdfs://centos03:9000/datas/rdd_seq",
inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat",
keyClass&＃61;"org.apache.hadoop.io.IntWritable",
valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd2.collect()) #1


#1 [(0, ‘1\torg.apache.hadoop.io.MapWritable&＃64;3e9840’), (0,‘2\torg.apache.hadoop.io.MapWritable&＃64;83dcb79’), (0,‘3\torg.apache.hadoop.io.MapWritable&＃64;7493c20’)]

从上面代码看出&＃xff0c;保存数据时还是使用序列化的形式比较好&＃xff0c;能够保存原数据的结构

saveAsHadoopDataset

Output a Python RDD of key-value pairs (of form RDD[(K, V)])

Parameters:

• conf – Hadoop job configuration, passed in as a dict
• keyConverter – (None by default)
• valueConverter – (None by default)

# saveAsHadoopDataset
confs &＃61; {"outputFormatClass": "org.apache.hadoop.mapred.TextOutputFormat","keyClass": "org.apache.hadoop.io.LongWritable","valueClass": "org.apache.hadoop.io.Text","mapred.output.dir": "hdfs://centos03:9000/datas/rdd"}
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsHadoopDataset(conf&＃61;confs) # conf中配置job参数rdd2 &＃61; sc.hadoopFile("hdfs://centos03:9000/datas/rdd", inputFormatClass&＃61;"org.apache.hadoop.mapred.TextInputFormat", keyClass&＃61;"org.apache.hadoop.io.LongWritable", valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd2.collect()) #1


#1 [(0, ‘good\t1’), (0, ‘spark\t4’), (0, ‘beats\t3’)]

# saveAsHadoopDataset
confs &＃61; {"outputFormatClass":"org.apache.hadoop.mapred.SequenceFileOutputFormat", "keyClass": "org.apache.hadoop.io.LongWritable", "valueClass": "org.apache.hadoop.io.Text","mapred.output.dir": "hdfs://centos03:9000/datas/rdd"}
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsHadoopDataset(conf&＃61;confs)rdd2 &＃61; sc.textFile("hdfs://centos03:9000/datas/rdd") # 序列化的文件可以被textFile读取
print(rdd2.collect()) #1


#1 [‘good\t1’, ‘spark\t4’, ‘beats\t3’]

saveAsNewAPIHadoopDataset

Output a Python RDD of key-value pairs (of form RDD[(K, V)])

Parameters:

• conf – Hadoop job configuration, passed in as a dict
• keyConverter – (None by default)
• valueConverter – (None by default)

# saveAsNewAPIHadoopDataset
confs &＃61; {"outputFormatClass":"org.apache.hadoop.mapreduce.lib.output.TextOutputFormat","keyClass": "org.apache.hadoop.io.LongWritable","valueClass": "org.apache.hadoop.io.Text","mapreduce.output.fileoutputformat.outputdir": "hdfs://centos03:9000/datas/rdd"}
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsNewAPIHadoopDataset(conf&＃61;confs)rdd1 &＃61; sc.newAPIHadoopFile(path&＃61;"hdfs://centos03:9000/datas/rdd", inputFormatClass&＃61;"org.apache.hadoop.mapreduce.lib.input.TextInputFormat", keyClass&＃61;"org.apache.hadoop.io.LongWritable", valueClass&＃61;"org.apache.hadoop.io.Text")
print(rdd1.collect()) #1rdd2 &＃61; sc.textFile("hdfs://centos03:9000/datas/rdd")
print(rdd2.collect()) #2


#1 [(0, ‘good\t1’), (0, ‘spark\t4’), (0, ‘beats\t3’)]

#2 [‘good\t1’, ‘spark\t4’, ‘beats\t3’]

saveAsPickleFile

Save this RDD as a SequenceFile of serialized objects. The serializer used is pyspark.serializers.PickleSerializer, default batch size is 10.

• path
• batchSize&＃61;10

# saveAsPickleFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsPickleFile("hdfs://centos03:9000/datas/rdd")rdd1 &＃61; sc.pickleFile("hdfs://centos03:9000/datas/rdd")
print(rdd1.collect()) #1


#1 [(‘good’, 1), (‘spark’, 4), (‘beats’, 3)]

saveAsSequenceFile

Output a Python RDD of key-value pairs (of form RDD[(K, V)])

中间做了两次转换&＃xff1a;1. pickled python RDD -> java RDD; 2. java RDD -> writables; 3. written out

Parameters:

• path – path to sequence file
• compressionCodecClass – (None by default)

# saveAsSequenceFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsSequenceFile("hdfs://centos03:9000/datas/rdd")rdd1 &＃61; sc.sequenceFile("hdfs://centos03:9000/datas/rdd")
print(rdd1.collect()) #1rdd2 &＃61; sc.textFile("hdfs://centos03:9000/datas/rdd")
print(rdd2.collect()) #2


#1 [(‘good’, 1), (‘spark’, 4), (‘beats’, 3)]
#2 [&＃39;SEQ\x06\x19org.apache.hadoop.io.Text org.apache.hadoop.io.IntWritable\x00\x00\x00\x00\x00\x00&＃xfffd;ekpR2\x08&＃xfffd; U&＃xfffd;&＃xfffd;Yn$’, &＃39;SEQ\x06\x19org.apache.hadoop.io.Text org.apache.hadoop.io.IntWritable\x00\x00\x00\x00\x00\x00&＃xfffd;4&＃xfffd;&＃xfffd;E&＃xfffd;}βZ;&＃xfffd;v\x1f\t\x00\x00\x00\t\x00\x00\x00\x05\x04good\x00\x00\x00\x01&＃39;, &＃39;SEQ\x06\x19org.apache.hadoop.io.Text org.apache.hadoop.io.IntWritable\x00\x00\x00\x00\x00\x00\x14&＃xfffd;&＃xfffd;˹\x02oM&＃xfffd;g&＃xfffd;&＃xfffd;f&＃xfffd;\x02v\x00\x00\x00&＃39;, &＃39;\x00\x00\x00\x06\x05spark\x00\x00\x00\x04&＃39;, &＃39;SEQ\x06\x19org.apache.hadoop.io.Text org.apache.hadoop.io.IntWritable\x00\x00\x00\x00\x00\x00F\x0b&＃xfffd;&＃xfffd;\x04lD\x116&＃43;\x16n&＃xfffd;&＃xfffd;d&＃xfffd;\x00\x00\x00&＃39;, &＃39;\x00\x00\x00\x06\x05beats\x00\x00\x00\x03&＃39;]

saveAsTextFile

# saveAsTextFile
rdd &＃61; sc.parallelize([(&＃39;good&＃39;, 1), ("spark", 4), ("beats", 3)])
rdd.saveAsTextFile("hdfs://centos03:9000/datas/rdd")
rdd2 &＃61; sc.textFile("hdfs://centos03:9000/datas/rdd")
print(rdd2.collect()) #1


#1 ["(‘good’, 1)", “(‘spark’, 4)”, “(‘beats’, 3)”]