PCA在Spark2.0用法比较简单&＃xff0c;只需要设置&＃xff1a;
.setInputCol(“features”)//保证输入是特征值向量
.setOutputCol(“pcaFeatures”)//输出
.setK(3)//主成分个数
注意&＃xff1a;PCA前一定要对特征向量进行规范化&＃xff08;标准化&＃xff09;&＃xff01;&＃xff01;&＃xff01;


//Spark 2.0 PCA主成分分析
//注意&＃xff1a;PCA降维前必须对原始数据&＃xff08;特征向量&＃xff09;进行标准化处理
package my.spark.ml.practice;import org.apache.spark.ml.feature.PCA;
import org.apache.spark.ml.feature.PCAModel;//不是mllib
import org.apache.spark.ml.feature.StandardScaler;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;public class myPCA {public static void main(String[] args) {SparkSession spark&＃61;SparkSession.builder().appName("myLR").master("local[4]").getOrCreate();Dataset rawDataFrame&＃61;spark.read().format("libsvm").load("/home/hadoop/spark/spark-2.0.0-bin-hadoop2.6" &＃43;"/data/mllib/sample_libsvm_data.txt");//首先对特征向量进行标准化Dataset scaledDataFrame&＃61;new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithMean(false)//对于稀疏数据&＃xff08;如本次使用的数据&＃xff09;&＃xff0c;不要使用平均值.setWithStd(true).fit(rawDataFrame).transform(rawDataFrame);//PCA ModelPCAModel pcaModel&＃61;new PCA().setInputCol("scaledFeatures").setOutputCol("pcaFeatures").setK(3)//.fit(scaledDataFrame);//进行PCA降维pcaModel.transform(scaledDataFrame).select("label","pcaFeatures").show(100,false); }
}/*** 没有标准化特征向量&＃xff0c;直接进行PCA主成分&＃xff1a;各主成分之间值变化太大&＃xff0c;有数量级的差别。
&＃43;-----&＃43;------------------------------------------------------------&＃43;
|label|pcaFeatures |
&＃43;-----&＃43;------------------------------------------------------------&＃43;
|0.0 |[-1730.496937303442,6.811910953794295,2.8044962135250024] |
|1.0 |[290.7950975587044,21.14756134360174,0.7002807351637692] |
|1.0 |[149.4029441007031,-13.733854376555671,9.844080682283838] |
|1.0 |[200.47507801105797,18.739201694569232,22.061802015132024] |
|1.0 |[236.57576401934855,36.32142445435475,56.49778957910826] |
|0.0 |[-1720.2537550195714,25.318146742090196,2.8289957152580136] |
|1.0 |[285.94940382351075,-6.729431266185428,-33.69780131162192] |
|1.0 |[-323.70613777909136,2.72250162998038,-0.528081577573507] |
|0.0 |[-1150.8358810584655,5.438673892459839,3.3725913786301804] |*/
/*** 标准化特征向量后PCA主成分&＃xff0c;各主成分之间值基本上在同一水平上&＃xff0c;结果更合理|label|pcaFeatures |
&＃43;-----&＃43;-------------------------------------------------------------&＃43;
|0.0 |[-14.998868464839624,-10.137788261664621,-3.042873539670117] |
|1.0 |[2.1965800525589754,-4.139257418439533,-11.386135042845101] |
|1.0 |[1.0254645688925883,-0.8905813756164163,7.168759904518129] |
|1.0 |[1.5069317554093433,-0.7289177578028571,5.23152743564543] |
|1.0 |[1.6938250375084654,-0.4350617717494331,4.770263568537382] |
|0.0 |[-15.870371979062549,-9.999445137658528,-6.521920373215663] |
|1.0 |[3.023279951602481,-4.102323190311296,-9.451729897327345] |
|1.0 |[3.500670997961283,-4.1791886802435805,-9.306353932746568] |
|0.0 |[-15.323114679599747,-16.83241059234951,2.0282183995400374] |
*/

如何选择k值&＃xff1f;

//PCA ModelPCAModel pcaModel&＃61;new PCA().setInputCol("scaledFeatures").setOutputCol("pcaFeatures").setK(100)//.fit(scaledDataFrame);int i&＃61;1;for(double x:pcaModel.explainedVariance().toArray()){System.out.println(i&＃43;"\t"&＃43;x&＃43;" ");i&＃43;&＃43;;}
输出100个降序的explainedVariance&＃xff08;和scikit-learn中PCA一样&＃xff09;&＃xff1a;
1 0.25934799275530857
2 0.12355355301486977
3 0.07447670060988294
4 0.0554545717486928
5 0.04207050513264405
6 0.03715986573644129
7 0.031350566055423544
8 0.027797304129489515
9 0.023825873477496748
10 0.02268054946233242
11 0.021320060154167115
12 0.019764029918116235
13 0.016789082901450734
14 0.015502412597350008
15 0.01378190652256973
16 0.013539546429755526
17 0.013283518226716669
18 0.01110412833334044
...

大约选择20个主成分就足够了
随便做一个图可以选择了&＃xff08;详细可参考Scikit-learn例子&＃xff09;
http://scikit-learn.org/stable/auto_examples/plot_digits_pipe.html

Scikit中使用PCA

参考http://blog.csdn.net/u012162613/article/details/42192293
sklearn.decomposition.PCA(n_components&＃61;None, copy&＃61;True, whiten&＃61;False)
参数说明&＃xff1a;

n_components:
意义&＃xff1a;PCA算法中所要保留的主成分个数n&＃xff0c;也即保留下来的特征个数n
类型&＃xff1a;int 或者 string&＃xff0c;缺省时默认为None&＃xff0c;所有成分被保留。
赋值为int&＃xff0c;比如n_components&＃61;1&＃xff0c;将把原始数据降到一个维度。
赋值为string&＃xff0c;比如n_components&＃61;’mle’&＃xff0c;将自动选取特征个数n&＃xff0c;使得满足所要求的方差百分比。
copy:
类型&＃xff1a;bool&＃xff0c;True或者False&＃xff0c;缺省时默认为True。
意义&＃xff1a;表示是否在运行算法时&＃xff0c;将原始训练数据复制一份。若为True&＃xff0c;则运行PCA算法后&＃xff0c;原始训练数据的值不 会有任何改变&＃xff0c;因为是在原始数据的副本上进行运算&＃xff1b;若为False&＃xff0c;则运行PCA算法后&＃xff0c;原始训练数据的 值会改&＃xff0c;因为是在原始数据上进行降维计算。
whiten:
类型&＃xff1a;bool&＃xff0c;缺省时默认为False
意义&＃xff1a;白化&＃xff0c;使得每个特征具有相同的方差。关于“白化”&＃xff0c;可参考&＃xff1a;Ufldl教程

简单例子&＃xff1a;

#!/usr/bin/env python2
# -*- coding: utf-8 -*-from sklearn import datasets
from sklearn.decomposition import PCAiris &＃61; datasets.load_iris()X &＃61; iris.data
y &＃61; iris.target
target_names &＃61; iris.target_namespca &＃61; PCA(n_components&＃61;3)
X_r &＃61; pca.fit(X).transform(X)
print "X_r"
print X_rprint "X"
print Xprint "pca.explained_variance_ratio"
print pca.explained_variance_ratio_