cache只有一个默认的缓存级别MEMORY_ONLY ，而persist可以根据StorageLevel设置其它的缓存级别。 cache以及persist都不是action。
被重复使用的(但是)不能太大的RDD需要cache
cache 只使用 memory，checkpoint写磁盘
rdd.persist(StorageLevel.DISK_ONLY) 与 checkpoint 的区别：
persist将 RDD 的 partition 持久化到磁盘，但该 partition 由 blockManager 管理。一旦 driver program 执行结束，也就是 executor 所在进程 CoarseGrainedExecutorBackend stop，blockManager 也会 stop，被 cache 到磁盘上的 RDD 也会被清空，而 checkpoint 将 RDD 持久化到 HDFS 或本地文件夹，如果不被手动 remove 掉，是一直存在的，也就是说可以被下一个 driver program 使用，而 cached RDD 不能被其他 dirver program 使用。
使用checkponint首先需要设置setCheckpointDir
scala> bb.checkpoint
org.apache.spark.SparkException: Checkpoint directory has not been set in the SparkContext
at org.apache.spark.rdd.RDD.checkpoint(RDD.scala:1544)
at org.apache.spark.sql.Dataset.checkpoint(Dataset.scala:517)
at org.apache.spark.sql.Dataset.checkpoint(Dataset.scala:502)
... 48 elided
创建文件夹（可不创建）
[root@host ~]# hdfs dfs -mkdir /tmp/checkpoint
scala> sc.setCheckpointDir("/tmp/checkpoint")
[root@host ~]# hdfs dfs -ls -R /tmp/checkpoint
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x - root supergroup 0 2018-07-31 17:33 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b
scala> bb.checkpoint
res25: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [sex: int, count(1): bigint]
Dataset使用checkpoint不是lazy的，RDD使用checkpoint是lazy的
[root@host ~]# hdfs dfs -ls -R /tmp/checkpoint
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x - root supergroup 0 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b
drwxr-xr-x - root supergroup 0 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b/rdd-438
-rw-r--r-- 1 root supergroup 163 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b/rdd-438/part-00000
-rw-r--r-- 1 root supergroup 163 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b/rdd-438/part-00001
-rw-r--r-- 1 root supergroup 163 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b/rdd-438/part-00002
-rw-r--r-- 1 root supergroup 4 2018-07-31 17:35 /tmp/checkpoint/68309a1b-6e2d-4d03-8282-60abbbc8845b/rdd-438/part-00003
scala> bb.show
+----+--------+
| sex|count(1)|
+----+--------+
|null| 51|
| 0| 19|
| 1| 32|
+----+--------+
-------------------------------------------------------------------
scala> val weblogrdd=sc.textFile("hdfs://localhost:9000/spark/log/web.log")
weblogrdd: org.apache.spark.rdd.RDD[String] = hdfs://localhost:9000/spark/log/web.log MapPartitionsRDD[1] at textFile at <console>:24
scala> sc.setCheckpointDir("/tmp/checkpoint4")
[root@host ~]# hdfs dfs -ls -R /tmp/checkpoint4
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x - root supergroup 0 2018-08-01 13:34 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c
scala> weblogrdd.checkpoint
[root@host ~]# hdfs dfs -ls -R /tmp/checkpoint4
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x - root supergroup 0 2018-08-01 13:34 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c
scala> weblogrdd.count
res2: Long = 26
[root@host ~]# hdfs dfs -ls -R /tmp/checkpoint4
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x - root supergroup 0 2018-08-01 13:35 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c
drwxr-xr-x - root supergroup 0 2018-08-01 13:35 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c/rdd-1
-rw-r--r-- 1 root supergroup 464 2018-08-01 13:35 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c/rdd-1/part-00000
-rw-r--r-- 1 root supergroup 457 2018-08-01 13:35 /tmp/checkpoint4/c6f50081-6c31-4a5c-a1d8-afe19dcef98c/rdd-1/part-00001