Yarn作业信息的获取

方式1–查看作业记录文件

CDH中，在HDFS的”/user/history/done”目录下，包含了全部已完成的MR作业，”done_intermediate”包含了全部正在进行的作业。

在”/user/history/done”目录下，每个job有两个文件：job.jhist和job.xml，job.jhist是作业运行过程的详细记录，格式为json。job.xml是作业的配置文件，两者的示例结构如下。

• job–.jhist文件结构

  • task信息（每行一个）

    {"type":"TASK_STARTED","event":{"org.apache.hadoop.mapreduce.jobhistory.TaskStarted":{"taskid":"task_1465461051654_0001_r_000002","taskType":"REDUCE","startTime":1465461143745,"splitLocations":""}}}
    

  • counts（最后一行）

    {"name":"DATA_LOCAL_MAPS","displayName":"Data-local map tasks","value":1},{"name":"SLOTS_MILLIS_MAPS","displayName":"Total time spent by all maps in occupied slots (ms)","value":2229},{"name":"SLOTS_MILLIS_REDUCES","displayName":"Total time spent by all reduces in occupied slots (ms)","value":17791},{"name":"MILLIS_MAPS","displayName":"Total time spent by all map tasks (ms)","value":2229},{"name":"MILLIS_REDUCES","displayName":"Total time spent by all reduce tasks (ms)","value":17791},{"name":"VCORES_MILLIS_MAPS","displayName":"Total vcore-seconds taken by all map tasks","value":2229},{"name":"VCORES_MILLIS_REDUCES","displayName":"Total vcore-seconds taken by all reduce tasks","value":17791},{"name":"MB_MILLIS_MAPS","displayName":"Total megabyte-seconds taken by all map tasks","value":2282496},{"name":"MB_MILLIS_REDUCES","displayName":"Total megabyte-seconds taken by all reduce tasks","value":18217984}]},
    

• job–.xml文件结构

  • 输入

    • name: mapreduce.input.fileinputformat.inputdir

    • value:/user/hdfs/test

  • 输出

    • name:mapreduce.output.fileoutputformat.outputdir

    • value:/user/hdfs/out3

方式2–REST API

Hadoop YARN - Introduction to the web services REST API’s通过一个示例作业的完整运行过程，介绍了其所用到的全部REST API，包括

• MapReduce History Server REST API
• MapReduce History Server REST API
• ResourceManager REST API
• NodeManager REST API

以查看某一作业详情为例，

• request

  curl --compressed -H "Accept: application/json" -X GET "http://lyhadoop4.com:8088/ws/v1/cluster/apps/application_1465461051654_0001" 
  

• response

  {
    "app": {
      "id": "application_1465461051654_0001",
      "user": "hdfs",
      "name": "word count",
      "queue": "root.hdfs",
      "state": "FINISHED",
      "finalStatus": "SUCCEEDED",
      "progress": 100,
      "trackingUI": "History",
      "trackingUrl": "http://lyhadoop4.com:8088/proxy/application_1465461051654_0001/jobhistory/job/job_1465461051654_0001",
      "diagnostics": "",
      "clusterId": 1465461051654,
      "applicationType": "MAPREDUCE",
      "applicationTags": "",
      "startedTime": 1465461140788,
      "finishedTime": 1465461160863,
      "elapsedTime": 20075,
      "amContainerLogs": "http://lyhadoop6.com:8042/node/containerlogs/container_1465461051654_0001_01_000001/hdfs",
      "amHostHttpAddress": "lyhadoop6.com:8042",
      "allocatedMB": -1,
      "allocatedVCores": -1,
      "runningContainers": -1,
      "memorySeconds": 59076,
      "vcoreSeconds": 53,
      "preemptedResourceMB": 0,
      "preemptedResourceVCores": 0,
      "numNonAMContainerPreempted": 0,
      "numAMContainerPreempted": 0
    }
  }
  

方式3–Hadoop Command

• mapred job -option
• yarn application