APT29ATTCK知识库评测分析指南

作者：鱼mm不会游泳456 | 来源：互联网 | 2023-08-22 15:50

ATT&CK真实情况分析报告本报告结合真实数据对ATT&CK的技术矩阵情况和APT29评测进行数据统计分析In[1]:fromattackctiimportattack_cl

ATT&CK真实情况分析报告

本报告结合真实数据对ATT&CK的技术矩阵情况和APT29评测进行数据统计分析

In [1]:

from attackcti import attack_client
import pandas as pd
import matplotlib.pyplot as plt
import numpy as nppd.set_option(&＃39;max_colwidth&＃39;,3000)
pd.set_option(&＃39;display.max_rows&＃39;, None)
pd.set_option(&＃39;precision&＃39;,2) lift &＃61; attack_client()
all_techniques&＃61; lift.get_techniques(stix_format&＃61;False)

ATT&CK矩阵数量

ATT&CK一共有535个技术矩阵&＃xff0c;每个技术矩阵都是一个攻防检测点。

In [2]:

techniques_normalized &＃61; pd.json_normalize(all_techniques)
techniques &＃61; techniques_normalized.reindex( \[&＃39;matrix&＃39;,&＃39;platform&＃39;,&＃39;tactic&＃39;,&＃39;technique&＃39;,&＃39;technique_id&＃39;,&＃39;data_sources&＃39;], axis&＃61;1)
techniques.shape[0]

Out[2]:

535

ATT&CK的技术矩阵涉及平台分类

ATT&CK现按平台分为windows、linux、macos、office365、saas共5类技术矩阵&＃xff0c;其中windows平台涉及222个技术矩阵。

附:柱状统计图

In [3]:

platform &＃61; {&＃39;Windows&＃39;:&＃39;Windows&＃39;,&＃39;Linux&＃39;:&＃39;Linux&＃39;,&＃39;macOS&＃39;:&＃39;macOS&＃39;,&＃39;office365&＃39;:&＃39;Office 365&＃39;,&＃39;SaaS&＃39;:&＃39;SaaS&＃39;}counts_list &＃61; []
for (data_key,data_str) in platform.items():counts_df &＃61; techniques[ (techniques[&＃39;platform&＃39;].apply(str).str.contains(data_str)) ]counts_list.append( ( data_key, counts_df.shape[0] ) )platforms_df &＃61; pd.DataFrame( counts_list ).sort_values(1,ascending&＃61;False)
platforms_df &＃61; platforms_df.rename(columns&＃61;{0:&＃39;platforms&＃39;,1:&＃39;techniques&＃39;})ax &＃61; platforms_df.plot( kind&＃61;&＃39;bar&＃39;,figsize&＃61;(10,8), fontsize&＃61;20 ,x&＃61;0, y&＃61;1, rot&＃61;360 )
for p in ax.patches:ax.annotate(str(p.get_height()), (p.get_x() &＃43; 0.1 , p.get_height() &＃43; 2 ) ,size&＃61; 20 )
plt.show()

ATT&CK检测建议数据源

ATT&CK的每个技术矩阵的检测&＃xff0c;都建议了对应的数据源&＃xff0c;其中windows平台建议了56种数据源。安全人员可以参考建议的数据源&＃xff0c;检测ATT&CK技术矩阵归类的攻击活动。

In [4]:

win_data &＃61; techniques[ (techniques[&＃39;platform&＃39;].apply(str).str.contains(&＃39;Windows&＃39;))
]
win_data &＃61; pd.DataFrame(win_data[[&＃39;technique&＃39;,&＃39;data_sources&＃39;]])
win_data &＃61; pd.DataFrame(win_data.explode(&＃39;data_sources&＃39;))
source_data &＃61; win_data.groupby(by&＃61;&＃39;data_sources&＃39;) \.count() \.sort_values(by&＃61;&＃39;technique&＃39;,ascending&＃61;True)
source_data.reset_index(inplace&＃61;True)
source_data.shape[0]

Out[4]:

56个数据源可以对应检测出不同的技术矩阵。

Process monitoring &＃xff08;进程监控&＃xff09;
Process command-line parameters &＃xff08;进程命令参数&＃xff09;
File monitoring &＃xff08;文件读写监控&＃xff09;
API monitoring &＃xff08;API调用监控&＃xff09;
Process use of network &＃xff08;进程网络关联&＃xff09;
Windows Registry &＃xff08;Windows注册表&＃xff09;
Packet capture &＃xff08;本机抓包&＃xff09;
Netflow/Enclave netflow &＃xff08;网络流量&＃xff09;
Windows event logs &＃xff08;windows事件日志&＃xff09;
Authentication logs &＃xff08;身份认证日志&＃xff09;
Network protocol analysis &＃xff08;网络协议分析&＃xff09;
DLL monitoring &＃xff08;DLL加载监控&＃xff09;
Binary file metadata &＃xff08;二进制文件元数据&＃xff09;
Loaded DLLs &＃xff08;已加载dll文件&＃xff09;
SSL/TLS inspection &＃xff08;SSL/TLS检查&＃xff09;
Azure activity logs &＃xff08;Azure活动日志&＃xff09;
PowerShell logs &＃xff08;Powershell日志&＃xff09;
Network intrusion detection system &＃xff08;网络入侵检测系统&＃xff09;
Malware reverse engineering &＃xff08;病毒逆向工程&＃xff09;
AWS CloudTrail logs &＃xff08;AWS云日志&＃xff09;
Anti-virus &＃xff08;杀毒软件&＃xff09;
Network device logs &＃xff08;网络设备日志&＃xff09;
Application logs &＃xff08;程序日志&＃xff09;
Kernel drivers &＃xff08;驱动文件&＃xff09;
Stackdriver logs &＃xff08;Stackdriver日志&＃xff09;
System calls &＃xff08;系统调用&＃xff09;
Data loss prevention &＃xff08;数据泄漏防护&＃xff09;
Web proxy &＃xff08;网页代理&＃xff09;
Email gateway &＃xff08;邮件网关&＃xff09;
Office 365 account logs &＃xff08;Office365账户日志&＃xff09;
Host network interface &＃xff08;主机网络接口&＃xff09;
User interface &＃xff08;用户接口&＃xff09;
Web logs &＃xff08;网页日志&＃xff09;
Mail server &＃xff08;邮件服务器&＃xff09;
Services &＃xff08;服务&＃xff09;
Windows Error Reporting &＃xff08;Windows错误报告&＃xff09;
Web application firewall logs &＃xff08;网页防火墙日志&＃xff09;
BIOS &＃xff08;主板系统&＃xff09;
Third-party application logs &＃xff08;第三方程序日志&＃xff09;
MBR &＃xff08;磁盘主引导记录&＃xff09;
DNS records &＃xff08;DNS记录&＃xff09;
Detonation chamber &＃xff08;引爆作业&＃xff09;
Office 365 trace logs &＃xff08;Office365跟踪日志&＃xff09;
Sensor health and status &＃xff08;传感器健康状态&＃xff09;
Component firmware &＃xff08;固件组件&＃xff09;
VBR &＃xff08;卷引导记录&＃xff09;
Access tokens &＃xff08;访问令牌&＃xff09;
Environment variable &＃xff08;环境变量&＃xff09;
Asset management &＃xff08;资产管理平台&＃xff09;
EFI &＃xff08;可扩展固件接口&＃xff09;
Named Pipes &＃xff08;命名管道&＃xff09;
Disk forensics &＃xff08;磁盘取证&＃xff09;
WMI Objects &＃xff08;WMI对象&＃xff09;
Browser extensions &＃xff08;浏览器扩展&＃xff09;
Digital certificate logs &＃xff08;数字证书日志&＃xff09;
OAuth audit logs &＃xff08;OAuth审计日志&＃xff09;

附&＃xff1a;56个检测数据源对应的技术矩阵数横向柱状统计

可以看到排名靠前的进程监控、进程命令参数、文件读写监控、API调用监控、进程网络关联等这些EDR类产品的重点关注数据。

In [5]:

tlist &＃61; source_data[&＃39;data_sources&＃39;].tolist()
ax &＃61; source_data.plot(kind&＃61;&＃39;barh&＃39;,figsize&＃61;(110,70),x&＃61;0, y&＃61;1, fontsize&＃61;65)
for i in ax.patches:ax.text(i.get_width(), i.get_y(), str(i.get_width()), fontsize&＃61;70)
my_x_ticks &＃61; np.arange(0, 160, 40)
plt.xticks(my_x_ticks)
plt.show()

ATT&CK的APT29攻击模拟评估

此次评估一共有21家安全厂商参与&＃xff0c;评估数据公开透明&＃xff0c;非常适合安全人员分析研究安全厂商的真实能力。

数据来源&＃xff1a; https://attackevals.mitre.org

In [6]:

import json,glob,osfiles &＃61;[]
for infile in sorted(glob.glob(os.path.join(&＃39;data&＃39;, &＃39;*json&＃39;))):files.append(infile)
print(len(files))
files

Out[6]:

[&＃39;data/Bitdefender.1.APT29.1_Results.json&＃39;,&＃39;data/CrowdStrike.1.APT29.1_Results.json&＃39;,&＃39;data/Cybereason.1.APT29.1_Results.json&＃39;,&＃39;data/Cycraft.1.APT29.1_Results.json&＃39;,&＃39;data/Cylance.1.APT29.1_Results.json&＃39;,&＃39;data/Elastic.1.APT29.1_Results.json&＃39;,&＃39;data/F-Secure.1.APT29.1_Results.json&＃39;,&＃39;data/FireEye.1.APT29.1_Results.json&＃39;,&＃39;data/GoSecure.1.APT29.1_Results.json&＃39;,&＃39;data/HanSight.1.APT29.1_Results.json&＃39;,&＃39;data/Kaspersky.1.APT29.1_Results.json&＃39;,&＃39;data/Malwarebytes.1.APT29.1_Results.json&＃39;,&＃39;data/McAfee.1.APT29.1_Results.json&＃39;,&＃39;data/Microsoft.1.APT29.1_Results.json&＃39;,&＃39;data/PaloAltoNetworks.1.APT29.1_Results.json&＃39;,&＃39;data/ReaQta.1.APT29.1_Results.json&＃39;,&＃39;data/Secureworks.1.APT29.1_Results.json&＃39;,&＃39;data/SentinelOne.1.APT29.1_Results.json&＃39;,&＃39;data/Symantec.1.APT29.1_Results.json&＃39;,&＃39;data/TrendMicro.1.APT29.1_Results.json&＃39;,&＃39;data/VMware.1.APT29.1_Results.json&＃39;]

In [7]:

from natsort import index_natsorted, order_by_index
import copyall_data &＃61; {}for f_path in files:vendor &＃61; f_path.split(os.sep, 2)[-1]vendor &＃61; vendor.split(&＃39;.&＃39;, 1)[0]with open(f_path, &＃39;r&＃39;, encoding&＃61;&＃39;utf-8&＃39;) as infile:data&＃61;infile.read()obj &＃61; json.loads(data)[&＃39;Techniques&＃39;]df &＃61; pd.json_normalize(obj,&＃39;Steps&＃39;, [&＃39;TechniqueId&＃39;,&＃39;TechniqueName&＃39;, &＃39;Tactics&＃39;])all_data.update({ vendor: df })

APT29评估方法

此次评估一共有140个攻击动作步骤&＃xff0c;每个步骤对应不同的战术Tactics和技术Technique。

战术Tactics &＃xff08;攻击动作的意图分类&＃xff09;
技术Technique &＃xff08;攻击动作的技术分类&＃xff09;
标准Criteria &＃xff08;攻击动作的详细过程&＃xff09;
程序Procedure &＃xff08;攻击动作的技术细节&＃xff09;

In [8]:

test_data &＃61; copy.deepcopy(all_data)
test_data &＃61; test_data.values()
test_data &＃61; list(test_data)[0]
eval_step &＃61; test_data.reindex(index&＃61;order_by_index(test_data.index, index_natsorted(test_data[&＃39;SubStep&＃39;])))
eval_step.reset_index(drop&＃61;True, inplace&＃61;True)
eval_step[&＃39;TacticsName&＃39;]&＃61;eval_step[&＃39;Tactics&＃39;].apply(lambda x: x[0][&＃39;TacticName&＃39;])
eval_step &＃61; eval_step.reindex([&＃39;SubStep&＃39;,&＃39;TacticsName&＃39;,&＃39;TechniqueName&＃39;,&＃39;Criteria&＃39;,&＃39;Procedure&＃39;], axis&＃61;1)
eval_step

Out[8]:

	SubStep	TacticsName	TechniqueName	Criteria	Procedure
0	1.A.1	Execution	User Execution	The rcs.3aka3.doc process spawning from explorer.exe	User Pam executed payload rcs.3aka3.doc
1	1.A.2	Defense Evasion	Masquerading	Evidence of the right-to-left override character (U&＃43;202E) in the rcs.3aka.doc process OR the original filename (cod.3aka.scr)	Used unicode right-to-left override (RTLO) character to obfuscate file name rcs.3aka3.doc (originally cod.3aka.scr)
2	1.A.3	Command and Control	Uncommonly Used Port	Established network channel over port 1234	Established C2 channel (192.168.0.5) via rcs.3aka3.doc payload over TCP port 1234
3	1.A.4	Command and Control	Standard Cryptographic Protocol	Evidence that the network data sent over the C2 channel is encrypted	Used RC4 stream cipher to encrypt C2 (192.168.0.5) traffic
4	1.B.1	Execution	Command-Line Interface	cmd.exe spawning from the rcs.3aka3.doc process	Spawned interactive cmd.exe
5	1.B.2	Execution	PowerShell	powershell.exe spawning from cmd.exe	Spawned interactive powershell.exe
6	2.A.1	Discovery	File and Directory Discovery	powershell.exe executing (Get-)ChildItem	Searched filesystem for document and media files using PowerShell
7	2.A.2	Collection	Automated Collection	powershell.exe executing (Get-)ChildItem	Scripted search of filesystem for document and media files using PowerShell
8	2.A.3	Collection	Data from Local System	powershell.exe reading files in C:\Users\Pam\	Recursively collected files found in C:\Users\Pam\ using PowerShell
9	2.A.4	Exfiltration	Data Compressed	powershell.exe executing Compress-Archive	Compressed and stored files into ZIP (Draft.zip) using PowerShell
10	2.A.5	Collection	Data Staged	powershell.exe creating the file draft.zip	Staged files for exfiltration into ZIP (Draft.zip) using PowerShell
11	2.B.1	Exfiltration	Exfiltration Over Command and Control Channel	The rcs.3aka3.doc process reading the file draft.zip while connected to the C2 channel	Read and downloaded ZIP (Draft.zip) over C2 channel (192.168.0.5 over TCP port 1234)
12	3.A.1	Command and Control	Remote File Copy	The rcs.3aka3.doc process creating the file monkey.png	Dropped stage 2 payload (monkey.png) to disk
13	3.A.2	Defense Evasion	Obfuscated Files or Information	Evidence that a PowerShell payload was within monkey.png	Embedded PowerShell payload in monkey.png using steganography
14	3.B.1	Defense Evasion	Component Object Model Hijacking	Addition of the DelegateExecute subkey in HKCU\Software\Classes\Folder\shell\open\command	Modified the Registry to enable COM hijacking of sdclt.exe using PowerShell
15	3.B.2	Privilege Escalation	Bypass User Account Control	High integrity powershell.exe spawning from control.exe (spawned from sdclt.exe)	Executed elevated PowerShell payload
16	3.B.3	Command and Control	Commonly Used Port	Established network channel over port 443	Established C2 channel (192.168.0.5) via PowerShell payload over TCP port 443
17	3.B.4	Command and Control	Standard Application Layer Protocol	Evidence that the network data sent over the C2 channel is HTTPS	Used HTTPS to transport C2 (192.168.0.5) traffic
18	3.B.5	Command and Control	Standard Cryptographic Protocol	Evidence that the network data sent over the C2 channel is encrypted	Used HTTPS to encrypt C2 (192.168.0.5) traffic
19	3.C.1	Defense Evasion	Modify Registry	Deletion of of the HKCU\Software\Classes\Folder\shell\Open\command subkey	Modified the Registry to remove artifacts of COM hijacking
20	4.A.1	Command and Control	Remote File Copy	powershell.exe creating the file SysinternalsSuite.zip	Dropped additional tools (SysinternalsSuite.zip) to disk over C2 channel (192.168.0.5)
21	4.A.2	Execution	PowerShell	powershell.exe spawning from powershell.exe	Spawned interactive powershell.exe
22	4.A.3	Defense Evasion	Deobfuscate/Decode Files or Information	powershell.exe executing Expand-Archive	Decompressed ZIP (SysinternalsSuite.zip) file using PowerShell
23	4.B.1	Discovery	Process Discovery	powershell.exe executing Get-Process	Enumerated current running processes using PowerShell
24	4.B.2	Defense Evasion	File Deletion	sdelete64.exe deleting the file rcs.3aka3.doc	Deleted rcs.3aka3.doc on disk using SDelete
25	4.B.3	Defense Evasion	File Deletion	sdelete64.exe deleting the file draft.zip	Deleted Draft.zip on disk using SDelete
26	4.B.4	Defense Evasion	File Deletion	sdelete64.exe deleting the file SysinternalsSuite.zip	Deleted SysinternalsSuite.zip on disk using SDelete
27	4.C.1	Discovery	File and Directory Discovery	powershell.exe executing $env:TEMP	Enumerated user&＃39;s temporary directory path using PowerShell
28	4.C.2	Discovery	System Owner/User Discovery	powershell.exe executing $env:USERNAME	Enumerated the current username using PowerShell
29	4.C.3	Discovery	System Information Discovery	powershell.exe executing $env:COMPUTERNAME	Enumerated the computer hostname using PowerShell
30	4.C.4	Discovery	System Network Configuration Discovery	powershell.exe executing $env:USERDOMAIN	Enumerated the current domain name using PowerShell
31	4.C.5	Discovery	Process Discovery	powershell.exe executing $PID	Enumerated the current process ID using PowerShell
32	4.C.6	Discovery	System Information Discovery	powershell.exe executing Gwmi Win32_OperatingSystem	Enumerated the OS version using PowerShell
33	4.C.7	Discovery	Security Software Discovery	powershell.exe executing Get-WmiObject ... -Class AntiVirusProduct	Enumerated anti-virus software using PowerShell
34	4.C.8	Discovery	Security Software Discovery	powershell.exe executing Get-WmiObject ... -Class FireWallProduct	Enumerated firewall software using PowerShell
35	4.C.9	Discovery	Permission Groups Discovery	powershell.exe executing the NetUserGetGroups API	Enumerated user&＃39;s domain group membership via the NetUserGetGroups API
36	4.C.10	Execution	Execution through API	The NetUserGetGroups API function loaded into powershell.exe from Netapi32.dll	Executed API call by reflectively loading Netapi32.dll
37	4.C.11	Discovery	Permission Groups Discovery	powershell.exe executing the NetUserGetLocalGroups API	Enumerated user&＃39;s local group membership via the NetUserGetLocalGroups API
38	4.C.12	Execution	Execution through API	The NetUserGetLocalGroups API function loaded into powershelle.exe from Netapi32.dll	Executed API call by reflectively loading Netapi32.dll
39	5.A.1	Persistence	New Service	powershell.exe creating the Javamtsup service	Created a new service (javamtsup) that executes a service binary (javamtsup.exe) at system startup
40	5.B.1	Persistence	Registry Run Keys / Startup Folder	powershell.exe creating the file hostui.lnk in the Startup folder	Created a LNK file (hostui.lnk) in the Startup folder that executes on login
41	6.A.1	Credential Access	Credentials in Files	accesschk.exe reading files within %APPDATALOCAL%\Google\chrome\user data\default\	Read the Chrome SQL database file to extract encrypted credentials
42	6.A.2	Credential Access	Credential Dumping	accesschk.exe executing the CryptUnprotectedData API	Executed the CryptUnprotectedData API call to decrypt Chrome passwords
43	6.A.3	Defense Evasion	Masquerading	Evidence that accesschk.exe is not the legitimate Sysinternals tool	Masqueraded a Chrome password dump tool as accesscheck.exe, a legitimate Sysinternals tool
44	6.B.1	Credential Access	Private Keys	powershell.exe creating a certificate file exported from the system	Exported a local certificate to a PFX file using PowerShell
45	6.C.1	Credential Access	Credential Dumping	powershell.exe injecting into lsass.exe OR lsass.exe reading Registry keys under HKLM:\SAM\SAM\Domains\Account\Users\	Dumped password hashes from the Windows Registry by injecting a malicious DLL into Lsass.exe
46	7.A.1	Collection	Screen Capture	powershell.exe executing the CopyFromScreen function from System.Drawing.dll	Captured and saved screenshots using PowerShell
47	7.A.2	Collection	Clipboard Data	powershell.exe executing Get-Clipboard	Captured clipboard contents using PowerShell
48	7.A.3	Collection	Input Capture	powershell.exe executing the GetAsyncKeyState API	Captured user keystrokes using the GetAsyncKeyState API
49	7.B.1	Collection	Data from Local System	powershell.exe reading files in C:\Users\pam\Downloads\	Read data in the user&＃39;s Downloads directory using PowerShell
50	7.B.2	Exfiltration	Data Compressed	powershell.exe creating the file OfficeSupplies.7z	Compressed data from the user&＃39;s Downloads directory into a ZIP file (OfficeSupplies.7z) using PowerShell
51	7.B.3	Exfiltration	Data Encrypted	powershell.exe executing Compress-7Zip with the password argument used for encryption	Encrypted data from the user&＃39;s Downloads directory using PowerShell
52	7.B.4	Exfiltration	Exfiltration Over Alternative Protocol	powershell executing Copy-Item pointing to an attack-controlled WebDav network share (192.168.0.4:80)	Exfiltrated collection (OfficeSupplies.7z) to WebDAV network share using PowerShell
53	8.A.1	Discovery	Remote System Discovery	powershell.exe making LDAP queries over port 389 to the Domain Controller (10.0.0.4)	Enumerated remote systems using LDAP queries
54	8.A.2	Execution	Windows Remote Management	Network connection to Scranton (10.0.1.4) over port 5985	Established WinRM connection to remote host Scranton (10.0.1.4)
55	8.A.3	Discovery	Process Discovery	powershell.exe executing Get-Process	Enumerated processes on remote host Scranton (10.0.1.4) using PowerShell
56	8.B.1	Command and Control	Remote File Copy	The file python.exe created on Scranton (10.0.1.4)	Copied python.exe payload from a WebDAV share (192.168.0.4) to remote host Scranton (10.0.1.4)
57	8.B.2	Defense Evasion	Software Packing	Evidence that the file python.exe is packed	python.exe payload was packed with UPX
58	8.C.1	Defense Evasion	Valid Accounts	Successful logon as user Pam on Scranton (10.0.1.4)	Logged on to remote host Scranton (10.0.1.4) using valid credentials for user Pam
59	8.C.2	Lateral Movement	Windows Admin Shares	SMB session to Scanton (10.0.1.4) over TCP port 445/135 OR evidence of usage of a Windows share	Established SMB session to remote host Scranton&＃39;s (10.0.1.4) IPC$ share using PsExec
60	8.C.3	Execution	Service Execution	python.exe spawned by PSEXESVC.exe	Executed python.exe using PSExec
61	9.A.1	Command and Control	Remote File Copy	python.exe creating the file rar.exe	Dropped rar.exe to disk on remote host Scranton (10.0.1.4)
62	9.A.2	Command and Control	Remote File Copy	python.exe creating the file sdelete64.exe	Dropped sdelete.exe to disk on remote host Scranton (10.0.1.4)
63	9.B.1	Execution	PowerShell	powershell.exe spawning from python.exe	Spawned interactive powershell.exe
64	9.B.2	Discovery	File and Directory Discovery	powershell.exe executing (Get-)ChildItem	Searched filesystem for document and media files using PowerShell
65	9.B.3	Collection	Automated Collection	powershell.exe executing (Get-)ChildItem	Scripted search of filesystem for document and media files using PowerShell
66	9.B.4	Collection	Data from Local System	powershell.exe reading files in C:\Users\Pam\	Recursively collected files found in C:\Users\Pam\ using PowerShell
67	9.B.5	Collection	Data Staged	powershell.exe creating the file working.zip	Staged files for exfiltration into ZIP (working.zip in AppData directory) using PowerShell
68	9.B.6	Exfiltration	Data Encrypted	powershell.exe executing rar.exe with the -a parameter for a password to use for encryption	Encrypted staged ZIP (working.zip in AppData directory) into working.zip (on Desktop) using rar.exe
69	9.B.7	Exfiltration	Data Compressed	powershell.exe executing rar.exe	Compressed staged ZIP (working.zip in AppData directory) into working.zip (on Desktop) using rar.exe
70	9.B.8	Exfiltration	Exfiltration Over Command and Control Channel	python.exe reading the file working.zip while connected to the C2 channel	Read and downloaded ZIP (working.zip on Desktop) over C2 channel (192.168.0.5 over TCP port 8443)
71	9.C.1	Defense Evasion	File Deletion	sdelete64.exe deleting the file rar.exe	Deleted rar.exe on disk using SDelete
72	9.C.2	Defense Evasion	File Deletion	sdelete64.exe deleting the file \Desktop\working.zip	Deleted working.zip (from Desktop) on disk using SDelete
73	9.C.3	Defense Evasion	File Deletion	sdelete64.exe deleting the file \AppData\Roaming\working.zip	Deleted working.zip (from AppData directory) on disk using SDelete
74	9.C.4	Defense Evasion	File Deletion	cmd.exe deleting the file sdelete64.exe	Deleted SDelete on disk using cmd.exe del command
75	10.A.1	Execution	Service Execution	javamtsup.exe spawning from services.exe	Executed persistent service (javamtsup) on system startup
76	10.B.1	Persistence	Registry Run Keys / Startup Folder	Evidence that the file hostui.lnk (which executes hostui.bat as a byproduct) was executed from the Startup Folder	Executed LNK payload (hostui.lnk) in Startup Folder on user login
77	10.B.2	Execution	Execution through API	hostui.exe executing the\nCreateProcessWithToken API	Executed PowerShell payload via the CreateProcessWithToken API
78	10.B.3	Defense Evasion	Access Token Manipulation	hostui.exe manipulating the token of powershell.exe via the CreateProcessWithToken API OR \npowershell.exe executing with the stolen token of explorer.exe	Manipulated the token of the PowerShell payload via the CreateProcessWithToken API
79	11.A.1	Execution	User Execution	powershell.exe spawning from explorer.exe	User Oscar executed payload 37486-the-shocking-truth-about-election-rigging-in-america.rtf.lnk
80	11.A.2	Defense Evasion	NTFS File Attributes	powershell.exe executing the schemas ADS via Get-Content and IEX	Executed an alternate data stream (ADS) using PowerShell
81	11.A.3	Discovery	Virtualization/Sandbox Evasion	powershell.exe executing a Get-WmiObject\nquery for Win32_BIOS	Checked that the BIOS version and serial number are not associated with VirtualBox or VMware using PowerShell
82	11.A.4	Discovery	System Information Discovery	powershell.exe executing a Get-WmiObject gwmi queries for Win32_BIOS and Win32_ComputerSystem	Enumerated computer manufacturer, model, and version information using PowerShell
83	11.A.5	Discovery	Peripheral Device Discovery	powershell.exe executing a Get-WmiObject query for Win32_PnPEntity	Enumerated devices/adapters to check for presence of VirtualBox driver(s) using PowerShell
84	11.A.6	Discovery	System Owner/User Discovery	powershell.exe executing a Get-WmiObject query for Win32_ComputerSystem	Checked that the username is not related to admin or a generic value (ex: user) using PowerShell
85	11.A.7	Discovery	System Network Configuration Discovery	powershell.exe executing a Get-WmiObject query for Win32_ComputerSystem	Checked that the computer is joined to a domain using PowerShell
86	11.A.8	Discovery	Process Discovery	powershell.exe executing a Get-WmiObject query for Win32_Process	Checked that processes such as procexp.exe, taskmgr.exe, or wireshark.exe are not running using PowerShell
87	11.A.9	Discovery	File and Directory Discovery	powershell.exe executing (Get-Item -Path ".\" -Verbose).FullName	Checked that the payload is not inside a folder path that contains "sample" or is the length of a hash value using PowerShell
88	11.A.10	Defense Evasion	Deobfuscate/Decode Files or Information	certutil.exe decoding kxwn.lock	Decoded an embedded DLL payload to disk using certutil.exe
89	11.A.11	Persistence	Registry Run Keys / Startup Folder	Addition of the Webcache subkey in HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run	Established Registry Run key persistence using PowerShell
90	11.A.12	Execution	PowerShell	powershell.exe spawning from from the schemas ADS (powershell.exe)	Executed PowerShell stager payload
91	11.A.13	Command and Control	Commonly Used Port	Established network channel over port 443	Established C2 channel (192.168.0.4) via PowerShell payload over port 443
92	11.A.14	Command and Control	Standard Application Layer Protocol	Established network channel over the HTTPS protocol	Used HTTPS to transport C2 (192.168.0.4) traffic
93	11.A.15	Command and Control	Standard Cryptographic Protocol	Evidence that the network data sent over the C2 channel is encrypted	Used HTTPS to encrypt C2 (192.168.0.4) traffic
94	12.A.1	Discovery	File and Directory Discovery	powershell.exe executing (gci ((gci env:windir).Value &＃43; &＃39;\system32&＃39;)	Enumerated the System32 directory using PowerShell
95	12.A.2	Defense Evasion	Timestomp	powershell.exe modifying the creation, last access, and last write times of kxwn.lock	Modified the time attributes of the kxwn.lock persistence payload using PowerShell
96	12.B.1	Discovery	Security Software Discovery	powershell.exe executing a Get-WmiObject query for AntiVirusProduct	Enumerated registered AV products using PowerShell
97	12.C.1	Discovery	Query Registry	powershell.exe executing a Registry query for HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall	Enumerated installed software via the Registry (Wow6432 Uninstall key) using PowerShell
98	12.C.2	Discovery	Query Registry	powershell.exe executing a Registry query for HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall	Enumerated installed software via the Registry (Uninstall key) using PowerShell
99	13.A.1	Discovery	System Information Discovery	powershell.exe executing the GetComputerNameEx API	Enumerated the computer name using the GetComputerNameEx API
100	13.B.1	Discovery	System Network Configuration Discovery	powershell.exe executing the NetWkstaGetInfo API	Enumerated the domain name using the NetWkstaGetInfo API
101	13.C.1	Discovery	System Owner/User Discovery	powershell.exe executing the GetUserNameEx API	Enumerated the current username using the GetUserNameEx API
102	13.D.1	Discovery	Process Discovery	powershell.exe executing the CreateToolhelp32Snapshot API	Enumerated running processes using the CreateToolhelp32Snapshot API
103	14.A.1	Defense Evasion	Component Object Model Hijacking	Addition of the DelegateExecute subkey in HKCU\Software\Classes\Folder\shell\open\command	Modified the Registry to enable COM hijacking of sdclt.exe using PowerShell
104	14.A.2	Privilege Escalation	Bypass User Account Control	High integrity powrshell.exe spawning from control.exe (spawned from sdclt.exe)	Executed elevated PowerShell payload
105	14.A.3	Defense Evasion	Modify Registry	Deletion of the HKCU\Software\Classes\Folder\shell\Open\command subkey	Modified the Registry to remove artifacts of COM hijacking using PowerShell
106	14.B.1	Execution	Windows Management Instrumentation	WMI Process (WmiPrvSE.exe) executing powershell.exe	Created and executed a WMI class using PowerShell
107	14.B.2	Discovery	Process Discovery	powershell.exe executing Get-Process	Enumerated and tracked PowerShell processes using PowerShell
108	14.B.3	Command and Control	Remote File Copy	powershell.exe downloading and/or the file write of m.exe	Downloaded and dropped Mimikatz (m.exe) to disk
109	14.B.4	Credential Access	Credential Dumping	m.exe injecting into lsass.exe to dump credentials	Dumped plaintext credentials using Mimikatz (m.exe)
110	14.B.5	Defense Evasion	Obfuscated Files or Information	powershell.exe executing Set-WmiInstance	Encoded and wrote Mimikatz output to a WMI class property using PowerShell
111	14.B.6	Defense Evasion	Deobfuscate/Decode Files or Information	powershell.exe executing Get-WmiInstance	Read and decoded Mimikatz output from a WMI class property using PowerShell
112	15.A.1	Discovery	System Owner/User Discovery	powershell.exe executing $env:UserName	Enumerated logged on users using PowerShell
113	15.A.2	Persistence	Windows Management Instrumentation Event Subscription	powershell.exe creating the WindowsParentalControlMigration WMI filter, consumer, and binding created in root/subscription	Established WMI event subscription persistence using PowerShell
114	16.A.1	Discovery	Remote System Discovery	powershell.exe making LDAP queries over port 389 via functions from System.DirectoryServices.dll	Enumerated the domain controller host NewYork (10.0.0.4) using LDAP queries
115	16.B.1	Discovery	System Owner/User Discovery	powershell.exe executing the ConvertSidToStringSid API	Enumerated the domain SID (from current user SID) using the ConvertSidToStringSid API
116	16.B.2	Execution	Execution through API	powershell.exe executing the ConvertSidToStringSid API function by loading Advapi32.dll	Executed the ConvertSidToStringSid API call by reflectively loading Advapi32.dll
117	16.C.1	Execution	Windows Remote Management	Network connection to NewYork (10.0.0.4) over port 5985	Established a WinRM connection to the domain controller host NewYork (10.0.0.4)
118	16.C.2	Defense Evasion	Valid Accounts	Successful logon as user MScott on NewYork (10.0.0.4)	Logged on to the domain controller host NewYork (10.0.0.4) using valid credentials for user MScott
119	16.D.1	Command and Control	Remote File Copy	File write of m.exe by the WinRM process (wsmprovhost.exe)	Dropped Mimikatz (m.exe) to disk on the domain controller host NewYork (10.0.0.4) over a WinRM connection
120	16.D.2	Credential Access	Credential Dumping	m.exe injecting into lsass.exe to dump credentials	Dumped the KRBTGT hash on the domain controller host NewYork (10.0.0.4) using Mimikatz (m.exe)
121	17.A.1	Collection	Email Collection	outlook.exe spawning from svchost.exe or powershell.exe	Dumped messages from the local Outlook inbox using PowerShell
122	17.B.1	Collection	Data from Local System	powershell.exe reading the file MITRE-ATTACK-EVALS.HTML	Read and collected a local file using PowerShell
123	17.B.2	Collection	Data Staged	powershell.exe creating the file \WindowsParentalControlMigration\MITRE-ATTACK-EVALS.HTML	Staged collected file into directory using PowerShell
124	17.C.1	Exfiltration	Data Compressed	powershell.exe executing the ZipFile.CreateFromDirectory .NET method	Compressed a staging directory using PowerShell
125	17.C.2	Defense Evasion	Obfuscated Files or Information	powershell.exe executing Set-Content	Prepended the GIF file header to a compressed staging file using PowerShell
126	18.A.1	Defense Evasion	Web Service	net.exe with command-line arguments then making a network connection to a public IP over port 443	Mapped a network drive to an online OneDrive account using PowerShell
127	18.A.2	Exfiltration	Exfiltration Over Alternative Protocol	powershell.exe executing Copy-Item pointing to drive mapped to an attack-controlled OneDrive account	Exfiltrated staged collection to an online OneDrive account using PowerShell
128	19.A.1	Defense Evasion	File Deletion	File delete event for C:\Windows\System32\m.exe	Deleted Mimikatz (m.exe) on disk using SDelete
129	19.A.2	Defense Evasion	Process Injection	Injection into PowerShell via Invoke-ReflectivePEInjection	Reflectively injected SDelete binary into PowerShell
130	19.B.1	Defense Evasion	File Deletion	File delete event for C:\Windows\Temp\WindowsParentalControlMigration.tmp	Deleted exfiltrated data on disk using SDelete
131	19.B.2	Defense Evasion	Process Injection	Injection into PowerShell via Invoke-ReflectivePEInjection	Reflectively injected SDelete binary into PowerShell
132	19.C.1	Defense Evasion	File Deletion	File delete event for C:\Windows\Temp\WindowsParentalControlMigration\MITRE-ATTACK-EVALS.HTML	Deleted staged data on disk using SDelete
133	19.C.2	Defense Evasion	Process Injection	Injection into PowerShell via Invoke-ReflectivePEInjection	Reflectively injected SDelete binary into PowerShell
134	20.A.1	Execution	Rundll32	rundll32.exe executing kxwn.lock	Executed Run key persistence payload on user login using RunDll32
135	20.A.2	Persistence	Windows Management Instrumentation Event Subscription	The WMI process (wmiprvse.exe) executing powershell.exe	Executed WMI persistence on user login
136	20.A.3	Execution	PowerShell	SYSTEM-level powershell.exe spawned from the powershell.exe	Executed PowerShell payload from WMI event subscription persistence
137	20.B.1	Lateral Movement	Pass the Ticket	powershell.exe executing Invoke-Mimikatz with command-line arguments to create a golden ticket	Created Kerberos Golden Ticket using Invoke-Mimikatz
138	20.B.2	Execution	Windows Remote Management	Network connection to Scranton (10.0.1.4) over port 5985	Established a WinRM connection to the remote host Scranton (10.0.1.4) using the Golden Ticket as credentials
139	20.B.3	Persistence	Create Account	net.exe adding the user Toby	Added a new user to the remote host Scranton (10.0.1.4) using net.exe

关于如何评测APT29模拟攻击的检出数据

APT29攻击的测试环境工具已经完全开源 https://github.com/mitre-attack/attack-arsenal

厂商的检出结果公开透明&＃xff0c;MITRE ATT&CK官方的评估标准颗粒度较细&＃xff0c;检出结果类型分主要检测类型和修饰检测类型。修饰检测类型相当于附加描叙,可以算是加分项。

由于MITRE ATT&CK官方只给出统计数据&＃xff0c;而各方的评分标准不统一&＃xff0c;导致各路分析师和厂商对应检测成绩的评定过于两极化&＃xff0c;同时如果过多关注检测类型组合情况反而复杂化了评估&＃xff0c;反应不出厂商真实水平&＃xff0c;所以我以一线技术人员的理解给出了评估方法。

主要按如下方法评估厂商

非None类型都可认为是检出情况&＃xff0c;可统计技术矩阵覆盖度。
Telemetry和MSSP两种类型告警进行统计&＃xff0c;可统计需要分析师运营跟进的模糊告警数据。
General、Tactic、Technique三种类型进行统计&＃xff0c;可统计明确的恶意告警数据。
按主要检测类型的价值梯度进行打分&＃xff0c;以检出结果的运营价值得出厂商排名。

主要检测类型

None &＃xff08;无检出&＃xff09;
Telemetry &＃xff08;遥测型检出&＃xff0c;仅作为打点数据&＃xff0c;无上下文参考或明确恶意标记的告警&＃xff09;
MSSP &＃xff08;安全运营型检出&＃xff0c;需要分析师进一步关联分析判断才能确认威胁的告警&＃xff09;
General &＃xff08;通用型检出&＃xff0c;标记为通用的恶意行为告警&＃xff0c;无准确的技战术分类&＃xff09;
Tactic &＃xff08;战术型检出&＃xff0c;向分析师提供了攻击动作潜在意图信息的告警&＃xff09;
Technique &＃xff08;技术型检出&＃xff0c;向分析师提供了攻击动作的详细技战术信息的告警&＃xff09;

修饰检测类型

Alert &＃xff08;告警&＃xff09;
Correlated &＃xff08;相关&＃xff09;
Delayed &＃xff08;延迟&＃xff09;
Host Interrogation &＃xff08;主机响应&＃xff09;
Residual Artifact &＃xff08;残留工件&＃xff09;
Configuration Change &＃xff08;配置更改&＃xff09;
Innovative &＃xff08;创新检测&＃xff09;

此次APT29评估一共涉及了ATT&CK共58个技术矩阵&＃xff0c;每个技术矩阵涉及多个模拟攻击动作步骤。

附&＃xff1a;统计列表

In [9]:

tech_count &＃61; eval_step.reindex([&＃39;TechniqueName&＃39;,&＃39;SubStep&＃39;],axis&＃61;1)
tech_count &＃61; tech_count.groupby([&＃39;TechniqueName&＃39;]).count(). \sort_values(by&＃61;&＃39;SubStep&＃39;,ascending&＃61;False)
tech_count.reset_index(inplace&＃61;True)
tech_count
#tech_count.to_csv(&＃39;tech_count.csv&＃39;,encoding&＃61;&＃39;utf_8_sig&＃39;)

Out[9]:

	TechniqueName	SubStep
0	File Deletion	10
1	Remote File Copy	7
2	Process Discovery	6
3	PowerShell	5
4	System Owner/User Discovery	5
5	File and Directory Discovery	5
6	Execution through API	4
7	System Information Discovery	4
8	Credential Dumping	4
9	Data Compressed	4
10	Data from Local System	4
11	Obfuscated Files or Information	3
12	Process Injection	3
13	Registry Run Keys / Startup Folder	3
14	Security Software Discovery	3
15	Windows Remote Management	3
16	Data Staged	3
17	Standard Cryptographic Protocol	3
18	System Network Configuration Discovery	3
19	Deobfuscate/Decode Files or Information	3
20	Service Execution	2
21	Automated Collection	2
22	Remote System Discovery	2
23	Standard Application Layer Protocol	2
24	Query Registry	2
25	User Execution	2
26	Valid Accounts	2
27	Windows Management Instrumentation Event Subscription	2
28	Permission Groups Discovery	2
29	Data Encrypted	2
30	Component Object Model Hijacking	2
31	Bypass User Account Control	2
32	Modify Registry	2
33	Masquerading	2
34	Exfiltration Over Command and Control Channel	2
35	Exfiltration Over Alternative Protocol	2
36	Commonly Used Port	2
37	Windows Management Instrumentation	1
38	Windows Admin Shares	1
39	Web Service	1
40	Virtualization/Sandbox Evasion	1
41	Clipboard Data	1
42	Command-Line Interface	1
43	Uncommonly Used Port	1
44	Timestomp	1
45	Peripheral Device Discovery	1
46	Create Account	1
47	Credentials in Files	1
48	Pass the Ticket	1
49	Software Packing	1
50	Email Collection	1
51	Screen Capture	1
52	Rundll32	1
53	Input Capture	1
54	NTFS File Attributes	1
55	New Service	1
56	Private Keys	1
57	Access Token Manipulation	1

挑一个厂商的File Deletion技术矩阵检出日志查看一下检出类型

检出类型有Telemetry类型&＃xff0c;有MSSP类型&＃xff0c;也有Technique类型&＃xff0c;以及具有争议的N/A无类型检出。

In [10]:

from IPython.display import JSON
import warnings
warnings.filterwarnings(&＃39;ignore&＃39;)cy &＃61; all_data[&＃39;Cycraft&＃39;][ (all_data[&＃39;Cycraft&＃39;][&＃39;TechniqueName&＃39;]&＃61;&＃61;&＃39;File Deletion&＃39;)
]print(list(all_data.keys()))
JSON(cy[&＃39;Detections&＃39;].to_json(),expanded&＃61;True)

[&＃39;Bitdefender&＃39;, &＃39;CrowdStrike&＃39;, &＃39;Cybereason&＃39;, &＃39;Cycraft&＃39;, &＃39;Cylance&＃39;, &＃39;Elastic&＃39;, &＃39;F-Secure&＃39;, &＃39;FireEye&＃39;, &＃39;GoSecure&＃39;, &＃39;HanSight&＃39;, &＃39;Kaspersky&＃39;, &＃39;Malwarebytes&＃39;, &＃39;McAfee&＃39;, &＃39;Microsoft&＃39;, &＃39;PaloAltoNetworks&＃39;, &＃39;ReaQta&＃39;, &＃39;Secureworks&＃39;, &＃39;SentinelOne&＃39;, &＃39;Symantec&＃39;, &＃39;TrendMicro&＃39;, &＃39;VMware&＃39;]

Out[10]:

APT29评估检出覆盖度统计

对应140个步骤的检测结果&＃xff0c;在不考虑检测类型即准确性和误报的情况下,仅对厂商有检出结果的技术矩阵覆盖度进行统计排名。

结论&＃xff1a; 绝大多数厂商都已经覆盖了100个以上攻击动作步骤的检出&＃xff0c;说明各厂商产品对ATT&CK技术矩阵分类的攻击动作相关告警跟进力度都很大。

In [11]:

def check_all_Detection(): flag_list &＃61; []flag_data &＃61; copy.deepcopy(all_data)def check_deep_Detection(DetectionNote):Detection &＃61; 0for xl in DetectionNote:if xl[&＃39;DetectionType&＃39;]&＃61;&＃61;&＃39;None&＃39;:Detection &＃61; 0else:Detection &＃61; &＃43;1return Detectionfor i,d in flag_data.items():d[&＃39;Detections&＃39;] &＃61; d[&＃39;Detections&＃39;].apply(lambda x: check_deep_Detection(x)) d_count &＃61;d[&＃39;Detections&＃39;].sum() flag_list.append( (i,d_count) )return flag_listflag_list &＃61;check_all_Detection()
flag_df &＃61; pd.DataFrame( flag_list ).sort_values(1,ascending&＃61;True)
ax &＃61; flag_df.plot(kind&＃61;&＃39;barh&＃39;,figsize&＃61;(110,70),x&＃61;0, y&＃61;1, fontsize&＃61;65)
for i in ax.patches:#ax.text(i.get_width(), i.get_y() &＃43; 0.2, &＃39;{:.0%}&＃39;.format(i.get_width()/140), fontsize&＃61;70)ax.text(i.get_width(), i.get_y() &＃43; 0.2, i.get_width() , fontsize&＃61;70)
my_x_ticks &＃61; np.arange(0, 160, 40)
plt.xticks(my_x_ticks)
plt.show()

APT29评估遥测和安全运营类型检出数据统计

厂商的遥测运营能力统计&＃xff0c;只统计每个攻击动作步骤中厂商检出的Telemetry和MSSP类型告警数据。

结论&＃xff1a;在140个攻击动作步骤检出结果里&＃xff0c;厂商的检出数据绝大多数都是遥测运营类数据&＃xff0c;这类检出数据都需要分析师进一步关联分析才能确认威胁,可以看出厂商的产品路线都是在走分析师参与的重运营路线。

In [12]:

def check_custom_Detection( check_list ): flag_list &＃61; []flag_data &＃61; copy.deepcopy(all_data)def check_deep1_Detection(DetectionNote):Detection &＃61; 0for xl in DetectionNote:for cl in check_list:if xl[&＃39;DetectionType&＃39;]&＃61;&＃61; cl:Detection &＃61; &＃43;1return Detectionfor i,d in flag_data.items():d[&＃39;Detections&＃39;] &＃61; d[&＃39;Detections&＃39;].apply(lambda x: check_deep1_Detection(x)) d_count &＃61;d[&＃39;Detections&＃39;].sum() flag_list.append( (i,d_count) )tel_df &＃61; pd.DataFrame( flag_list ).sort_values(1,ascending&＃61;True).round(1)ax &＃61; tel_df.plot(kind&＃61;&＃39;barh&＃39;,figsize&＃61;(110,70),x&＃61;0, y&＃61;1, fontsize&＃61;65)for i in ax.patches:ax.text(i.get_width(), i.get_y() &＃43; 0.2, i.get_width() , fontsize&＃61;70)my_x_ticks &＃61; np.arange(0, 100, 20)plt.xticks(my_x_ticks)plt.show()

In [13]:

check_custom_Detection([&＃39;Telemetry&＃39;,&＃39;MSSP&＃39;])

APT29评估厂商的精准检出统计

对General、Technique、Tactic类型的检出数据进行统计。

结论&＃xff1a;此类数值并不是越大就代表厂商越牛&＃xff0c;仅能反应厂商对部分低误报、低噪点的技术矩阵跟进力度。

In [14]:

check_custom_Detection([&＃39;General&＃39;,&＃39;Technique&＃39;,&＃39;Tactic&＃39;])

APT29评估厂商按照主要检测类型的价值进行打分排名

打分标准&＃xff1a;

None 无检出 0分
Telemetry 遥测型检出 0.5分
MSSP 安全运营型检出 0.6分
General 通用型检出 0.7分
Tactic 战术型检出和Technique 技术型检出均为1分

结论&＃xff1a;

60分以上的形成第一梯队&＃xff0c;厂商只有微小差距

60分以下至55分形成第二梯队&＃xff0c;厂商以1分左右形成梯度差距

55分以下的厂商形成第三梯队&＃xff0c;开始完全掉队

In [15]:

def check_flag_Detection(): flag_list &＃61; []flag_data &＃61; copy.deepcopy(all_data)def check_deep1_Detection(DetectionNote):Detection &＃61; 0for xl in DetectionNote:if xl[&＃39;DetectionType&＃39;]&＃61;&＃61;&＃39;None&＃39;:Detection &＃61; 0elif xl[&＃39;DetectionType&＃39;]&＃61;&＃61;&＃39;N/A&＃39;:Detection &＃61; 0elif xl[&＃39;DetectionType&＃39;]&＃61;&＃61; &＃39;Telemetry&＃39;:Detection &＃61; 0.5elif xl[&＃39;DetectionType&＃39;]&＃61;&＃61; &＃39;MSSP&＃39;:Detection &＃61; 0.6elif xl[&＃39;DetectionType&＃39;]&＃61;&＃61; &＃39;General&＃39;:Detection &＃61; 0.7elif xl[&＃39;DetectionType&＃39;]&＃61;&＃61; (&＃39;Tactic&＃39; or &＃39;Technique&＃39;):Detection &＃61; 1return Detectionfor i,d in flag_data.items():d[&＃39;Detections&＃39;] &＃61; d[&＃39;Detections&＃39;].apply(lambda x: check_deep1_Detection(x)) d_count &＃61;d[&＃39;Detections&＃39;].sum() flag_list.append( (i,d_count) )tel_df &＃61; pd.DataFrame( flag_list ).sort_values(1,ascending&＃61;True).round(1)ax &＃61; tel_df.plot(kind&＃61;&＃39;barh&＃39;,figsize&＃61;(110,70),x&＃61;0, y&＃61;1, fontsize&＃61;65)for i in ax.patches:ax.text(i.get_width(), i.get_y() &＃43; 0.2, i.get_width() , fontsize&＃61;70)my_x_ticks &＃61; np.arange(0, 100, 20)plt.xticks(my_x_ticks)plt.show()

In [16]:

check_flag_Detection()