自定义调度器负载均衡之概念篇(一)

cpu topology

参考文章&＃xff1a;参考文章

基本概念

NUMA

Non-uniform memory access (NUMA) is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data is often associated strongly with certain tasks or users.[1]

NUMA architectures logically follow in scaling from symmetric multiprocessing (SMP) architectures.

**非统一内存访问&＃xff08;NUMA&＃xff09;**是一种用于多处理器的电脑内存体设计&＃xff0c;内存访问时间取决于处理器的内存位置。 在NUMA下&＃xff0c;处理器访问它自己的本地存储器的速度比非本地存储器&＃xff08;存储器的地方到另一个处理器之间共享的处理器或存储器&＃xff09;快一些。

NUMA架构在逻辑上遵循对称多处理&＃xff08;SMP&＃xff09;架构。

DIE/SoC

MC

SMT

SMP

一个NUMA node包括一个或者多个Socket&＃xff0c;以及与之相连的local memory。一个多核的Socket有多个Core。如果CPU支持HT&＃xff0c;OS还会把这个Core看成 2个Logical Processor。

查看cpu拓扑

以使用的服务器为例&＃xff1a;其拓扑结构如下

Topology 2 Processors, 28 Cores, 56 Threads


node节点

numactl命令可以查看node节点信息

[root&＃64;localhost /]# numactl --hardware
available: 2 nodes (0-1) #有个node节点
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 28 29 30 31 32 33 34 35 36 37 38 39 40 41
node 0 size: 78848 MB
node 0 free: 836 MB
node 1 cpus: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 42 43 44 45 46 47 48 49 50 51 52 53 54 55
node 1 size: 62091 MB
node 1 free: 289 MB
node distances: #表示跨node之间的距离&＃xff0c;这里表示跨node之间访问成本是本node之内访问成本的2倍
node 0 10: 10 211: 21 10

查看node节点的详细信息

[root&＃64;localhost /]# ls /sys/devices/system/node/
has_cpu has_memory has_normal_memory node0 node1 online possible power uevent


查看node0节点

[root&＃64;localhost /]# ls /sys/devices/system/node/node0/
compact cpu12/ cpu3/ cpu34/ cpu39/ cpu6/ cpumap memory10/ memory15/ memory2/ memory24/ memory29/ memory33/ memory38/ memory6/ power/
cpu0/ cpu13/ cpu30/ cpu35/ cpu4/ cpu7/ distance memory11/ memory16/ memory20/ memory25/ memory3/ memory34/ memory39/ memory7/ subsystem/
cpu1/ cpu2/ cpu31/ cpu36/ cpu40/ cpu8/ hugepages/ memory12/ memory17/ memory21/ memory26/ memory30/ memory35/ memory4/ memory8/ uevent
cpu10/ cpu28/ cpu32/ cpu37/ cpu41/ cpu9/ meminfo memory13/ memory18/ memory22/ memory27/ memory31/ memory36/ memory40/ memory9/ vmstat
cpu11/ cpu29/ cpu33/ cpu38/ cpu5/ cpulist memory0/ memory14/ memory19/ memory23/ memory28/ memory32/ memory37/ memory5/ numastat


以cpu0举例&＃xff1a;

index0: 1级数据cache
index1: 1级指令cache
index2: 2级cache
index3: 3级cache ,对应cpuinfo里的cache

目录里的文件是cache信息描述&＃xff0c;以本机的cpu0/index0为例简单解释一下&＃xff1a;

[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index0/size
32K
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index1/size
32K
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index2/size
256K
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index3/size
35840K[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index0/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index1/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index2/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu0/cache/index3/shared_cpu_map
0003ff,f0003fff[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu28/cache/index0/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu28/cache/index1/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu28/cache/index2/shared_cpu_map
000000,10000001
[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu28/cache/index3/shared_cpu_map
0003ff,f0003fff[root&＃64;localhost /]# cat /sys/devices/system/node/node0/cpu2/cache/index3/shared_cpu_map
0003ff,f0003fff


processor/socket

[root&＃64;localhost /]# cat /proc/cpuinfo | grep "physical id" | cut -d: -f2 | sort -un01


表示系统含有2个socket。

core

[root&＃64;localhost /]# cat /proc/cpuinfo | grep "cpu cores" | cut -d: -f2 | sort -un14


说明系统每个socket上有14个cores。

[root&＃64;localhost /]# cat /proc/cpuinfo | grep "core id" | cut -d: -f2 | sort -un0123456891011121314


但是core id不一定连续的&＃xff0c;这里就没有core id “7”。

logical processor

[root&＃64;localhost /]# cat /proc/cpuinfo | grep "processor" | cut -d: -f2 | sort -un012345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455


OS认为有56个逻辑CPU&＃xff0c;这是因为开启了HT超线程技术&＃xff0c;每个core同时运行两个线程&＃xff0c;让OS看做实际上有56个逻辑cpu。