在实际的虚拟化平台建设项目中，交付物的测试用例都一直是个需要注意的重要事件。期间，设计原始规划是否合理、资源配置是否正确，以及和甲方以及其他参与者的扯皮……
05 摄于北京通州，一个不请自来的松鼠
CPU测试 # 查看CPU信息 [root@beijing ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel( R ) Xeon( R ) Platinum stepping : 4 microcode : 0x1 cpu MHz : 2500. 002 cache size : 33792 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data retbleed gds bogomips : 5000. 00 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: # 使用sysbench命令来进行压力测试 dnf install -y sysbench # 调用2个CPU线程来执行20000次素数计算 sysbench cpu --cpu-max-prime=20000 --threads=2 run # 99元主机 Prime numbers limit: 20000 Initializing worker threads... Threads started! CPU speed: events per second: 823. 16 General statistics: total time: 10. 0024s total number of events: 8235 Latency ( ms ): min: 2. 35 avg: 2. 43 max: 10. 11 95th percentile: 2. 52 sum: 19991. 07 Threads fairness: events ( avg/stddev ): 4117. 5000/18. 50 execution time ( avg/stddev ): 9. 9955/0. 00 # 199元主机 Prime numbers limit: 20000 Initializing worker threads... Threads started! CPU speed: events per second: 552. 77 General statistics: total time: 10. 0021s total number of events: 5530 Latency ( ms ): min: 3. 29 avg: 3. 62 max: 4. 48 95th percentile: 3. 68 sum: 19999. 93 Threads fairness: events ( avg/stddev ): 2765. 0000/7. 00 execution time ( avg/stddev ): 10. 0000/0. 00
内存测试 # 内存测试同样使用sysbench来进行 # memory options: --memory-block-size=SIZE # 内存块大小 [1K] --memory-total-size=SIZE # 传输数据的总大小 [100G] --memory-scope=STRING # 内存访问范围 {global, local} [global] --memory-hugetlb[=on|off] # 从HugeTLB池中分配内存 [off] --memory-oper=STRING # 内存操作类型 {read, write, none} [write] --memory-access-mode=STRING # 内存访问模式 {seq, rnd} [seq] # 测试用例 # 线程数=8 每1s输出一次中间结果 内存块大小=8K 传输数据总量=100G 内存访问模式=顺序访问 内存操作=写 sysbench --threads=8 --time=60 --report-interval=1 --test=memory --memory-block-size=4K --memory-total-size=10G --memory-access-mode=seq run # 99元主机 Total operations: 2621440 ( 1130797. 63 per second ) 10240. 00 MiB transferred ( 4417. 18 MiB/sec ) General statistics: total time: 2. 3168s total number of events: 2621440 Latency ( ms ): min: 0. 00 avg: 0. 01 max: 21. 02 95th percentile: 0. 00 sum: 16055. 58 Threads fairness: events ( avg/stddev ): 327680. 0000/0. 00 execution time ( avg/stddev ): 2. 0069/0. 08 # 199元主机 Total operations: 2621440 ( 3070102. 62 per second ) 10240. 00 MiB transferred ( 11992. 59 MiB/sec ) General statistics: total time: 0. 8521s total number of events: 2621440 Latency ( ms ): min: 0. 00 avg: 0. 00 max: 17. 02 95th percentile: 0. 00 sum: 5260. 94 Threads fairness: events ( avg/stddev ): 327680. 0000/0. 00 execution time ( avg/stddev ): 0. 6576/0. 02
顺序读写测试 测试 # 写测试，创建一个10GB测试文件 dd if=/dev/zero of=/tmp/testspeed bs=1M count=10240 # 清除缓存 sync # 读测试 dd if=/tmp/testspeed of=/dev/null bs=1M
结果 # 99元主机 # 写测试 10737418240 bytes ( 11 GB, 10 GiB ) copied, 88. 9942 s, 121 MB/s # 读测试 10737418240 bytes ( 11 GB, 10 GiB ) copied, 91. 3912 s, 117 MB/s # 199元主机 # 写测试 10737418240 bytes ( 11 GB, 10 GiB ) copied, 83. 6892 s, 128 MB/s # 读测试 10737418240 bytes ( 11 GB, 10 GiB ) copied, 67. 7363 s, 159 MB/s # 阿里云c7. xlarge 规格主机 # 写测试 10737418240 bytes ( 11 GB ) copied, 62. 7431 s, 171 MB/s # 读测试 10737418240 bytes ( 11 GB ) copied, 59. 4358 s, 181 MB/s
随机读写测试 测试用例 # fileio options( []为默认参数 ): --file-num=N # 创建的文件数量 [128] --file-block-size=N # 在所有IO操作中使用的块大小 [16384] --file-total-size=SIZE # 要创建的文件的总大小 [2G] --file-test-mode=STRING # 测试模式 {seqwr( 顺序写 ), seqrewr( 顺序重写 ), seqrd( 顺序读 ), rndrd( 随机读 ), rndwr( 随机写 ), rndrw( 随机读写 )} --file-io-mode=STRING # 文件操作模式 {sync( 同步 ), async( 异步 ), mmap} [sync] --file-extra-flags=[LIST, ...] # 用于打开文件的附加标志列表 {sync, dsync, direct} [] --file-fsync-freq=N # 执行N条请求数量后执行fsync() ( 0 - don't use fsync() ) [100] --file-fsync-all[=on|off] # 每条写指令后执行fsync() [off] --file-fsync-end[=on|off] # 测试执行后执行fsync() [on] --file-fsync-mode=STRING # 同步方式 {fsync, fdatasync} [fsync] --file-merged-requests=N # 允许范围内，最多合并IO请求数量 （ 0 - don't merge ） [0] --file-rw-ratio=N # 组合测试读/写比率 [1. 5] # 线程数=4 每隔4s输出一次结果 测试时间=60s # 文件数=2 文件总大小=4G 文件操作模式=随机读写 # 块大小 16384 sysbench --threads=4 --report-interval=4 --time=60 --test=fileio --file-num=2 --file-total-size=4G --file-test-mode=rndrw prepare sysbench --threads=4 --report-interval=4 --time=60 --test=fileio --file-num=2 --file-total-size=4G --file-test-mode=rndrw run sysbench --threads=4 --report-interval=4 --time=60 --test=fileio --file-num=2 --file-total-size=4G --file-test-mode=rndrw cleanup
测试结果 # 99元主机 File operations: reads/s: 2239. 65 writes/s: 1493. 08 fsyncs/s: 74. 78 Throughput: read, MiB/s: 34. 99 written, MiB/s: 23. 33 General statistics: total time: 60. 0113s total number of events: 228492 Latency ( ms ): min: 0. 00 avg: 1. 05 max: 107. 37 95th percentile: 7. 17 sum: 239909. 29 Threads fairness: events ( avg/stddev ): 57123. 0000/437. 20 execution time ( avg/stddev ): 59. 9773/0. 00 # 199元主机 File operations: reads/s: 1574. 11 writes/s: 1049. 40 fsyncs/s: 52. 58 Throughput: read, MiB/s: 24. 60 written, MiB/s: 16. 40 General statistics: total time: 60. 0261s total number of events: 160631 Latency ( ms ): min: 0. 00 avg: 1. 49 max: 44. 26 95th percentile: 7. 84 sum: 239862. 23 Threads fairness: events ( avg/stddev ): 40157. 7500/118. 32 execution time ( avg/stddev ): 59. 9656/0. 00
网络测试 该测试首先测试在使用阿里云公共网络的情况下，张家口主机访问北京主机的网速和北京主机访问张家口主机的网速。在阿里云内网测试中，则需要在北京和张家口的主机对应区域和vpc再创建同规格主机进行测试。
公网测试 # 99元主机 [root@zhangjiakou ~]# scp /tmp/testspeed test@39. 102. 209. 9:/home/test/ testspeed 0% 29MB 405. 5KB/s # 199元主机 [root@beijing ~]# scp /tmp/testspeed test@39. 98. 54. 184:/home/test/ testspeed 0% 15MB 745. 4KB/s
内网测试 使用scp测试 # 99元主机 [root@zhangjiakou ~]# scp /tmp/testspeed test@172. 24. 78. 59:/home/test/ testspeed 31% 3266MB 118. 2MB/s # 199元主机 [root@beijing ~]# scp /tmp/testspeed test@172. 17. 83. 163:/home/test/ testspeed 60% 6216MB 112. 6MB/s
使用iperf3测试 # 在所有主机上安装iperf3，然后配置安全组添加9001端口、防火墙开启9001端口或者关闭防火墙 dnf install -y iperf3 # 99元主机 # 被测主机启动监听 [root@zhangjiakou ~]# iperf3 -s -p 9001 ----------------------------------------------------------- Server listening on 9001 ----------------------------------------------------------- # 实施测试主机，-c 目标IP -i 每3秒刷新一次 -p 连接端口 -t 执行时长 -f 数据单位 -P 并发数量 [root@Test ~]# iperf3 -c 172. 24. 78. 59 -i 3 -p 9001 -t 100 -f M -P 20 [ 37] 3. 00-4. 17 sec 13. 8 MBytes 11. 7 MBytes/sec 299 2. 83 KBytes [ 39] 3. 00-4. 17 sec 8. 75 MBytes 7. 47 MBytes/sec 239 1. 41 KBytes [ 41] 3. 00-4. 17 sec 16. 2 MBytes 13. 9 MBytes/sec 555 4. 24 KBytes [ 43] 3. 00-4. 17 sec 12. 9 MBytes 11. 0 MBytes/sec 362 4. 24 KBytes [SUM] 3. 00-4. 17 sec 282 MBytes 241 MBytes/sec 8224 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0. 00-4. 17 sec 78. 7 MBytes 18. 9 MBytes/sec 2006 sender [ 5] 0. 00-4. 17 sec 0. 00 Bytes 0. 00 MBytes/sec receiver [ 7] 0. 00-4. 17 sec 45. 6 MBytes 10. 9 MBytes/sec 1386 sender [ 7] 0. 00-4. 17 sec 0. 00 Bytes 0. 00 MBytes/sec receiver [ 9] 0. 00-4. 17 sec 33. 5 MBytes 8. 03 MBytes/sec 970 sender [ 9] 0. 00-4. 17 sec 0. 00 Bytes 0. 00 MBytes/sec receiver # 被测试主机抓取瞬时流量，rxkB/s为接受流量，txkB/s为发送流量 # 270571KB/s折合250MB/s [root@zhangjiakou ~]# sar -n DEV 2 Linux 5. 14. 0-427. 31. 1. el9_4. x86_64 ( zhangjiakou ) 08/28/2024 _x86_64_ ( 2 CPU ) 10:11:05 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 10:11:07 PM lo 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 10:11:07 PM eth0 183009. 50 23761. 50 270571. 87 1568. 17 0. 00 0. 00 0. 00 # 199元主机，128446. 22KB/s折合128MB/s [root@beijing ~]# iperf3 -s -p 9001 [root@Test ~]# iperf3 -c 172. 17. 83. 162 -i 3 -p 9001 -t 100 -f M -P 20 [root@beijing ~]# sar -n DEV 2 Linux 5. 14. 0-427. 31. 1. el9_4. x86_64 ( beijing ) 08/28/2024 _x86_64_ ( 2 CPU ) 10:28:40 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 10:28:42 PM lo 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 0. 00 10:28:42 PM eth0 86878. 00 24077. 00 128446. 22 1589. 40 0. 00 0. 00 0. 00 0. 00
使用iftop
99元主机，显示为1. 88Gb，和sar命令结果相符（注意，iftop显示单位为b而非B）
199元主机，显示结果为710Mb
测试结论 先说结果，两者都是垃圾。最有意思的是张家口的99元主机性能比北京的199元主机性能除了内存性能明显落后外，其他并不占下风。这100块是多出了个寂寞……
