Linux 线程的状态

通常的操作系统概念里, 都会区分进程和线程, 可是在 Linux 内核里面, 对于系统调度来说, 它不区分进程和线程, 统一的术语都是线程(Thread), 线程做为调度的基本单位.

对于一个系统来说, 硬件的处理器(CPU)个数(插槽数), 一个 CPU 上的核(core)的数量, 一个核上的硬件线程数(Hyper-Thread)数都是有限的. 有的线程正在 CPU 上运行(on-cpu), 有的不在 CPU 上运行(off-cpu). 那么对于 Thread 来说, 它有哪些状态呢?

通常我们会在 Linux 上面看到这些状态:
• Runnable state 可运行状态,正在 CPU 运行或在队列等待运行
• Sleeping state 等待 I/O 或 调用 sleep 函数 响应 其它 interrupt
• Uninterruptable sleep state 等待 I/O 不响应其它 interrupt
• Defunct or Zombie state 等待父线程/root 线程 ack 死亡

为什么会有僵尸(Zombie)状态?
每个进程运行结束之后, 会返回一个状态码, 本身结束运行之后, 进入 Zombie 状态, 然后创建它的父进程(除了 1 号进程没有父进程)收到这个状态码, 做个响应, 它就可以正常死亡了. 如果它的父进程一直没时间响应, 它就会一直停留在 Zombie 状态. 如果某个进程的父进程早就自己死亡了, 那么 1 号进程就会作为它的父进程进行响应.

如果在 ps 命令下, 我们能看到稍多状态:

D    uninterruptible sleep (usually IO)
I    Idle kernel thread
R    running or runnable (on run queue)
S    interruptible sleep (waiting for an event to complete)
T    stopped by job control signal
t    stopped by debugger during the tracing
X    dead (should never be seen)
Z    defunct ("zombie") process, terminated but not reaped by its parent

对于 ps 命令, 这些状态从哪里获得?
从 /proc/<pid>/task/<pid>/status 得到, 比如:

supra@suprabox:/proc/3119/task/3125$ cat /proc/3119/task/3125/status
Name:    gmain
Umask:    0002
State:    S (sleeping)
Tgid:    3119

对于 sleep 的线程, 它到底从哪个函数进入的 sleep?
通过 cat /proc/<pid>/task/<pid>/wchan 可获得(wchan: waiting channel), 比如:

cat /proc/3119/task/3125/wchan
poll_schedule_timeout.constprop.13

进入 sleep (S & D) 状态的线程, Scheduler 不会再给他们安排 CPU 执行它们的代码. 当线程进入 sleep 之前, 它会进入内核态, 更新它自己在内核的里面 scheduler 数据结构的状态, 告诉内核scheduler 把自己从等待执行的队列中放到一个等待队列中. 当自己等待的信号(I/O interrupt 或者 sleep的时间到), 内核处理 interrupt 的内核进程会重新把它们放到认为等待执行队列. 于是它们重新获得执行机会.

Linux 进程状态 D 状态

有时候, 我们会看到有些进程进入 D 状态, 如下:
cpuD.png

什么是 D 状态:

State "D" (uninterruptible sleep) means that the process is in kernel space (in a system call), usually attempting to perform IO. These processes will not respond to signals (or SIGKILL) and cannot be debugged with gdb or pstack.
An uninterruptible sleep state is a sleep state that won't handle a signal right away. It will wake only as a result of a waited-upon resource becoming available or after a time-out occurs during that wait (if specified when put to sleep). It is mostly used by device drivers waiting for disk or network IO (input/output). When the process is sleeping uninterruptibly, signals accumulated during the sleep will be noticed when the process returns from the system call or trap.

为什么 D 状态影响 load

When load averages first appeared in Linux, they reflected CPU demand, as with other operating systems. But later on Linux changed them to include not only runnable tasks, but also tasks in the uninterruptible state (TASK_UNINTERRUPTIBLE or nr_uninterruptible). This state is used by code paths that want to avoid interruptions by signals, which includes tasks blocked on disk I/O and some locks. You may have seen this state before: it shows up as the "D" state in the output ps and top. The ps(1) man page calls it "uninterruptible sleep (usually IO)".
更多解释: https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

它们不是僵尸进程 (Zombie)

You may sometimes see entries marked Z (or H under Linux, I don't know what the distinction is) in the ps or top output. These are technically not processes, they are zombie processes, which are nothing more than an entry in the process table, kept around so that the parent process can be notified of the death of its child. They will go away when the parent process pays attention (or dies).

为什么 kill -9 <pid> 不能杀掉它?

参见: https://unix.stackexchange.com/questions/5642/what-if-kill-9-does-not-work/5648#5648

Linux 上的 Java 线程优先级

最近看了一篇关于 Java 在 Linux 上设置线程优先级的文章, 受益良多. 原来写了这么多年 Java 代码, 线程优先级在 Linux 上竟然是默认竟然是无效的. 这个网站可能在国内无法访问:
Thread Priority in Linux and Java

大致意思是:
Java 有个 Flag 是: -XX:ThreadPriorityPolicy. 它的值默认是 0, 还有另外一个值是 1. 对着 2 个值的解释是:

0 : Normal.
    VM chooses priorities that are appropriate for normal
    applications. On Solaris NORM_PRIORITY and above are mapped
    to normal native priority. Java priorities below
    NORM_PRIORITY map to lower native priority values. On
    Windows applications are allowed to use higher native
    priorities. However, with ThreadPriorityPolicy=0, VM will
    not use the highest possible native priority,
    THREAD_PRIORITY_TIME_CRITICAL, as it may interfere with
    system threads. On Linux thread priorities are ignored
    because the OS does not support static priority in
    SCHED_OTHER scheduling class which is the only choice for
    non-root, non-realtime applications.
1 : Aggressive.
    Java thread priorities map over to the entire range of
    native thread priorities. Higher Java thread priorities map
    to higher native thread priorities. This policy should be
    used with care, as sometimes it can cause performance
    degradation in the application and/or the entire system. On
    Linux this policy requires root privilege.

所以, 要想 Java 的优先级在 Linux 上生效, 要这么做(既要 sudo 又要设置 flag):

sudo java -XX:ThreadPriorityPolicy=1 ThreadPrioTest

就能看到下面的优先级输出:

supra@suprabox:~$ ps -l -m  -p $(pgrep java)
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 -     0 30388 30387 99   -   - - 2033328 -    pts/2    00:03:00 java
1 S     0     -     - 62  84   4 -     - -      -        00:00:14 -
1 S     0     -     - 66  83   3 -     - -      -        00:00:15 -
1 S     0     -     - 68  82   2 -     - -      -        00:00:15 -
1 S     0     -     - 74  81   1 -     - -      -        00:00:17 -
1 S     0     -     - 78  80   0 -     - -      -        00:00:17 -
1 S     0     -     - 76  79  -1 -     - -      -        00:00:17 -
1 S     0     -     - 79  78  -2 -     - -      -        00:00:18 -
1 S     0     -     - 85  77  -3 -     - -      -        00:00:19 -
1 S     0     -     - 88  76  -4 -     - -      -        00:00:20 -
1 S     0     -     - 90  75  -5 -     - -      -        00:00:20 -

上面的测试文件内容:

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.util.Random;

public class ThreadPrioTest {

    public static void main(String[] args) throws InterruptedException {
        Runnable runnable = new Runnable() {
            public void run() {
                StringBuffer sb = new StringBuffer("");
                while (true) {
                    Random r = new Random();
                    sb.append(r.nextDouble());
                    sb.setLength(300);
                    sb.trimToSize();
                }
            }
        };
        
        for (int i = 0; i < 10; i++) {
            Thread t = new Thread(runnable, "ThreadTest_ " + i);
            t.setPriority(i+1);
            t.start();
        }
        
        ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
        ThreadInfo[] threadInfos = threadMxBean.dumpAllThreads(false, false);
        String name = null;
        for (int i = 0; i < 1000; i++) {
            System.out.println();
            for (ThreadInfo info : threadInfos) {
                name = info.getThreadName();
                if (name.contains("ThreadTest")) {
                    System.out.println(name + " " + threadMxBean.getThreadCpuTime(info.getThreadId()));
                }
            }
            
            Thread.sleep(30000);
        }
        
    }    
}

运行的结果(可以看到占用时间的差别):

supra@suprabox$ sudo java -XX:ThreadPriorityPolicy=1 ThreadPrioTest

ThreadTest_ 0 31115073
ThreadTest_ 1 88458982
ThreadTest_ 2 57653703
ThreadTest_ 3 47742366
ThreadTest_ 4 87852759
ThreadTest_ 5 81910285
ThreadTest_ 6 87854949
ThreadTest_ 7 74596281
ThreadTest_ 8 73344747
ThreadTest_ 9 79761175

   ...
ThreadTest_ 0 94775603625
ThreadTest_ 1 100380714690
ThreadTest_ 2 100932887535
ThreadTest_ 3 108438013681
ThreadTest_ 4 113013836327
ThreadTest_ 5 118102801890
ThreadTest_ 6 121683982580
ThreadTest_ 7 127791024493
ThreadTest_ 8 130891651909
ThreadTest_ 9 134574994529

如果你是在一个有大于 10 个 CPU 核的机器上跑这个测试, 可能结果并不是这样, 因为他们又足够的 CPU, 所以各个优先级的线程使用 CPU 差别不是很大. 所以线程优先级只有在 CPU 不够用的机器上才比较明显.

另外 java 里面关于线程优先级的 flags:

supra@suprabox:~$ java -XX:+PrintFlagsFinal -version | grep riori
     intx CompilerThreadPriority                   = -1                                        {product} {default}
     intx JavaPriority10_To_OSPriority             = -1                                        {product} {default}
     intx JavaPriority1_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority2_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority3_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority4_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority5_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority6_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority7_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority8_To_OSPriority              = -1                                        {product} {default}
     intx JavaPriority9_To_OSPriority              = -1                                        {product} {default}
     intx ThreadPriorityPolicy                     = 0                                         {product} {default}
     bool ThreadPriorityVerbose                    = false                                     {product} {default}
     bool UseThreadPriorities                      = true                                   {pd product} {default}
     intx VMThreadPriority                         = -1                                        {product} {default}
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)

关于之前 Java 里面关于那个 flag 的 bug:
http://www.akshaal.info/2008/04/javas-thread-priorities-in-linux.html
关于 Java 线程优先级的另外一个问答:
https://stackoverflow.com/questions/46936170/cassandra-and-java-9-threadprioritypolicy-42-is-outside-the-allowed-range

gdb

Debugging Symbol Table 把编译的二进制代码对应到响应的变量名, 方法名 和 行号. 符号表可以嵌入到二进制程序中或者在一个单独的文件中.

通过 -g 的参数把符号表嵌入到二进制代码中:

gcc -g hello.cc -o hello 

# 把 debug 信息分离出来
objcopy --only-keep-debug hello hello.debug
# 或者
cp hello hello.debug
strip --only-keep-debug hello.debug

# 去掉 debug 信息
strip --strip-debug --strip-unneeded hello

# 加入 debug 信息
objcopy --add-gnu-debuglink hello.debug hello

# debug 分离的信息
gdb -s hello.debug -e hello

# 或者 debug 启动之后
gdb
(gdb) exec-file  hello
(gdb) symbol-file hello.debug

基本的命令:

file a.out #指定要 debug 的二进制文件
run #启动程序
run 202 "abc" > out.txt # 给要执行的文件参数 并把结果放入 out.txt

break #添加断点到当前行
break myFuction #添加断点到 myFuction
break 5 #添加断点到第五行
break +5 #添加断点到 -> 从当前行往后数5行

info break #查看断点 
info threads

watch x == 3 #如果条件满足, 则暂停

delete 5 #删除第五个断点
clear #清除当前函数的断点

continue #继续运行直到下个断点或错误
finish #继续运行直到当前函数结束
step #运行下一行指令, 如果下一行是函数, 直接进入这个函数, 然后暂停
step 5 #运行后边的5条指令
next #运行下一行, 如果下一行是函数, 直接执行完这个函数

print var #打印变量 var 的值

set x = 3 #设置变量 x 的值为3
set x = y #设置变量 x 的值为 y 的值

bt #查看当前栈
up #到上一层帧
down #到下一层帧
stack #查看当前帧
return #从当前函数返回
select-frame 3 #

help list #打印 list 命令帮助
list 210 #打印出当前文件的第210行左右的行代码
list hello.c:43 #打印 hello.c 的第43行左右的代码
list hello.c:test #打印 hello.c 的test 函数

call myFunc() #执行 myFunc 函数
call myFunc(4) #执行 myFunc 函数
call strlen(str) #执行系统函数 strlen

display x #每一步都显示 x 的值
undisplay x #不要再每一步都显示 x 的值

一篇比较好的关于 GDB debug Java native 代码或 JDK 代码的文章:
https://medium.com/@pirogov.alexey/gdb-debug-native-part-of-java-application-c-c-libraries-and-jdk-6593af3b4f3f

Windows 上解密 chrome 的 https 流量

https 其实是使用传输层的 SSL/TLS 对内容进行加密. SSL/TLS 首先使用非对称加密传输客户端产生的 pre-master secret key. 然后之后使使用对称加密传输内存, 对称加密的的算法使用之前双方都有的 pre-master secret key 作为秘钥. 如果知道了 pre-master secret key, 那么就能解密 https 的流量了. 下面是 windows 上使用 chrome 截取 pre-master secret key 并使用 wireshark 解密的例子.

  1. 设置环境变量 SSLKEYLOGFILE 的值是本地一个文件
    win_env.png
  2. 重新启动 chrome;
  3. 随便访问一个 https 网站, 查看上面设置的文件里是不是已经有 key 产生;
  4. 启动 wireshark, 设置 TLS 的 key 文件.

    菜单: Edit -> Preferences -> Protocols -> TLS -> (pre-)master secret key log file

    wins_key.png

  5. 使用 wireshark 抓包, 或者 wincap 抓包. 抓包后最好使用 wireshark filter 一下. 找到 http1.1 或者 http2 的内容, 就能看到原文了
    wins_wire.png