分类 默认分类 下的文章

静态代码分析工具 Spoon 使用

官网: https://spoon.gforge.inria.fr/launcher.html
它是基于 Eclipse JDT 打造的静态代码分析工具. 它把原代码拆解成包,模块,类,方法,语句,表达式,各种语法单元, 各个语法单元又形成了父子包含等关系. 可以对原代码编译, 检查, 分析, 过滤, 替换, 转换 等.

AST 语法树的元素

语法树包含的各类语法基本单元: https://spoon.gforge.inria.fr/structural_elements.html
详细的代码块形成的各个基本单元: https://spoon.gforge.inria.fr/code_elements.html

setup project

可以分析一个基本的 Java 项目, 一个 Maven 项目, 或者一个 Jar包(通过反编译).

# 分析一个 Maven project 的 source code
MavenLauncher launcher = new MavenLauncher("/Users/tianxiaohui/codes/myProj", SOURCE_TYPE.APP_SOURCE,"/Users/tianxiaohui/apache-maven-3.9.1/");

launcher.getEnvironment().setComplianceLevel(17);
launcher.getEnvironment().setNoClasspath(true); //有些类没提供, 比如 servlet jar 里面的类
launcher.buildModel();

CtModel model = launcher.getModel();

3种情况:

  1. 有源代码 reference.getDeclaration() = reference.getTypeDeclaration()
  2. 没有源代码, 只有binary(jar). reference.getDeclaration() = null, reference.getTypeDeclaration() 反射得来, isShadow = true.
  3. 没有源代码, 也没有binary. reference.getDeclaration() = reference.getTypeDeclaration() = false.
    上面的 getTypeDeclaration 适用于 getFieldDeclaration, getExecutableDeclaration.

常见的代码分析

返回原代码中所有的包和类

for(CtPackage p : model.getAllPackages()) {
  System.out.println("package: " + p.getQualifiedName());
}
// list all classes of the model
for(CtType<?> s : model.getAllTypes()) {
  System.out.println("class: " + s.getQualifiedName());
}

找到一个方法的定义

当你知道一个方法名的时候, 你要查看这个类具体的定义, 可以通过下面的查找方法.

    public static void findMethodDefinition(CtModel ctModel, String clazzName, String methodName) {
        CtClass<?> foundClass = ctModel.getElements(new TypeFilter<CtClass<?>>(CtClass.class) {
            @Override
            public boolean matches(CtClass<?> clazz) {
                return clazz.getQualifiedName().equals(clazzName);
            }
        }).stream().findFirst().orElse(null);

        if (foundClass != null) {
            //System.out.println("Found class definition: " + foundClass);
            foundClass.getMethodsByName(methodName).forEach(m -> {
                System.out.println("Found method definition: " + m.getSignature());
                System.out.println(m.toString());
            });
        } else {
            System.out.println("Class definition not found for: " + clazzName);
        }
    }

    public static void findInvocationPoints(CtModel ctModel, String className, String methodName) {
        System.out.println(" Method " + className + "." + methodName + " is called by:");
        findInvocation(ctModel, className, methodName, 0, false);
    }

查找一个类实例是在哪里构造的

有时候我们要查找某个类是在哪里被初始化的, 可以通过下面的代码获得.

    public static void findNewClassConstruct(CtModel ctModel, String clazzName) {
        long start = System.currentTimeMillis();
        ctModel.getRootPackage().getElements(new TypeFilter<>(CtConstructorCall.class)).forEach(e -> {
            if (e.getExecutable().getDeclaringType().getQualifiedName().equals(clazzName)) {
                System.out.println(clazzName + "is created at: " + e.getPosition());
            }
        });
        System.out.println("time0: " + (System.currentTimeMillis() - start));
    }
    public static void findNewClassConstruct1(CtModel ctModel, String clazzName) {
        long start = System.currentTimeMillis();
        ctModel.getRootPackage().getElements(new TypeFilter<>(CtConstructorCall.class)).forEach(e -> {
            System.out.println(e.getType());
            String type = e.getType().toString();
            if (type.equals(clazzName)) {
                System.out.println(clazzName + "is created at: " + e.getPosition());
            }
        });
        System.out.println("time1: " + (System.currentTimeMillis() - start));
    }

查找一个方法的调用点

一个方法被调用的时候, 它声明的类型可能是它本身的类型, 或者它实现的接口类型, 或者直接/非直接父类的类型. 为了查看完整的可能性, 要能要去每个父类, 实现的接口都去查看一遍. 下面的方法只是查看当前类型的直接调用.

    /**
     *  here we only find the invocation with the exactly class name and method name, not the declared method in
     *  Interface and parent class.
     *  Sometimes you want to find all the invocation points of a method, include the declared method in Interface and parent class.
     *  ex:
     *    IHello hello = new Hello();
     *    hello.sayHello();
     *    In this case, the invocation point of sayHello() is in IHello, not in Hello.
     * @param ctModel
     * @param className
     * @param methodName
     * @param depth
     * @param recursive
     */
    private static void findInvocation(CtModel ctModel, String className, String methodName, int depth, boolean recursive) {

        List<CtInvocation<?>> invocations = ctModel.getElements(new TypeFilter<CtInvocation<?>>(CtInvocation.class) {
            @Override
            public boolean matches(CtInvocation<?> element) {
                return element.getExecutable().getSignature().toString().equals(methodName) && containsRefType(element.getReferencedTypes(), className);
            }
        });

        for (CtInvocation<?> invocation : invocations) {
            CtExecutable<?> caller = invocation.getParent(CtExecutable.class);
            if (caller != null) {
                for (int i = 0; i < depth; i++) {
                    System.out.print("\t");
                }
                System.out.print(" - " + caller.getParent(CtClass.class).getPackage() + "." + caller.getParent(CtClass.class).getSimpleName() + "." + caller.getSignature());
                if (caller.getThrownTypes().size() > 0) {
                    System.out.print(" throws ");
                    System.out.print(caller.getThrownTypes().stream().map(t -> t.toString()).collect(Collectors.joining( ", ")));
                }
                System.out.println(" at line " + invocation.getPosition().getLine());
                if (recursive) {
                    findInvocation(ctModel, caller.getParent(CtClass.class).getQualifiedName(), caller.getSignature(), 1 + depth, recursive);
                }
            }
        }
    }

找到所有抛出异常的代码

下面的代码找出所有抛出异常的代码点. 当然你可以根据异常的类型去过滤.

    public static void findThrowStatements(CtModel ctModel) {
        List<CtThrow> throwStatements = ctModel.getElements(new TypeFilter<>(CtThrow.class));

        // Process each throw statement
        for (CtThrow throwStatement : throwStatements) {
            CtExecutable<?> executable = throwStatement.getParent(CtExecutable.class);
            if (executable != null) {
                System.out.println(executable.getParent(CtClass.class).getPackage() + " - " + executable.getParent(CtClass.class).getSimpleName() + "." + executable.getSimpleName()
                        + " at line " + throwStatement.getPosition().getLine());
            }
        }
    }

找到包含特定注解的类或方法

有时候你想找到特定注解的类, 比如有些注解定义了系统的所有API, 有些注解标注了系统将要废弃的API.

findWithAnnotation(ctModel, javax.ws.rs.ApplicationPath.class).forEach(clazz -> {
            System.out.println(clazz.getAnnotation(javax.ws.rs.ApplicationPath.class).value());
        });


public static List<CtClass> findWithAnnotation(CtModel ctModel, Class<? extends Annotation> annotationType) {
        return ctModel.getRootPackage().getElements(new AnnotationFilter<>(annotationType));

列出某个函数调用的其它函数列表

给出特定函数, 我们可以列出当前函数使用了其他哪些函数

public static void findNextCalls(CtModel ctModel, String pkg, String clazz, String methodName) {
        //find this method
        ctModel.getElements(new TypeFilter<CtMethod<?>>(CtMethod.class) {
            @Override
            public boolean matches(CtMethod<?> method) {
                if (!(method.getParent() instanceof CtClass)) {
                    return false;
                }

                CtClass cz = (CtClass) method.getParent();
                String curClazz = cz.getSimpleName();
                String curPkg = null != cz.getPackage() ? cz.getPackage().getQualifiedName() : "-";

                if (method.getSimpleName().equals(methodName)) {
                    System.out.println(curPkg + " - " + curClazz + " - " + method.getSimpleName());
                }
                return pkg.equals(curPkg) && clazz.equals(curClazz) && method.getSimpleName().equals(methodName);
            }
        }).forEach(m -> {
            System.out.println("Method found: " + m.getSimpleName());
            //find next calls
            List<CtInvocation<?>> invocations = m.getElements(new TypeFilter<>(CtInvocation.class));
            for (CtInvocation<?> invocation : invocations) {
                // Check if the invocation is a client call
                CtExecutableReference exec = invocation.getExecutable();
                System.out.println(m.getSimpleName() + " -> " + invocation.getExecutable().getDeclaringType() + "."
                        + invocation.getExecutable().getSignature());
            }
        });
    }

关于 PromQL 的 histogram_quantile 的算法

关于 PromQL 的 histogram_quantile 的算法. 很长一段时间内, 我都以为是使用正态分布算的, 其实并没有那么复杂.

假如我们有11个桶, 我们的数据是从0开始的, 第一个桶是接收落在(0~10]区间的数量, 第二个桶是接收落在(10 ~ 20]区间的数量, 以此类推, 第10个桶接收落在(90~100]区间的数量, 那么第11个桶接收(100 ~ +Inf) 的区间, 通常最后一个区间应该没有值或非常少.

那么如果要求第 95分位等值, 我们就要统计这时总共有多少samples, 其中第95th/100 落在那个桶. 假如我们收到100个samples, 那么第95th/100 就是第95个sample, 看它落在哪个桶. 如果我们收到10000个samples, 那就是第9500个sample, 看它落在哪个桶.

当我们看到它落到那个桶之后, 就在数一下这个桶共有多少个数, 然后算一下这个桶占据了第多少分位(低)到第多少分位(高), 所以就知道了这个桶占据了从多少分位到多少分位.

然后按照这个桶内的数据是平均分布的, 然后算出第nth(95th)是到底处于哪个值.

这篇问答很好的解释了这个算法是如何工作的:
https://stackoverflow.com/questions/55162093/understanding-histogram-quantile-based-on-rate-in-prometheus/69938239#69938239

官方的开源代码: https://github.com/prometheus/prometheus/blob/main/promql/quantile.go

其中要注意的是:

  1. 分位数值在桶内假定为线性分布进行插值计算
  2. 如果分位数落在最高的桶内,将返回第二高桶的上界值
  3. 如果最低桶的上界大于0,则假定自然下界为0

设计好桶是很关键的一步:

  1. 尽量在较低的桶内平均分布;
  2. 最大值不要超过第二高桶的上界;

对于上面提到的 如果分位数落在最高的桶内,将返回第二高桶的上界值, 下面是一个演示: 其中从 10240 到正无穷也有一些samples, 但是不论我们使用多少9999, 这里最多返回10240.

http_server_requests_duration_ms_bucket{le="5",method="GET",commandName="SigninLegacyView"} 0
http_server_requests_duration_ms_bucket{le="20",method="GET",commandName="SigninLegacyView"} 0
http_server_requests_duration_ms_bucket{le="80",method="GET",commandName="SigninLegacyView"} 30
http_server_requests_duration_ms_bucket{le="320",method="GET",commandName="SigninLegacyView"} 5881
http_server_requests_duration_ms_bucket{le="640",method="GET",commandName="SigninLegacyView"} 8567
http_server_requests_duration_ms_bucket{le="1280",method="GET",commandName="SigninLegacyView"} 8831
http_server_requests_duration_ms_bucket{le="2560",method="GET",commandName="SigninLegacyView"} 8865
http_server_requests_duration_ms_bucket{le="5120",method="GET",commandName="SigninLegacyView"} 8865
http_server_requests_duration_ms_bucket{le="10240",method="GET",commandName="SigninLegacyView"} 8867
http_server_requests_duration_ms_bucket{le="+Inf",method="GET",commandName="SigninLegacyView"} 8885

histogram.png

eBPF 例子

介绍

What Is eBPF? 是一本非好的入门书.
Learning eBPF 是同一个作者的另外一本进阶书.

key notes:

  1. eBPF 程序分为用户态程序和内核态程序. 内核态使用C或Rust编写,然后clang编译成eBPF 字节码,当内核遇到某种event之后,就会执行这些内核态的eBPF程序, 然后生成数据, 放到一些eBPF程序定义的Map中. 用户态程序主要用来加载eBPF程序并且获取内核态写入Map的数据, 然后整理分析展现这些数据.
  2. 内核态的程序编写成代码, 然后编译成字节码, 然后提交给内核, 然后eBPF虚拟机验证(Verify)这些代码,如果安全, 则加载运行这些代码, 等待事件处罚, 执行Action,写入接口文件.
  3. eBPF 为什么需要一个虚拟机? 运行时动态编译, 验证, 不是预先编译链接. 它不是通用型, 而是特定场景.
  4. eBPF bpf_trace_printk() 到 /sys/kernel/debug/tracing/trace_pipe.
  5. eBPF Map 是内核态和用户态共享数据的渠道. 用户态写入配置, 内核态保存中间结果, 最终输出.
    各种类型的 BPF Map: https://elixir.bootlin.com/linux/v5.15.86/source/include/uapi/linux/bpf.h#L878
    Linux Kernel 关于BPF Map的文档: https://docs.kernel.org/bpf/maps.html
  6. eBPF 程序不允许使用其它函数, 除了helper 函数, 所以要 __always_inline .

bpftrace

经典的 one liner 的例子: https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md

教程: https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md

关于网络的部分: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/network-tracing-using-the-bpf-compiler-collection_configuring-and-managing-networking#doc-wrapper

内置变量: comm, pid, tid, args(所有参数: args.filename)
内置函数: str() 把指针的值变成string.
probe 详细信息, 包括参数: sudo bpftrace -vvl kfunc:vmlinux:tcp_set_state

列出所有的 tracepoint:

sudo bpftrace -l 

https://github.com/lizrice/learning-ebpf

etcd 学习笔记

在一个分布式系统中, ETCD 可以:

  1. 分布式的高度一致性的 Key-Value 存储.
  2. 支持 Service 自动发现.
  3. 高并发 每秒10000次写.
  4. 支持双向认证.
  5. 2379 客户请求, 2380 peer 通信.

CAP理论: The CAP theorem states that distributed systems can have at most two out of three of the following properties: high levels of data consistency, high availability of access to the data, and tolerance of network partitions. Networks cannot be assumed to be reliable, so distributed systems need to account for network partitions. Hence, in terms of the CAP theorem, distributed systems can either be AP or CP.

docker 安装 单实例

文档 https://etcd.io/docs/v2.3/docker_guide/ 命令稍有差别 1)改版本, 2) 改etcd 的命令路径

$ export HostIP="10.249.64.103"
$ docker run -d --restart always -v /usr/share/ca-certificates/:/etc/ssl/certs -p 4001:4001 -p 2380:2380 -p 2379:2379 \
 --name etcd quay.io/coreos/etcd /usr/local/bin/etcd \
 -name etcd0 \
 -advertise-client-urls http://${HostIP}:2379,http://${HostIP}:4001 \
 -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
 -initial-advertise-peer-urls http://${HostIP}:2380 \
 -listen-peer-urls http://0.0.0.0:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://${HostIP}:2380 \
 -initial-cluster-state new

登录测试:

$ docker exec -it etcd sh
$ etcdctl
$ etcdctl --endpoints=http://localhost:2379 member list

认证

参考: https://etcd.io/docs/v3.5/demo/
auth, user, rule

版本差别

如果你发现你的 etcdctl 的命令比较少, 那有可能是你使用的版本是V2的命令行, 虽然你的 etcdctl 是 V3 的. 要显示设置使用 v3 的命令 (If using released versions earlier than v3.4, set ETCDCTL_API=3 to use v3 API.):

$ ETCDCTL_API=3 ./etcdctl --endpoints=http://127.0.0.1:2379 get "" --prefix

证书认证参数

这4个global 参数可以通过环境变量设置:
--dial-timeout, --cacert, --cert, --key

ETCDCTL_DIAL_TIMEOUT=3s
ETCDCTL_CACERT=/tmp/ca.pem
ETCDCTL_CERT=/tmp/cert.pem
ETCDCTL_KEY=/tmp/key.pem

ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://localhost:2379 \
              --cert=/etc/etcdtls/etcd-client.crt \
              --key=/etc/etcdtls/etcd-client.key \
              --cacert=/etc/etcdtls/etcd-client-ca.crt get mykey

KV 命令

$ etcdctl put name eric --lease 2e128a4084e72b83
OK
$ etcdctl put name yuan  --lease 2e128a4084e72b83 --prev-kv
OK
name
eric

lease 命令

增: grant, 删: revoke, 改: keep-alive, 查: list, timetolive.

$ etcdctl lease grant  10
lease 2e128a4084e7269c granted with TTL(10s)
$ etcdctl lease revoke  2e128a4084e7269c
lease 2e128a4084e7269c revoked
$ etcdctl lease timetolive  2e128a4084e7269f
lease 2e128a4084e7269f already expired
$ etcdctl lease timetolive 2d8257079fa1bc0c --keys
# lease 2d8257079fa1bc0c granted with TTL(500s), remaining(472s), attached keys([foo2 foo1])
$ etcdctl lease list
32695410dcc0ca06
$ /etcdctl lease keep-alive 32695410dcc0ca0

lock 命令

$ etcdctl lock mylock
# mylock/1234534535445
$ etcdctl lock mylock1 --ttl 10
mylock1/2e128a4084e72b61
$ etcdctl lock mylock echo lock acquired
# lock acquired
$ etcdctl lock mylock ./etcdctl put foo bar
# OK

elect

candidate or observer

$ etcdctl elect president biden
president/2e128a4084e72b68
biden

$ etcdctl elect president -
president/2e128a4084e72b68
biden

$ etcdctl elect president trump

endpoint

查询 revision

etcdctl endpoint status

watch

etcdctl watch --rev=123 /foo

gremlin 操作 janusgraph 的基本语句

核心概念:

  1. Vertex, Edge, Property.
  2. Graph, Directed.
  3. 图数据库即是内存密集型也是CPU密集型.
  4. 图数据库做为其它数据库的一个补充.

建立本地连接

:remote connect tinkerpop.server conf/remote.yaml session
:remote console timeout none

打印 schema

mgmt = graph.openManagement()
mgmt.printSchema()
mgmt.printVertexLabels()
mgmt.printEdgeLabels()
mgmt.printPropertyKeys()
mgmt.printIndexes()

create schema & index

mgmt = graph.openManagement()
batch = mgmt.makeVertexLabel('batch').make()
date = mgmt.makePropertyKey('date').dataType(String.class).make()
hour = mgmt.makePropertyKey('hour').dataType(Integer.class).make()
mgmt.addProperties(batch, date, hour)

mgmt.buildIndex('indexBatchByDateHour', Vertex.class).addKey(date).addKey(hour).buildCompositeIndex()

# 只在这个 Vertex 上的 index
mgmt.buildIndex('indexBatchByDateHour', Vertex.class).addKey(date).addKey(hour).indexOnly(batch).unique().buildCompositeIndex()

mgmt.commit()
g.tx().commit();

对已有数据建立 index

graph.tx().rollback()
mgmt = graph.openManagement()
key = mgmt.getPropertyKey('key')
myIndex = mgmt.buildIndex('byKey', Vertex.class).addKey(key).unique().buildCompositeIndex()
mgmt.commit()
graph.tx().commit();

//Wait for the index to become available
ManagementSystem.awaitGraphIndexStatus(graph, 'byKey').call()


//Reindex the existing data
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("byKey"), SchemaAction.REINDEX).get()
mgmt.commit()

未结束的 transactions

graph.getOpenTransactions()
graph.getOpenTransactions().getAt(0).rollback()
graph.getOpenTransactions().getAt(0).commit()

查看 index

mgmt.getGraphIndexes(Vertex.class)
mgmt.getGraphIndexes(Edge.class)
mgmt.getGraphIndex("indexBatchByDateHour")

查看 console 信息

$> Gremlin.version()
$> :show imports

查看当前 Graph 支持的 features

$ graph.features()
$ graph.toString()

使用 tinkergraph

$ :plugin use tinkerpop.tinkergraph
$ graph = TinkerGraph.open()
$ g = graph.traversal()