微服务全链路跟踪：jaeger集成istio，并兼容ubertraceid与b3

微服务全链路跟踪：grpc集成zipkin

微服务全链路跟踪：grpc集成jaeger

微服务全链路跟踪：springcloud集成jaeger

微服务全链路跟踪：jaeger集成istio，并兼容uber-trace-id与b3

公司有自己的一套基于k8s的paas系统，并且集成了istio，这里主要是想讲解下springcloud服务如何集成istio

jaeger跨进程传递

在基于HTTP协议的分布式调用中，通常会使用HTTP Header来传递SpanContext的内容。常见的Wire Protocol包含Zipkin使用的b3 HTTP header，Jaeger使用的uber-trace-id HTTP Header,LightStep使用的"x-ot-span-context" HTTP Header等。Istio1.0支持b3 header和x-ot-span-context header,可以和Zipkin,Jaeger及LightStep对接；istio1.4以上支持uber-trace-id，请参考github官方说明：https://github.com/istio/istio/issues/12400

uber-trace-id

图中可以看到其中traceId、spanId等字段都拼接到一个头钟了

b3

istio的b3头详情可以参考：https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-request-id b3并没有将字段都拼接，二十单个传递，下面是案例：

X-B3-TraceId:427fde2dc7edb084 X-B3-ParentSpanId:427fde2dc7edb084 X-B3-SpanId:827e270489aafbd7 X-B3-Sampled:1 

变更jaeger传输为b3

如果需要集成istio的jaeger，则需要将传输方式修改为b3 微服务全链路跟踪：springcloud集成jaeger该章中已经描述了如何集成jaeger，这里只需要修改一个配置enable-b3-propagation，如下

opentracing: jaeger: enable-b3-propagation: true// 默认为false udp-sender: host: localhost port: 6831 remote-reporter: flush-interval: 1000 max-queue-size: 5000 log-spans: true probabilistic-sampler: sampling-rate: 1 

这样springboot服务就可以与istio中的jaeger信息串起来形成完整全链路。

兼容uber-trace-id与b3

现在遇到了另外一个问题，公司已经很早就搭建了一套全链路jaeger，并且已经接入了大部分系统，采用的是默认的header传输，即：uber-trace-id 而下游有很多paas内部系统都是非java的不方便接入jaeger，只是注入了istio，并自动注入了jaeger-agent，这里使用的是b3头传输，这就导致了部分链路上下游无法串联起来。而如果需要统一传输方式暂不现实，首先如果都改成b3，则需要上游很多已接入的系统修改配置为b3，如果是都改成uber-trace-id，istio当前版本不支持，如果需要升级istio，则需要升级kubernetes，风险比较大，所以这里根据实际情况先采用集成两种头，即上游都是uber-trace-id，到中间层服务时手动注入b3相关头如下。

grpc注入b3头

这里需要使用grpc的拦截器

import com.google.common.collect.ImmutableMap; import io.grpc.*; import io.opentracing.Span; import io.opentracing.Tracer; import io.opentracing.propagation.Format; import io.opentracing.propagation.TextMap; import lombok.extern.slf4j.Slf4j; import javax.annotation.Nullable; import java.util.Arrays; import java.util.HashSet; import java.util.Iterator; import java.util.Map.Entry; import java.util.Set; import java.util.concurrent.TimeUnit; /** * An intercepter that applies tracing via OpenTracing to all client requests. */ @Slf4j public class ClientTracingInterceptor implements ClientInterceptor { private final Tracer tracer; private final OperationNameConstructor operationNameConstructor; private final boolean streaming; private final boolean verbose; private final Set tracedAttributes; private final ActiveSpanSource activeSpanSource; private final Metadata.Key b3TraceIdKey = Metadata.Key.of("X-B3-TraceId", Metadata.ASCII_STRING_MARSHALLER); private final Metadata.Key b3SpanIdKey = Metadata.Key.of("X-B3-SpanId", Metadata.ASCII_STRING_MARSHALLER); private final Metadata.Key b3ParentSpanIdKey = Metadata.Key.of("X-B3-ParentSpanId", Metadata.ASCII_STRING_MARSHALLER); private final Metadata.Key b3SampledKey = Metadata.Key.of("X-B3-Sampled", Metadata.ASCII_STRING_MARSHALLER); /** * @param */ public ClientTracingInterceptor(Tracer tracer) { this.tracer=tracer; this.operatiOnNameConstructor= OperationNameConstructor.DEFAULT; this.streaming = false; this.verbose = false; this.tracedAttributes = new HashSet(); this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT; } private ClientTracingInterceptor(Tracer tracer, OperationNameConstructor operationNameConstructor, boolean streaming, boolean verbose, Set tracedAttributes, ActiveSpanSource activeSpanSource) { this.tracer = tracer; this.operatiOnNameConstructor= operationNameConstructor; this.streaming = streaming; this.verbose = verbose; this.tracedAttributes = tracedAttributes; this.activeSpanSource = activeSpanSource; } /** * Use this intercepter to trace all requests made by this client channel. * @param channel to be traced * @return intercepted channel */ public Channel intercept(Channel channel) { return ClientInterceptors.intercept(channel, this); } @Override public  ClientCall interceptCall( MethodDescriptor method, CallOptions callOptions, Channel next ) { final String operatiOnName= operationNameConstructor.constructOperationName(method); Span activeSpan = this.activeSpanSource.getActiveSpan(); final Span span = createSpanFromParent(activeSpan, operationName); for (ClientRequestAttribute attr : this.tracedAttributes) { switch (attr) { case ALL_CALL_OPTIONS: span.setTag("grpc.call_options", callOptions.toString()); break; case AUTHORITY: if (callOptions.getAuthority() == null) { span.setTag("grpc.authority", "null"); } else { span.setTag("grpc.authority", callOptions.getAuthority()); } break; case COMPRESSOR: if (callOptions.getCompressor() == null) { span.setTag("grpc.compressor", "null"); } else { span.setTag("grpc.compressor", callOptions.getCompressor()); } break; case DEADLINE: if (callOptions.getDeadline() == null) { span.setTag("grpc.deadline_millis", "null"); } else { span.setTag("grpc.deadline_millis", callOptions.getDeadline().timeRemaining(TimeUnit.MILLISECONDS)); } break; case METHOD_NAME: span.setTag("grpc.method_name", method.getFullMethodName()); break; case METHOD_TYPE: if (method.getType() == null) { span.setTag("grpc.method_type", "null"); } else { span.setTag("grpc.method_type", method.getType().toString()); } break; case HEADERS: break; } } return new ForwardingClientCall.SimpleForwardingClientCall(next.newCall(method, callOptions)) { @Override public void start(Listener responseListener, Metadata headers) { if (verbose) { span.log("Started call"); } if (tracedAttributes.contains(ClientRequestAttribute.HEADERS)) { span.setTag("grpc.headers", headers.toString()); } tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, new TextMap() { @Override public void put(String key, String value) { log.info("jaeger key:{},value:{}",key,value); Metadata.Key headerKey = Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER); headers.put(headerKey, value); String[] mm=value.split("%3A"); if("uber-trace-id".equals(key)&&mm.length==4){ headers.put(b3TraceIdKey,mm[0]); log.info("jaeger traceId:{}",mm[0]); headers.put(b3SpanIdKey,mm[1]); headers.put(b3ParentSpanIdKey,mm[2]); headers.put(b3SampledKey,mm[3]); } } @Override public Iterator> iterator() { throw new UnsupportedOperationException( "TextMapInjectAdapter should only be used with Tracer.inject()"); } }); Listener tracingRespOnseListener= new ForwardingClientCallListener .SimpleForwardingClientCallListener(responseListener) { @Override public void onHeaders(Metadata headers) { if (verbose) { span.log(ImmutableMap.of("Response headers received", headers.toString())); } delegate().onHeaders(headers); } @Override public void onMessage(RespT message) { if (streaming || verbose) { span.log("Response received"); } delegate().onMessage(message); } @Override public void onClose(Status status, Metadata trailers) { if (verbose) { if (status.getCode().value() == 0) { span.log("Call closed"); } else { span.log(ImmutableMap.of("Call failed", status.getDescription())); } } span.finish(); delegate().onClose(status, trailers); } }; delegate().start(tracingResponseListener, headers); } @Override public void cancel(@Nullable String message, @Nullable Throwable cause) { String errorMessage; if (message == null) { errorMessage = "Error"; } else { errorMessage = message; } if (cause == null) { span.log(errorMessage); } else { span.log(ImmutableMap.of(errorMessage, cause.getMessage())); } delegate().cancel(message, cause); } @Override public void halfClose() { if (streaming) { span.log("Finished sending messages"); } delegate().halfClose(); } @Override public void sendMessage(ReqT message) { if (streaming || verbose) { span.log("Message sent"); } delegate().sendMessage(message); } }; } private Span createSpanFromParent(Span parentSpan, String operationName) { if (parentSpan == null) { return tracer.buildSpan(operationName).startManual(); } else { return tracer.buildSpan(operationName).asChildOf(parentSpan).startManual(); } } /** * Builds the configuration of a ClientTracingInterceptor. */ public static class Builder { private Tracer tracer; private OperationNameConstructor operationNameConstructor; private boolean streaming; private boolean verbose; private Set tracedAttributes; private ActiveSpanSource activeSpanSource; /** * @param tracer to use for this intercepter * Creates a Builder with default configuration */ public Builder(Tracer tracer) { this.tracer = tracer; this.operatiOnNameConstructor= OperationNameConstructor.DEFAULT; this.streaming = false; this.verbose = false; this.tracedAttributes = new HashSet(); this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT; } /** * @param operationNameConstructor to name all spans created by this intercepter * @return this Builder with configured operation name */ public Builder withOperationName(OperationNameConstructor operationNameConstructor) { this.operatiOnNameConstructor= operationNameConstructor; return this; } /** * Logs streaming events to client spans. * @return this Builder configured to log streaming events */ public Builder withStreaming() { this.streaming = true; return this; } /** * @param tracedAttributes to set as tags on client spans * created by this intercepter * @return this Builder configured to trace attributes */ public Builder withTracedAttributes(ClientRequestAttribute... tracedAttributes) { this.tracedAttributes = new HashSet( Arrays.asList(tracedAttributes)); return this; } /** * Logs all request life-cycle events to client spans. * @return this Builder configured to be verbose */ public Builder withVerbosity() { this.verbose = true; return this; } /** * @param activeSpanSource that provides a method of getting the * active span before the client call * @return this Builder configured to start client span as children * of the span returned by activeSpanSource.getActiveSpan() */ public Builder withActiveSpanSource(ActiveSpanSource activeSpanSource) { this.activeSpanSource = activeSpanSource; return this; } /** * @return a ClientTracingInterceptor with this Builder's configuration */ public ClientTracingInterceptor build() { return new ClientTracingInterceptor(this.tracer, this.operationNameConstructor, this.streaming, this.verbose, this.tracedAttributes, this.activeSpanSource); } } public enum ClientRequestAttribute { METHOD_TYPE, METHOD_NAME, DEADLINE, COMPRESSOR, AUTHORITY, ALL_CALL_OPTIONS, HEADERS } } 

主要改动点在header那一块

feign注入b3头

@Configuration public class FeignConfig implements RequestInterceptor { @Autowired private final Tracer tracer; @Override public void apply(RequestTemplate requestTemplate) { ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes(); if(attributes!=null&&attributes.getRequest()!=null){ HttpServletRequest request = attributes.getRequest(); JaegerSpanContext cOntext=(JaegerSpanContext) tracer.activeSpan().context(); requestTemplate.header("X-B3-TraceId",String.valueOf(context.getTraceId())); requestTemplate.header("X-B3-SpanId", String.valueOf(context.getSpanId())); requestTemplate.header("X-B3-ParentSpanId", String.valueOf(context.getParentId())); requestTemplate.header("X-B3-Sampled", context.isSampled()?"1":"0"); } } @Bean Logger.Level feignLoggerLevel() { return Logger.Level.FULL; } } 

经过手动注入后，就可以实现上下游串起来，暂时达到目标，后面的方案是统一传输方式，慢慢升级。