Trace Compass Past Present - EclipseCon 2020 › sites › default › files › slides ›...

Ericsson Internal | 2018-02-212018-10-25 | Ericsson 1

Trace Compass PastPresentFuture

Bernd Hufmann, Ericsson AB

2018-10-25


Agenda

— Background

— Trace Compass History

— Use Cases

— Outlook

— Q & A


Background

Eclipse Trace Compass is an open source application to solve performance and reliability issues by reading and analyzing traces and logs of a system.

Eclipse Trace Compass is a framework for creating custom applications to solve performance and reliability issues of a system.


Background

Eclipse Trace Compass is an open source application to solve performance and reliability issues by reading and analyzing traces and logs of a system.

Eclipse Trace Compass is a framework for creating custom applications to solve performance and reliability issues of a system.


Trace Compass history

2009

Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable test/XML parsers— Trace Research Project



2009

2012

LTTng 2.0— Common Trace Format (CTF)— LTTng 2.0 Kernel and UST— LTTng Tracer Control




2009

2012

Standalone Application (RCP)— Analysis Framework— New views types— More trace types (e.g. GDB Trace)


20131




2009

2012



20131

Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable text/XML parsers— Trace Research Project

2014— Own Eclipse Project— More trace types (not only Linux)— Time synchronization of traces



2009

2012



20131


2014— Own Eclipse Project— More trace types (not only Linux)— Time synchronization of traces 2015 Performance Analysis

— Latency and real-time analysis— Pattern Analysis— Critical Path



2016

Call Graph Analysis— Flame Graph View— Function Duration Statistics— Function Duration Density



2016

Trace Compass Add-ons— More trace types

— ftrace, atrace, maven, trace event— Many different views, e.g.

— Generic Flame Chart/ Graph— VM analysis views


2017



2016





2017

2018Jaeger Tracing— Trace parser for Jaeger traces— Various views for Jaeger traces— Container Analysis— In Trace Compass incubator project



2016





2017

2018Jaeger Tracing— Trace parser for Jaeger traces— Various views for Jaeger traces— Container Analysis— In Trace Compass incubator project Distributed Architecture

— Trace Server Protocol (TSP)— Trace Compass trace server— Web-frontend

2019

2020


Collaboration with academia

— Trace Research Project ongoing over many years

— University “Polytechnique Montréal” and others

— Researchers and students contribute their outcome to the

— mainline Trace Compass project

— Trace Compass Incubator Project

— ~30% of commits come from “Polytechnique Montréal”

https://amdls.dorsal.polymtl.ca/

http://projects.eclipse.org/projects/tools.tracecompass.incubator


Data Flow


Data Flow


— Searching

— Filtering

Events Table


— Searching

— Filtering

Highlighting


— Searching

— Filtering

— Highlighting

Searching and Filtering


— Searching

— Filtering

Bookmarking


System tracing

— Linux Tracing Toolkit next generation (LTTng)

— Kernel analysis

— Static tracepoints of the Linux Kernel

— Same tracepoints used by Perf or Ftrace which populate same views

— User Space Tracing (UST)

— Static tracepoints inserted in user application


System tracing Linux kernel analysis


Visual grepPress key: /


Filtering


Resource usage


System tracing highlights

— Follow CPU and IRQ states

— Follow thread execution

— Find high processor load

— Find memory usage issues

— Many views available analyzing Linux kernel traces

— Enhanced highlighting and filtering


— Define a state machine for timing analysis

— Implementation in Java as Trace Compass extension

— Data-driven pattern matching (in XML)

— Defining timing analyses on-the-fly

— Measure between start and end

— Store in a built-in segment store

— Visualize data in various views

Latency analysis


Latency analysis views


Critical path

— Displays of system wait chains for given process across network


Latency analysis highlights

— Find long latencies

— Real-time deadline investigation and verification

— Follow critical path execution

— Performance investigations

— Customizable analysis


Call graph analysisFlame Chart (time based)


Call graph analysisFlame Graph (aggregation)


Call graph analysisFunction duration statistics


Call graph analysis highlights

— Profiling using trace data

— Flame Graph and other views

— Find long executions

— Performance investigation


Trace Compass add-ons

— More analyses

— Call stack analyses

— Trace Types (e.g. Perf, Ftrace, jaeger)

— Virtualization (e.g. VM analysis)

— How to install

— Menu “Tools -> Add-ons…”

— Update Site

http://download.eclipse.org/tracecompass.incubator/master/repository/


OpenTracing API Jaeger tracing support

— Traces according the OpenTracing API

— Implemented to trace Microservices

— Follow flows and spans across services

— Jaeger traces content:


Jaeger trace import

— JSON file directly

— Via HTTP request


Span live viewLogs

Spans

Logs Tooltip


Trace Compass Jaeger highlights

— Correlation with system and application traces

— Drill-down into spans

— Support of large traces

— Easy navigation and zooming

— Other views available

— Possibility to use critical path analysis to reduce spans

— Possibility of custom analysis (XML)


Trace Compass architecture

Trace Compass Core

Trace Compass UI

Trace State

System

Segment

store


Trace Compass architecture

Trace Compass Core

Trace Compass UI

Trace State

System

Segment

store

Trace Compass Core

Trace Compass UI

Trace State

System

Segment

store

Data Providers

Current (ongoing)


Server-client architectureProposal

Trace Compass Core

Trace Compass UI

Trace State

System

Segment

store

Trace Compass Core

Trace Visualization Client

Trace State

System

Segment

store

Trace Server Protocol (REST or RPC)

Data Serializer

Data Providers

Presentation Layer

Business Layer

Data Layer

TraceServer

Data Providers


— Handle the communication between core and UI of trace viewer

— Exchange visualization data between a client and a server

— Trace management

— Various visualization types

— Server side filtering and searching

— Not tracer control

— Proof-of-concept prototype done

— https://github.com/theia-ide/trace-server-protocol

— Currently build for REST over HTTP

— Trace server in Trace Compass incubator project

Scope of Trace Server Protocol (TSP)

https://github.com/theia-ide/trace-server-protocol


Distributed architecture highlights

— Leveraging modern UI technologies and other programming languages

— Better integration with existing web-based services

— CI integration (e.g. open traces directly from Jenkins)

— TR reporting tool (e.g. open traces)

— Integration with workspace management applications (e.g. Che)

— Integration with web-based IDEs

— Leveraging language and debug server (LSP/DAP)

— Allows for usage of cloud/container based technologies (e.g. docker)

— Enables Micro-services architecture

— Distributed services and trace analyses

— Higher scalability

— Easier to change services due decoupled micro-services architecture


— Project pages

— http://tracecompass.org

— http://projects.eclipse.org/projects/tools.tracecompass

— http://projects.eclipse.org/projects/tools.tracecompass.incubator

— Documentation

— Trace Compass User Guide

— Trace Compass Developer Guide

— Trace Research Projects

— http://hsdm.dorsal.polymtl.ca/

— http://amdls.dorsal.polymtl.ca/

References

http://tracecompass.org/

http://projects.eclipse.org/projects/tools.tracecompass

http://projects.eclipse.org/projects/tools.tracecompass.incubator

http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/User-Guide.html

http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.dev/Developer-Guide.html

http://hsdm.dorsal.polymtl.ca/

http://amdls.dorsal.polymtl.ca/


— Presenter

— Bernd Hufmann: [email protected]

— Mailing list

— [email protected]

— IRC

— oftc.net #tracecompass

— Mattermost

— https://mattermost-test.eclipse.org/eclipse/channels/trace-compass

Contacts

mailto:[email protected]

mailto:[email protected]

https://mattermost-test.eclipse.org/eclipse/channels/trace-compass



Q & A

Characters for Embedded characters:!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-

!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-



Date post:	24-Jun-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Trace Compass Past Present - EclipseCon 2020 › sites › default › files › slides ›...

Documents