jm + tracing   14

Dynamic tracing tools for Linux, a la dtrace, ktrace, etc. Built using BPF, using kernel features in the 4.x kernel series, requiring at least version 4.1 of the kernel
linux  tracing  bpf  dynamic  ops 
april 2016 by jm
uses the techniques invented by the authors of Paris-traceroute to enumerate the paths of ECMP flow-based load balancing, but introduces a new technique for NAT detection.

handy. written by AWS SDE Andrea Barberio!
internet  tracing  traceroute  networking  ecmp  nat  ip 
october 2015 by jm
Henry Robinson on testing and fault discovery in distributed systems

'Let's talk about finding bugs in distributed systems for a bit.
These chaos monkey-style fault testing systems are all well and good, but by being application independent they're a very blunt instrument.
Particularly they make it hard to search the fault space for bugs in a directed manner, because they don't 'know' what the system is doing.
Application-aware scripting of faults in a dist. systems seems to be rarely used, but allows you to directly stress problem areas.
For example, if a bug manifests itself only when one RPC returns after some timeout, hard to narrow that down with iptables manipulation.
But allow a script to hook into RPC invocations (and other trace points, like DTrace's probes), and you can script very specific faults.
That way you can simulate cross-system integration failures, *and* write reproducible tests for the bugs they expose!
Anyhow, I've been doing this in Impala, and it's been very helpful. Haven't seen much evidence elsewhere.'
henry-robinson  testing  fault-discovery  rpc  dtrace  tracing  distributed-systems  timeouts  chaos-monkey  impala 
september 2015 by jm
a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.
tracing  java  byteman  injection  jvm  ops  debugging  testing 
september 2015 by jm
a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.

via Eoin Brazil
via:eoinbrazil  httpry  http  networking  tools  ops  testing  tcpdump  tracing 
september 2015 by jm
VPC Flow Logs
we are introducing Flow Logs for the Amazon Virtual Private Cloud.  Once enabled for a particular VPC, VPC subnet, or Elastic Network Interface (ENI), relevant network traffic will be logged to CloudWatch Logs for storage and analysis by your own applications or third-party tools.

You can create alarms that will fire if certain types of traffic are detected; you can also create metrics to help you to identify trends and patterns. The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, the IANA protocol number, packet and byte counts, a time interval during which the flow was observed, and an action (ACCEPT or REJECT).
ec2  aws  vpc  logging  tracing  ops  flow-logs  network  tcpdump  packets  packet-capture 
june 2015 by jm
Apache HTrace
a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator
zipkin  tracing  trace  apache  incubator  java  debugging 
may 2015 by jm
iosnoop For Linux
it's a shell script! ftrace-based tool to snoop on Linux disk I/O and trace system-wide activity, more-or-less attributing it to the correct process
linux  disk  io  tracing  trace  ops  ftrace 
july 2014 by jm
a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements:

* ptrace monitor - tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses;
* pcap sniffer - using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached);
* garbage collector - periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses.

As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets - readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.
debugging  networking  linux  strace  ptrace  tracedump  tracing  tcp  udp  sniffer  ip  tcpdump 
may 2014 by jm
open source, system-level exploration: capture system state and activity from a running Linux instance, then save, filter and analyze.
Think of it as strace + tcpdump + lsof + awesome sauce.
With a little Lua cherry on top.

This sounds excellent. Linux-based, GPLv2.
debugging  tools  linux  ops  tracing  strace  open-source  sysdig  cli  tcpdump  lsof 
april 2014 by jm
"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" [PDF]
Google paper describing the infrastructure they've built for cross-service request tracing (ie. "tracer requests"). Features: low code changes required (since they've built it into the internal protobuf libs), low performance impact, sampling, deployment across the ~entire production fleet, output visibility in minutes, and has been live in production for over 2 years. Excellent read
dapper  tracing  http  services  soa  google  papers  request-tracing  tracers  protobuf  devops 
march 2014 by jm
A sampling profiler for your daily browsing - Google Groups
via Ilya Grigorik: Chrome Canary now has a built-in, always-on, zero-overhead code profiler. I want this in my server-side JVMs!
chrome  tracing  debugging  performance  profiling  google  sampling-profiler  javascript  blink  v8 
january 2014 by jm
DTrace and Erlang
from Basho, via istvan. DTrace is becoming more compelling as a deep instrumentation/monitoring API -- I didn't realise disabled DTrace probes were virtually 0-overhead (a "2 NOOP instruction placeholder", apparently), that's nifty. Wonder if they've fixed the licensing mess, though
dtrace  monitoring  instrumentation  debugging  tracing  unix  erlang  via:istvan 
november 2011 by jm

Copy this bookmark: