BbWorld: Forensics

The presentations are available at by Stephen Feldman, Senior Director of Performance Engineering and Architecture. Resouces also available at (Advanced search by and tag: bbworld08 or forensics). Goals of this presentation:

  • Introduce the practice of performance forensics
  • Present an argument fo session level analysis
  • Discuss the difference between resources and interfaces.
  • Present tools that can be used for performance forensics at different layers of the architectural stack and the client layer.
  • We will not be talking about system monitoring.
  • We’re primarily interested in the user’s actions and experiences. How can this be measured?
  • Capture the response time experience and the response time expectations of the user.
  • Everyone needs to be involved.

Performance problems are a result of one of two latencies: queue and response.

Measuring the Session

When should this happen?

  • Basic sampling (non-intrusive to ther user): Fiddler2, Jstack,
  • Continuous Collection (sampling over time, as in less than one hour)
  • Profiling (very intrusive on user performance, such as Profiler). Isolation and analysis of the issue.

Resources vs Interfaces

  • Resources (cpu, memory and I/O) are critical for undersanding the cost of performing a transaction.
  • Interface is the connection between two points, app and database,ie the jdbc connection.

Wait Events

  • Best source of wait event data is the database, such as 10046 tracing.
  • “Waits are statistical explanations of latency.”
  • Each individual wait event might be deceiving, but looking at both aggregates and outliers can explain why a performance problem exists.
  • statspak first 5 things reported are wait events. They are aggregates but are still valuable.
  • When sampling directly, usually only have about 1 per hour (profiling).

Categories of Tools

  • HTTP and User Experience
    • Fiddler 2 measures end-to-end client responsiveness of a web request, has littel or no overhwad (less intrusive forensics), captures requests in order to present http codes, size of objects, sequence of loading, time to process request, performance by bandwidth
    • Firebug, Charlesproxy, liveHTTPheaders and IEInspector
    • Coradiant Truesight a commerical tool which is used for passive user experience monitoring, captures page, object and session level data, capable of defining service level thresholds and automatic incident management, used to traceback sessions.
    • Log Analyzers: Urchin, Summary, WebTrends, SawMill*, Surfstats and AlterWind.
  • JVM Instrumentation Tools
    • JSTAT low intrusve tool, cna be invoked any time wihtout restarting the JVM, by obtaining the process ID
    • Two settings for JVM which can provide more detail -VerboseGC and -Xloggc (restart JVM)
    • A pattern modeling tool for Java GC is IMB Pattenr Modeling tool.
    • JHAT, JMAP and SAP Memory Analyzer (opensource)
  • Database Instrumentation
    • Session and Wait Event
      • ASH: Active Session History (to get squl statements and wait events), which can be used to run “Estimated Cost Plans” so this will tell you how the Optimizer works through the problem.
    • Cost Execution Plans
    • Profilers

Ways to speed up user experience for dial-up users

  • The app does not have compression turned on by default (Weblogic), however, the best place to compress is on your network or on your load balancer.

Recommends Oracle HotSauce Conference in Dallas for introduction to use of Oracle tools (new and old).

Okay, I’m saturated now. I, too, will reference and review the rest of the presentation as posted with slide notes.