A little while back, an old friend, who is an engineering exec at a very big internet company (no names will be given here), was complaining about the performance & stability of a social networking site that is within his company.  So just out of curiosity , I decided to take a look at what he was talking about & created and account on the site. This post presents some of my initial observations & impressions. It is an interesting challenge because, as an outsider to the company, I know virtually nothing about the site’s backend architecture. Since very smart performance folks such as Steve Souders are often saying  performance from the front end perspective is what really matters , I thought this would be a good exercise. To see what I can find out without any knowledge of what is behind the site. I want to stress these are very informal notes & observations which may change over time. I am mostly using very simple principles which are outlined in my more general post on Web Performnce.  For the moment here are my initial impressions & depending upon feedback, I may add a more detailed analysis later:

First a short Non-Performance Observation

Before I go into the performance of this site, there is something which struck me odd / undesirable whilst gathering data. It doesn’t pertain to performance, but I think noteworthy none the less:
The following JSON  is passed on many of page requests I make to this site:
{“Member”:{“id”:”89XXXX4″,
“chatUsername”:”philhXXX@XXX.com”,
“appearOnline”:true,
“age”:”XX”,
“avatarUrl”:”http://i2.XXXo.com/048/ .. /89…45s.jpg”,
“displayName”:”XXX XXX “},”CDN”:{“prefix”:”http://s.XXX.com”},
{ “FederatedLogin”:{“token”:”/wQA … sRgxA=”,
. . . },
{“SessionId”:”0XXX028:philhXXX@XXX.com”,
“API_URL”:”http://api.XXX.XXX.com”,”FetchUrl”:
“http://205….51/../fetchEvents?XXid=003.09028:
philhXXX@XXX.com&rnd=123…21″,
“LANG”:”en-us”,”API_KEY”:”mi1V…c4v”}
};

Looking at the request with a simple sniffer, the following Cookie Data is frequently passed “in the clear”:

Cookie:Email=XXXX@XXX.com; Username=PhilHXXX;
Age=XX;Gen=M;Password=(at least this is obfuscated!)

Just hanging out there with my Email , User Login, Age, IM Ids , state, etc… in Clear Un-obfuscated Text! This just seems wrong to me & in number of ways, which I won’t go into detail now. But you could imagine  a number of scenarios where there could be security & privacy issues here. I’ve checked a few other social networking sites & they all seem to do much more to obfuscate user identity & info.

Some basic performance stats

Let’s just get a few basic stats listed here & out of the way. For this first pass we a looking at 3 pages of the social site:

1. An opening Splash Page at the site’s entry point.
2. A User Home page (similar to Facebook’s Wall).
3. A User Profile page.

Essential Metrics

Table 1 :  YSlow Statistics –  Post Decompression , & Empty Cache.

Fiddler Page Breakdown

Figure 1 : Fiddler Page Component Breakdown.

Basic Load Time for 3 Test Page (No JS Execution)

Figure 2 : Basic Load Time for 3 test pages.

A Few Notes & Observations:

• CDN Use on all 3 measured pages is good.
• Use of GZIP compression on this site is pretty good, but there are a few places where this can be improved.
• Image optimization is good. I see good use of CSS Sprites to keep the request count & page size down.
• CSS Stylesheet usage is structured well & lean.
• The Profile page has a few compression & js minify holes & suffers from a bit of request bloat.
• Browser Perception (both Firefox & IE 7 ) for Profile Page is somewhat “jerky”.
• In Figure 1, Notice how the Splash Page (static content) load is  fairly stable. However, the two dynamic pages have lots of spikes in performance. This suggests some underlying  backend infrastructure issue which is causing a wider variance in performance. In fact, a point (which is not shown in this chart is the two dynamic pages are also registering load errors (component time out ) about 2% of the time!

On the surface, the site is doing many things right & applying a fair number of the principles of Exceptional Performance . However, this is just the surface. Let’s dig a little deeper & specifically to the Profile page:

Next Observations

In timing this site, I have noticed on the Home & Profile pages, frequent stalls in rendering. I have seen this for both Firefox & IE 7 browsers. The next sequence graphs of page component load may give us a clue to some of this perceived behavior.

Initial JS Load for the site

Figure 3: Examination of JavaScript load; Profile Page, Load Start.

The important thing to note in Figure 3 ( initial JS load) is how sequential the load is. This is not un-expected since the bulk of the JS here is being loaded in through conventional script tags:

<script language=”javascript”
src=”http://s.XXX.com/img/yui/yui_2.5.2/cookie/cookie-beta-min.js?fp=79…b8f”></script>
<script language=”javascript” src=”http://s.XXX.com/js/help.js?fp=5c1…337″></script>
<script language=”javascript” src=”http://s.XXX.com/js/tooltip.js?fp=a29…938″></script>
<script language=”javascript”
src=”http://s.XXX.com/js/platform_moo12.js?fp=163…9a0″></script>
<script language=”javascript”
src=”http://s.XXX.com/c/home/index_script.js?fp=f6b…84″></script>
etc, etc, etc,  . . .

Later JS Loads on Profile Page

Figure 4: Examination of JavaScript load; Profile Page, Mid – Page.
However, if we look at the load profile later down the page, we see the same sort of sequential load behavior. This is somewhat unexpected & is probably one of the causes of the “jerkiness” in rendering.
I would recommend loading these scripts with XHR Eval or XHR Injection or one of the other techniques described in Steve Souders’ cuzillion site.  In order to illustrate this point, lets take a look at the initial js load for another very popular social networking site:

Parallel JS Load on a different site

Figure 5 : Javascript Load for a different social site. Notice how the loads are being performed in parallel.

Conclusions

I want to stress here, that this is a very “first pass” simple look at the performance of this social site. In general the traditional tools (that I have been using here) for site performance measurement are really inadequate to fully evaluate very dynamic social networking sites.  The timing information I’ve gathered does suggest some backend issues may be going on. On the JavaScript side, in general I see many things being done right. However, I think more effort place into making JS load parallel would help user experience considerably.  I am also encouraged by what seems to be the adoption of the  mootools framework. I’ve recently worked on a mootools project & I am very impressed with the framework. However in terms of performance  you may want to look at these benchmarks :

- http://tinyurl.com/3tuv7s # Good Comparison Benchmark of JS Frameworks.
- http://mootools.net/slickspeed/ # The Slickspeed Benchmark
- http://tinyurl.com/2lrxnz # a retort to the Slickspeed Benchmark

I’ve spent a good deal of my professional career writing software tools which measure  (& hopefully  improve) the performance of high traffic websites. I’ve worked as both a consultant & permanent engineering staff for a number of sites,  which face the technical & engineering challenges of “getting what they wish for” (i.e. sudden growth & tons of traffic).  This blog category presents an aggregation of  technical notes, data,  & optimization techniques which I have worked on over the years. Some of the data & techniques presented  here may be a little “vintage”. I am including them in this blog  with the intent of keeping notes to myself on certain topics regarding web performance. Other topics, are more current and deal with some of the challenges pertaining to  recent developments (such as AJAX ) . It is clear (to me at least) that the traditional methods & tools for  evaluating & tracking performance just don’t give “us”  (we overworked  & sleep deprived technical folk who are charged with keeping websites performing well ) the full picture.  Some of what is presented in this category strives to look at some of these issues.

Basic Resources

A good place to start in the area of web performance is with a list of resources which I find useful  (updated 04/02/2009):

• Steve Souders’ Performance Site. This is the place to go to get a good picture of some of the latest developements. Some interesting projects here. Lately it has been focused on issues of Javascript performance. His recent talk @ SXSW gives some glimpses into his new book ( Even Faster Web Sites )  which looks very promising. I wasn’t crazy about his last book since for me it seemed like a compliation of knowledge already well know from the web.
• YSlow : A Firefox Plugin which grades webpage performance based on the principles of Exceptional Performance . Note : A recent project I’ve been working on puts many of the principles of YSlow into a scriptable tool which can regularly perform modified YSlow type evaluations.
• Jiffy : An excellent tool for measuring javascript rich sites (supports multiple browsers) .  However, it requires  insertion of Jiffy.js code onto your site (which is not always practical.  I just wish they had  MySQL support ( I couldn’t get the gen_jiffy_base_schema_mysql.rb working!! ).  If I get time, I’m going to look at the MySQL & Ingestor perhaps modifying the ingestor  (Python re-write). A non-performance related  side note: the principle author of Jiffy (Bill Scott of Netflix) has an excellent blog on UI design which I highly recommend.
• Proxy based tools, such as Charles & Fiddler are often very useful tools for getting started with website performance analysis. However, I find  most of these just scratch the surface.
• Some sites which track current developments in website performance I would recommend include ;  website-performance.org , Velocity Conference Proceedings ( 2008 , 2009 ) , & numerous articles on  withing developerWorks & of course , Steve Souders’ Blog .
• If you lean towards the  theoretical , sites such as performance dynamics  & resources from the ACM portal & Springer (  e.g.  sigmetric , IJWET , Chang 2002 ,

Is that all there is ?

Although the basic resources listed above  provide a good starting point , I find a lack of  practical real world data, observations from the trenches out there on the web.  I find this to be especially true when considering issues such as monitoring & analyzing the performance of AJAX based sites.  It is in this area, I hope to develop  a body of techniques , data, & tools for future blog posts to this category.

My biggest pet peave with the current state of the art in performance tools, is their dependancy on manual interaction with the browser.  Modern production websites are constantly changing, often with diverse contributors. Some may be performance aware, & others not so much… This situation begs for the need to have automated systems which check for some of the simple yet effective performance practices continually. Sure, there are services such as Keynote & Gomez out there. But these options are often very  expensive & impractical if you want to  get broad coverage of a site. One of my desired intentions with this blog category is to discuss how we might better automate many of the current state of the art performance measurement & debugging techniques. But that is for later posts. For now, the basics:

Some Frightfully Simple Examples of website performance optimization.

In the course of working on many projects to optimize website performance, I’ve found that it’s often  the little things ( which we sometimes take for granted ) that will get very significant results. For the rest of this post , I am going to put “out there”  a  summary of some past performance work, observations, & experiences which may be useful to by fellow performance engineering bretheren.  In all the examples I present here, I do my best to anonymize the identity of the sites ( I don’t want to get current or past employers / clients upset at me ). Sometimes, I may miss a detail or two which lets the identity slip. Apologies in advance if I do so. This is just a starting point. Future posts will get into some of the thornier issues.

Example 1 : Backend Stability is still important

I know, I know, Lately it is all the rage in various blogs about performance to take the stance that many of the interesting issues are often in the domain of the front end and I completely agree with this idea which is very eloquently discussed by Steve Souders in this video . However, we shouldn’t loose sight  of backend stability. The graph below shows a project I worked on which illustrates this. This Gomez chart shows the load time for a very high trafficked website which I worked on before & after, I did a frightfully simple , common sense review of the site’s backend infrastructure. We are talking really simple stuff here such as adjusting timeout values, buffer pools, load balancer settings & heap size !!!

Page Response Time Before & After Simple Backend Stability Review

Figure 1 Site Load Time Before & After some simple backend optimizations. The picture speaks for itself. In about 1 day, I was able to turn  poorly performing high trafficked website into one that was fairly stable. The first thing I do when I get a call about a website with poor performance is to review the backend architecture & its configuration. Often I find that fixing poor performance is simply a matter of looking at the architecture”s resource allocation  (connection pools, threads per process, heap usage patterns, etc…) and apply a little queuing theory and Viola’  stablity returns & the complaints stop!  This isn’t rocket science & I find that , most of the time, the problem is simply a lack of communication between development, operations, & sysadmin staff. Talk to your admins ! Don’t be afraid to ask them why the TCP window size, or max threads per process on a service  is set to a certain number. Nine times out of ten, the response will be “I don’t know, do you want me to change it ?”. Here is a list of some useful links which often help in such tuning.

• Neil Gunther The Practical Performance Analyst . Although book is somewhat dated, the principles & practices here are timeless. Full of simple examples on the practical application of queuing theory.
• Get to know your servers. Be it Apache , Tomcat, or MySQL tuning, there are a lot of resources out there which give good advice on tuning.
• Look at your connectors ! Often, I find it isn’t the service or service that is the problem. It is often the connectors between. Review your connector implementations be they mod_jk , php-memcached , or your DBCP ,  reviewing & adjusting these settings often yield huge results.

Example 2: Size does Matter.

Yes, it's true. Size does matter

Figure 2, Real World Example of Load Time to Page Size Correlation.

Example 3, Number of Requests (Objects) per Page also matters.

Example of Load Time & Request Bloat

Figure 3 : In the previous example we saw how size matters. Another common malady of poor performance sites (which often can easily be fixed) is what I call “Request Bloat” .

Example 4 : Look at long term trends.

Long Term Performance trends are very important & often missed...

Figure 4: Often we are so busy with the day to day health of a site that we loose our perspective of the long term. If an operator is just looking at the data shown here month by month of week by week, they will miss important trends which signal an underlying performance issue.

In big production environments, the responibility for various system architecture parameters may be spread out  across a number of engineers.

• Site Performance Monitoring using a modified (& scriptable) YSlow .
• Plugin Development work for Hyperic V 3.2 Development.
• Python, JSON, XML projects.
• AWS Elastic Compute Cloud implementations.
• Web Site Performance analysis.
• Various Graphics & Visualization tools & projects.