Data, Maps, Usability, and Performance

Internet and Website Trends with HTTParchive

Last updated on October 8, 2013 in Data Visualization

website trends

In my last web performance tools article I have mentioned Ilya Grigorik’s Bigqueries forum where people can share and discuss interesting queries on the HTTParchive. These queries can give us some insight into internet trends and the state of the web. Today, I wanted to share two of my favorite queries from that forum and follow up with some of my own queries to identify some interesting website trends. I have decided to execute my queries in MySQL with the latest imported HTTParchive dump. This is mostly because I am constantly running into limits with using the HTTPArchive database on Google BigQuery.

Content Delivery Networks on Top URLs

This query analyzes CDN values in HTTP Archive urls and I am mainly interested in websites that are in the Alexa top 1000. Here are the results (keep in mind that Google is up there mostly due to blogs on Blogger).

  1. No CDN solutions exists on 71.3% of sites
  2. Akamai is implemented on 14.7% of sites
  3. Google is implemented on 8.3% of sites
  4. Lxdns.com is implemented on 1.7% of sites
  5. CloudFlare is implemented on 1.4% of sites
  6. Edgecast is implemented on .5% of sites
  7. ChinaCache is implemented on .4% of sites
  8. Fastly is implemented on .4% of sites
  9. Level 3 is implemented on .4% of sites
  10. CDNetworks is implemented on .3% of sites
  11. LimeLight is implemented on .2% of sites
  12. CubeCDN is implemented on .1% of sites
  13. Amazon CloudFront is implemented on .1% of sites

Akamai clearly owns the CDN market but I really find it hard to believe that 71.3% of websites in the Alexa top 1K are not using any content delivery network.

Top Third-Party Domains

This query attempts to identify domains in the HTTParchive that are the top 3rd party dependencies while also providing the number of requests. Here are the top 40 results:

  1. google.com – 123182 requests
  2. fbcdn.net – 107494 requests
  3. doubleclick.net – 71125 requests
  4. google-analytics.com – 63812 requests
  5. facebook.com – 60707 requests
  6. googlesyndication.com – 55020 requests
  7. twitter.com – 52833 requests
  8. googleusercontent.com – 48093 requests
  9. addthis.com – 44251 requests
  10. googleapis.com – 37042 requests
  11. sharethis.com – 32742 requests
  12. gstatic.com – 32670 requests
  13. blogspot.com – 32063 requests
  14. ytimg.com – 31971 requests
  15. scorecardresearch.com – 22478 requests
  16. adnxs.com – 16942 requests
  17. cloudfront.net – 16159 requests
  18. amazonaws.com – 16056 requests
  19. youtube.com – 15538 requests
  20. gravatar.com – 12461 requests
  21. 2mdn.net – 11970 requests
  22. twimg.com – 10927 requests
  23. mathtag.com – 10412 requests
  24. quantserve.com – 10036 requests
  25. facebook.net – 9565 requests
  26. betrad.com – 9354 requests
  27. wp.com – 9339 requests
  28. truste.com – 9095 requests
  29. akamaihd.net – 8476 requests
  30. blogger.com – 8437 requests
  31. baidu.com – 8147 requests
  32. lijit.com – 7668 requests
  33. pinterest.com – 7437 requests
  34. yandex.ru – 7195 requests
  35. demdex.net – 6996 requests
  36. turn.com – 6871 requests
  37. adsafeprotected.com – 6726 requests
  38. yieldmanager.com – 6669 requests
  39. invitemedia.com – 6231 requests
  40. bluekai.com – 6040 requests

If you consider all the Google properties (google.com, doubleclick.net, google-analytics.com, googlesyndication.com, googleusercontent.com, googleapis.com, gstatic.com, blogspot.com, ytimg.com, youtube.com, 2mdn.net, blogger.com) it is pretty amazing how much of the web leverages Google. It’s not surprising to see Facebook (fbcdn.net, facebook.com, acebook.net), Twitter (twitter.com, twimg.com), and link sharing websites (addthis.com, gravatar.com) on top of this list as well. Amazon (cloudfront.net, amazonaws.com) and advertising companies (adnxs.com, mathtag.com, lijit.com) also make sense but I was a bit surprised to see scorecardresearch.com and turn.com in the top 40 list. Where you surprised by anything on this list?

There are many more awesome queries you can run on the HTTParchive and I decided to import the data into my local version of myQSL and play around with sorting some of the columns. I don’t really like the results from just a complete column sort as you get many unpopular websites so I often filter by Alexa rankings to segment the results. For example, I wanted to see what web pages have the highest amount of errors, but I only care about popular domains, so I limited the search to Alexa top 1000:

SELECT url,numErrors FROM pages WHERE rank<1000 ORDER BY pages.numErrors DESC

Popular Websites with more than 2 errors on page

  1. www.imageshack.us - 11 errors
  2. www.enet.com.cn - 10 errors
  3. www.hubspot.com - 10 errors
  4. www.techcrunch.com - 8 errors
  5. www.p5w.net - 7 errors
  6. www.chron.com - 6 errors
  7. www.elpais.com - 6 errors
  8. www.independent.co.uk - 5 errors
  9. www.peyvandha.ir - 5 errors
  10. www.directrev.com - 3 errors
  11. www.qtrax.com - 3 errors
  12. www.samsung.com - 3 errors
  13. www.intoday.in - 3 errors
  14. www.ci123.com - 3 errors
  15. www.yellowpages.com - 3 errors
  16. www.tutsplus.com - 3 errors
  17. www.sulekha.com - 3 errors
  18. www.today.com - 3 errors
  19. www.r7.com - 3 errors
  20. www.panet.co.il - 3 errors

How about popular websites that have a really bad pagespeed score? First I looked at the Alexa top 1K:

SELECT url,PageSpeed FROM pages WHERE rank<1000 AND PageSpeed<70 ORDER BY pages.PageSpeed ASC

Alexa 1K websites with PageSpeed score under 70

  1. www.soku.com - 16 PageSpeed Score
  2. www.zippyshare.com - 24 PageSpeed Score
  3. www.wunderground.com - 29 PageSpeed Score
  4. www.jvzoo.com - 37 PageSpeed Score
  5. www.correios.com.br - 38 PageSpeed Score
  6. www.zazzle.com - 40 PageSpeed Score
  7. www.ccb.com - 41 PageSpeed Score
  8. www.worldstarhiphop.com - 41 PageSpeed Score
  9. www.workercn.cn - 45 PageSpeed Score
  10. www.onlinesbi.com - 45 PageSpeed Score
  11. www.icbc.com.cn - 45 PageSpeed Score
  12. www.instagram.com - 47 PageSpeed Score
  13. www.directrev.com - 47 PageSpeed Score
  14. www.theverge.com - 47 PageSpeed Score
  15. www.blogfa.com - 48 PageSpeed Score
  16. www.wmmail.ru - 49 PageSpeed Score
  17. www.hdfcbank.com - 50 PageSpeed Score
  18. www.sakura.ne.jp - 50 PageSpeed Score
  19. www.acesse.com - 50 PageSpeed Score
  20. www.nih.gov - 51 PageSpeed Score
  21. www.caixa.gov.br - 51 PageSpeed Score
  22. www.mayoclinic.com - 51 PageSpeed Score
  23. www.naver.jp - 52 PageSpeed Score
  24. www.ocn.ne.jp - 53 PageSpeed Score
  25. www.ctrip.com - 54 PageSpeed Score
  26. www.p5w.net - 55 PageSpeed Score
  27. www.gc.ca - 55 PageSpeed Score
  28. www.bitshare.com - 55 PageSpeed Score
  29. www.novinky.cz - 55 PageSpeed Score
  30. www.mixi.jp - 55 PageSpeed Score
  31. www.xbox.com - 56 PageSpeed Score
  32. www.adf.ly - 57 PageSpeed Score
  33. www.hatena.ne.jp - 57 PageSpeed Score
  34. www.idealo.de - 57 PageSpeed Score
  35. www.persianblog.ir - 58 PageSpeed Score
  36. www.kdnet.net - 58 PageSpeed Score
  37. www.17ok.com - 58 PageSpeed Score
  38. www.webmoney.ru - 59 PageSpeed Score
  39. www.aljazeera.net - 59 PageSpeed Score
  40. www.prweb.com - 59 PageSpeed Score
  41. www.siteadvisor.com - 60 PageSpeed Score
  42. www.yxlady.com - 60 PageSpeed Score
  43. www.pr-cy.ru - 60 PageSpeed Score
  44. www.flippa.com - 60 PageSpeed Score
  45. www.optmd.com - 61 PageSpeed Score
  46. www.sape.ru - 61 PageSpeed Score
  47. www.rbc.ru - 61 PageSpeed Score
  48. www.adobe.com - 62 PageSpeed Score
  49. www.cloob.com - 62 PageSpeed Score
  50. www.cocolog-nifty.com - 62 PageSpeed Score
  51. www.qtrax.com - 63 PageSpeed Score
  52. www.wellsfargo.com - 63 PageSpeed Score
  53. www.clicksor.com - 63 PageSpeed Score
  54. www.oracle.com - 63 PageSpeed Score
  55. www.adclickxpress.com - 63 PageSpeed Score
  56. www.xhamstercams.com - 63 PageSpeed Score
  57. www.free.fr - 64 PageSpeed Score
  58. www.ig.com.br - 64 PageSpeed Score
  59. www.m2newmedia.com - 64 PageSpeed Score
  60. www.ad4game.com - 64 PageSpeed Score
  61. www.adnxs.com - 64 PageSpeed Score
  62. www.icicibank.com - 65 PageSpeed Score
  63. www.leo.org - 65 PageSpeed Score
  64. www.glispa.com - 65 PageSpeed Score
  65. www.pornhublive.com - 65 PageSpeed Score
  66. www.xyxy.net - 66 PageSpeed Score
  67. www.ucoz.ru - 66 PageSpeed Score
  68. www.cj.com - 66 PageSpeed Score
  69. www.webcrawler.com - 66 PageSpeed Score
  70. www.taleo.net - 66 PageSpeed Score
  71. www.itau.com.br - 66 PageSpeed Score
  72. www.ca.gov - 66 PageSpeed Score
  73. www.homeway.com.cn - 66 PageSpeed Score
  74. www.rednet.cn - 66 PageSpeed Score
  75. www.wmtransfer.com - 66 PageSpeed Score
  76. www.wikipedia.org - 67 PageSpeed Score
  77. www.china.com.cn - 67 PageSpeed Score
  78. www.goo.ne.jp - 67 PageSpeed Score
  79. www.jqw.com - 67 PageSpeed Score
  80. www.linkbucks.com - 67 PageSpeed Score
  81. www.lifehacker.com - 67 PageSpeed Score
  82. www.bloglovin.com - 67 PageSpeed Score
  83. www.evernote.com - 67 PageSpeed Score
  84. www.excite.co.jp - 67 PageSpeed Score
  85. www.w3.org - 68 PageSpeed Score
  86. www.java.com - 68 PageSpeed Score
  87. www.t-online.de - 68 PageSpeed Score
  88. www.wiktionary.org - 68 PageSpeed Score
  89. www.bet365.com - 69 PageSpeed Score
  90. www.chaturbate.com - 69 PageSpeed Score
  91. www.letitbit.net - 69 PageSpeed Score
  92. www.eonline.com - 69 PageSpeed Score
  93. www.yomiuri.co.jp - 69 PageSpeed Score
  94. www.makeuseof.com - 69 PageSpeed Score
  95. www.ucoz.com - 69 PageSpeed Score

And then I looked at Alexa 10K websites with a score under 50:

SELECT url,PageSpeed FROM pages WHERE rank<10000 AND PageSpeed<60 ORDER BY pages.PageSpeed ASC

Alexa 10K websites with PageSpeed score under 40

  1. www.rui.jp - 9 PageSpeed Score
  2. www.zocalo.com.mx - 10 PageSpeed Score
  3. www.soku.com - 16 PageSpeed Score
  4. www.firstrowus1.eu - 19 PageSpeed Score
  5. www.firstrownow.eu - 20 PageSpeed Score
  6. www.teamtreehouse.com - 20 PageSpeed Score
  7. www.rs-online.com - 22 PageSpeed Score
  8. www.sulia.com - 23 PageSpeed Score
  9. www.avenues.info - 23 PageSpeed Score
  10. www.begun.ru - 23 PageSpeed Score
  11. www.zippyshare.com - 24 PageSpeed Score
  12. www.dnaindia.com - 25 PageSpeed Score
  13. www.fahrinfo-berlin.de - 25 PageSpeed Score
  14. www.hotfile.com - 26 PageSpeed Score
  15. www.jsfiddle.net - 26 PageSpeed Score
  16. www.bancopopular.es - 26 PageSpeed Score
  17. www.bakecaincontrii.com - 27 PageSpeed Score
  18. www.flvrunner.com - 27 PageSpeed Score
  19. www.1337x.org - 28 PageSpeed Score
  20. www.dedecms.com - 28 PageSpeed Score
  21. www.anoox.com - 28 PageSpeed Score
  22. www.swedbank.lt - 28 PageSpeed Score
  23. www.wunderground.com - 29 PageSpeed Score
  24. www.the9gag.com - 29 PageSpeed Score
  25. www.bnpparibas.net - 30 PageSpeed Score
  26. www.inps.it - 30 PageSpeed Score
  27. www.odn.ne.jp - 30 PageSpeed Score
  28. www.mydocomo.com - 30 PageSpeed Score
  29. www.bidvertiser.com - 31 PageSpeed Score
  30. www.presstv.ir - 31 PageSpeed Score
  31. www.mktmobi.com - 31 PageSpeed Score
  32. www.gsis.gr - 31 PageSpeed Score
  33. www.steamcommunity.com - 32 PageSpeed Score
  34. www.subscribe.ru - 32 PageSpeed Score
  35. www.ana.co.jp - 32 PageSpeed Score
  36. www.4738.com - 32 PageSpeed Score
  37. www.watch32.com - 32 PageSpeed Score
  38. www.internetdownloadmanager.com - 33 PageSpeed Score
  39. www.societegenerale.fr - 33 PageSpeed Score
  40. www.service-public.fr - 33 PageSpeed Score
  41. www.hackforums.net - 33 PageSpeed Score
  42. www.serpbook.com - 33 PageSpeed Score
  43. www.oxfordjournals.org - 33 PageSpeed Score
  44. www.sxc.hu - 34 PageSpeed Score
  45. www.softgozar.com - 34 PageSpeed Score
  46. www.2checkout.com - 34 PageSpeed Score
  47. www.cafef.vn - 34 PageSpeed Score
  48. www.game2.cn - 34 PageSpeed Score
  49. www.cic.fr - 34 PageSpeed Score
  50. www.rqzao.com - 34 PageSpeed Score
  51. www.girlfriendvideos.com - 34 PageSpeed Score
  52. www.xdowns.com - 34 PageSpeed Score
  53. www.freakshare.com - 35 PageSpeed Score
  54. www.dardarkom.com - 35 PageSpeed Score
  55. www.ems.com.cn - 35 PageSpeed Score
  56. www.cwb.gov.tw - 35 PageSpeed Score
  57. www.trk4.com - 35 PageSpeed Score
  58. www.rahnama.com - 35 PageSpeed Score
  59. www.bancoestado.cl - 35 PageSpeed Score
  60. www.pc6.com - 35 PageSpeed Score
  61. www.sportschau.de - 35 PageSpeed Score
  62. www.unibytes.com - 35 PageSpeed Score
  63. www.womai.com - 35 PageSpeed Score
  64. www.pornhost.com - 35 PageSpeed Score
  65. www.enstage.com - 35 PageSpeed Score
  66. www.xtrasize.it - 35 PageSpeed Score
  67. www.adv-adserver.com - 36 PageSpeed Score
  68. www.thaiseoboard.com - 36 PageSpeed Score
  69. www.ingdirect.es - 36 PageSpeed Score
  70. www.centos.org - 36 PageSpeed Score
  71. www.thesitewizard.com - 36 PageSpeed Score
  72. www.canon.jp - 36 PageSpeed Score
  73. www.indiapost.gov.in - 36 PageSpeed Score
  74. www.khmer-news.org - 36 PageSpeed Score
  75. www.jvzoo.com - 37 PageSpeed Score
  76. www.bluedart.com - 37 PageSpeed Score
  77. www.rbi.org.in - 37 PageSpeed Score
  78. www.liex.ru - 37 PageSpeed Score
  79. www.internetbookshop.it - 37 PageSpeed Score
  80. www.baodatviet.vn - 37 PageSpeed Score
  81. www.correios.com.br - 38 PageSpeed Score
  82. www.sbisec.co.jp - 38 PageSpeed Score
  83. www.tnt.com - 38 PageSpeed Score
  84. www.bloggang.com - 38 PageSpeed Score
  85. www.ox.ac.uk - 38 PageSpeed Score
  86. www.live365.com - 38 PageSpeed Score
  87. www.planalto.gov.br - 38 PageSpeed Score
  88. www.valuecommerce.com - 38 PageSpeed Score
  89. www.kingeshop.com - 38 PageSpeed Score
  90. www.postgresql.org - 38 PageSpeed Score
  91. www.registro.br - 39 PageSpeed Score
  92. www.qq.cc - 39 PageSpeed Score
  93. www.deutsche-bank.de - 39 PageSpeed Score
  94. www.tre.it - 39 PageSpeed Score
  95. www.hsbc.com.mx - 39 PageSpeed Score
  96. www.warnerbros.com - 39 PageSpeed Score
  97. www.mojang.com - 39 PageSpeed Score
  98. www.virtualedge.com - 39 PageSpeed Score
  99. www.setlinks.ru - 39 PageSpeed Score
  100. www.mca.gov.in - 39 PageSpeed Score
  101. www.tinychat.com - 39 PageSpeed Score
  102. www.torrenty.org - 39 PageSpeed Score
  103. www.s5themes.com - 39 PageSpeed Score

How about websites with a really low speed index?

SELECT url,SpeedIndex FROM pages WHERE rank<1000 AND SpeedIndex<1000 ORDER BY pages.SpeedIndex ASC

Popular Websites with a SpeedIndex under 1000

  1. www.duckduckgo.com - 513 SpeedIndex Score
  2. www.clkmon.com - 516 SpeedIndex Score
  3. www.indeed.com - 588 SpeedIndex Score
  4. www.google.cn - 600 SpeedIndex Score
  5. www.wordreference.com - 617 SpeedIndex Score
  6. www.search-results.com - 648 SpeedIndex Score
  7. www.youtube-mp3.org - 659 SpeedIndex Score
  8. www.backpage.com - 675 SpeedIndex Score
  9. www.ezpowerads.com - 700 SpeedIndex Score
  10. www.dmoz.org - 710 SpeedIndex Score
  11. www.baidu.com - 711 SpeedIndex Score
  12. www.zeobit.com - 754 SpeedIndex Score
  13. www.blogfa.com - 786 SpeedIndex Score
  14. www.chinabroadcast.cn - 790 SpeedIndex Score
  15. www.optmd.com - 800 SpeedIndex Score
  16. www.kayak.com - 819 SpeedIndex Score
  17. www.intuit.com - 824 SpeedIndex Score
  18. www.pixlr.com - 861 SpeedIndex Score
  19. www.free-tv-video-online.me - 880 SpeedIndex Score
  20. www.grooveshark.com - 886 SpeedIndex Score
  21. www.gtmetrix.com - 892 SpeedIndex Score
  22. www.torrentz.eu - 900 SpeedIndex Score
  23. www.mpnrs.com - 900 SpeedIndex Score
  24. www.4chan.org - 900 SpeedIndex Score
  25. www.keepvid.com - 910 SpeedIndex Score
  26. www.ups.com - 919 SpeedIndex Score
  27. www.zoho.com - 926 SpeedIndex Score
  28. www.google.kz - 932 SpeedIndex Score
  29. www.examiner.com - 940 SpeedIndex Score
  30. www.nairaland.com - 945 SpeedIndex Score
  31. www.4dsply.com - 960 SpeedIndex Score
  32. www.mercadolibre.com.mx - 993 SpeedIndex Score

What other website trends interest you?

Tags: , , ,

Facebook Twitter Hacker News Reddit More...