Internet and Website Trends with HTTParchive

Last updated on October 8, 2013 in Data Visualization

website trends

In my last web performance tools article I have mentioned Ilya Grigorik’s Bigqueries forum where people can share and discuss interesting queries on the HTTParchive. These queries can give us some insight into internet trends and the state of the web. Today, I wanted to share two of my favorite queries from that forum and follow up with some of my own queries to identify some interesting website trends. I have decided to execute my queries in MySQL with the latest imported HTTParchive dump. This is mostly because I am constantly running into limits with using the HTTPArchive database on Google BigQuery.

Content Delivery Networks on Top URLs

This query analyzes CDN values in HTTP Archive urls and I am mainly interested in websites that are in the Alexa top 1000. Here are the results (keep in mind that Google is up there mostly due to blogs on Blogger).

No CDN solutions exists on 71.3% of sites
Akamai is implemented on 14.7% of sites
Google is implemented on 8.3% of sites
Lxdns.com is implemented on 1.7% of sites
CloudFlare is implemented on 1.4% of sites
Edgecast is implemented on .5% of sites
ChinaCache is implemented on .4% of sites
Fastly is implemented on .4% of sites
Level 3 is implemented on .4% of sites
CDNetworks is implemented on .3% of sites
LimeLight is implemented on .2% of sites
CubeCDN is implemented on .1% of sites
Amazon CloudFront is implemented on .1% of sites

Akamai clearly owns the CDN market but I really find it hard to believe that 71.3% of websites in the Alexa top 1K are not using any content delivery network.

Top Third-Party Domains

This query attempts to identify domains in the HTTParchive that are the top 3rd party dependencies while also providing the number of requests. Here are the top 40 results:

google.com – 123182 requests
fbcdn.net – 107494 requests
doubleclick.net – 71125 requests
google-analytics.com – 63812 requests
facebook.com – 60707 requests
googlesyndication.com – 55020 requests
twitter.com – 52833 requests
googleusercontent.com – 48093 requests
addthis.com – 44251 requests
googleapis.com – 37042 requests
sharethis.com – 32742 requests
gstatic.com – 32670 requests
blogspot.com – 32063 requests
ytimg.com – 31971 requests
scorecardresearch.com – 22478 requests
adnxs.com – 16942 requests
cloudfront.net – 16159 requests
amazonaws.com – 16056 requests
youtube.com – 15538 requests
gravatar.com – 12461 requests
2mdn.net – 11970 requests
twimg.com – 10927 requests
mathtag.com – 10412 requests
quantserve.com – 10036 requests
facebook.net – 9565 requests
betrad.com – 9354 requests
wp.com – 9339 requests
truste.com – 9095 requests
akamaihd.net – 8476 requests
blogger.com – 8437 requests
baidu.com – 8147 requests
lijit.com – 7668 requests
pinterest.com – 7437 requests
yandex.ru – 7195 requests
demdex.net – 6996 requests
turn.com – 6871 requests
adsafeprotected.com – 6726 requests
yieldmanager.com – 6669 requests
invitemedia.com – 6231 requests
bluekai.com – 6040 requests

If you consider all the Google properties (google.com, doubleclick.net, google-analytics.com, googlesyndication.com, googleusercontent.com, googleapis.com, gstatic.com, blogspot.com, ytimg.com, youtube.com, 2mdn.net, blogger.com) it is pretty amazing how much of the web leverages Google. It’s not surprising to see Facebook (fbcdn.net, facebook.com, acebook.net), Twitter (twitter.com, twimg.com), and link sharing websites (addthis.com, gravatar.com) on top of this list as well. Amazon (cloudfront.net, amazonaws.com) and advertising companies (adnxs.com, mathtag.com, lijit.com) also make sense but I was a bit surprised to see scorecardresearch.com and turn.com in the top 40 list. Where you surprised by anything on this list?

There are many more awesome queries you can run on the HTTParchive and I decided to import the data into my local version of myQSL and play around with sorting some of the columns. I don’t really like the results from just a complete column sort as you get many unpopular websites so I often filter by Alexa rankings to segment the results. For example, I wanted to see what web pages have the highest amount of errors, but I only care about popular domains, so I limited the search to Alexa top 1000:

SELECT url,numErrors FROM pages WHERE rank<1000 ORDER BY pages.numErrors DESC

Popular Websites with more than 2 errors on page

www.imageshack.us - 11 errors
www.enet.com.cn - 10 errors
www.hubspot.com - 10 errors
www.techcrunch.com - 8 errors
www.p5w.net - 7 errors
www.chron.com - 6 errors
www.elpais.com - 6 errors
www.independent.co.uk - 5 errors
www.peyvandha.ir - 5 errors
www.directrev.com - 3 errors
www.qtrax.com - 3 errors
www.samsung.com - 3 errors
www.intoday.in - 3 errors
www.ci123.com - 3 errors
www.yellowpages.com - 3 errors
www.tutsplus.com - 3 errors
www.sulekha.com - 3 errors
www.today.com - 3 errors
www.r7.com - 3 errors
www.panet.co.il - 3 errors

How about popular websites that have a really bad pagespeed score? First I looked at the Alexa top 1K:

SELECT url,PageSpeed FROM pages WHERE rank<1000 AND PageSpeed<70 ORDER BY pages.PageSpeed ASC

Alexa 1K websites with PageSpeed score under 70

www.soku.com - 16 PageSpeed Score
www.zippyshare.com - 24 PageSpeed Score
www.wunderground.com - 29 PageSpeed Score
www.jvzoo.com - 37 PageSpeed Score
www.correios.com.br - 38 PageSpeed Score
www.zazzle.com - 40 PageSpeed Score
www.ccb.com - 41 PageSpeed Score
www.worldstarhiphop.com - 41 PageSpeed Score
www.workercn.cn - 45 PageSpeed Score
www.onlinesbi.com - 45 PageSpeed Score
www.icbc.com.cn - 45 PageSpeed Score
www.instagram.com - 47 PageSpeed Score
www.directrev.com - 47 PageSpeed Score
www.theverge.com - 47 PageSpeed Score
www.blogfa.com - 48 PageSpeed Score
www.wmmail.ru - 49 PageSpeed Score
www.hdfcbank.com - 50 PageSpeed Score
www.sakura.ne.jp - 50 PageSpeed Score
www.acesse.com - 50 PageSpeed Score
www.nih.gov - 51 PageSpeed Score
www.caixa.gov.br - 51 PageSpeed Score
www.mayoclinic.com - 51 PageSpeed Score
www.naver.jp - 52 PageSpeed Score
www.ocn.ne.jp - 53 PageSpeed Score
www.ctrip.com - 54 PageSpeed Score
www.p5w.net - 55 PageSpeed Score
www.gc.ca - 55 PageSpeed Score
www.bitshare.com - 55 PageSpeed Score
www.novinky.cz - 55 PageSpeed Score
www.mixi.jp - 55 PageSpeed Score
www.xbox.com - 56 PageSpeed Score
www.adf.ly - 57 PageSpeed Score
www.hatena.ne.jp - 57 PageSpeed Score
www.idealo.de - 57 PageSpeed Score
www.persianblog.ir - 58 PageSpeed Score
www.kdnet.net - 58 PageSpeed Score
www.17ok.com - 58 PageSpeed Score
www.webmoney.ru - 59 PageSpeed Score
www.aljazeera.net - 59 PageSpeed Score
www.prweb.com - 59 PageSpeed Score
www.siteadvisor.com - 60 PageSpeed Score
www.yxlady.com - 60 PageSpeed Score
www.pr-cy.ru - 60 PageSpeed Score
www.flippa.com - 60 PageSpeed Score
www.optmd.com - 61 PageSpeed Score
www.sape.ru - 61 PageSpeed Score
www.rbc.ru - 61 PageSpeed Score
www.adobe.com - 62 PageSpeed Score
www.cloob.com - 62 PageSpeed Score
www.cocolog-nifty.com - 62 PageSpeed Score
www.qtrax.com - 63 PageSpeed Score
www.wellsfargo.com - 63 PageSpeed Score
www.clicksor.com - 63 PageSpeed Score
www.oracle.com - 63 PageSpeed Score
www.adclickxpress.com - 63 PageSpeed Score
www.xhamstercams.com - 63 PageSpeed Score
www.free.fr - 64 PageSpeed Score
www.ig.com.br - 64 PageSpeed Score
www.m2newmedia.com - 64 PageSpeed Score
www.ad4game.com - 64 PageSpeed Score
www.adnxs.com - 64 PageSpeed Score
www.icicibank.com - 65 PageSpeed Score
www.leo.org - 65 PageSpeed Score
www.glispa.com - 65 PageSpeed Score
www.pornhublive.com - 65 PageSpeed Score
www.xyxy.net - 66 PageSpeed Score
www.ucoz.ru - 66 PageSpeed Score
www.cj.com - 66 PageSpeed Score
www.webcrawler.com - 66 PageSpeed Score
www.taleo.net - 66 PageSpeed Score
www.itau.com.br - 66 PageSpeed Score
www.ca.gov - 66 PageSpeed Score
www.homeway.com.cn - 66 PageSpeed Score
www.rednet.cn - 66 PageSpeed Score
www.wmtransfer.com - 66 PageSpeed Score
www.wikipedia.org - 67 PageSpeed Score
www.china.com.cn - 67 PageSpeed Score
www.goo.ne.jp - 67 PageSpeed Score
www.jqw.com - 67 PageSpeed Score
www.linkbucks.com - 67 PageSpeed Score
www.lifehacker.com - 67 PageSpeed Score
www.bloglovin.com - 67 PageSpeed Score
www.evernote.com - 67 PageSpeed Score
www.excite.co.jp - 67 PageSpeed Score
www.w3.org - 68 PageSpeed Score
www.java.com - 68 PageSpeed Score
www.t-online.de - 68 PageSpeed Score
www.wiktionary.org - 68 PageSpeed Score
www.bet365.com - 69 PageSpeed Score
www.chaturbate.com - 69 PageSpeed Score
www.letitbit.net - 69 PageSpeed Score
www.eonline.com - 69 PageSpeed Score
www.yomiuri.co.jp - 69 PageSpeed Score
www.makeuseof.com - 69 PageSpeed Score
www.ucoz.com - 69 PageSpeed Score

And then I looked at Alexa 10K websites with a score under 50:

SELECT url,PageSpeed FROM pages WHERE rank<10000 AND PageSpeed<60 ORDER BY pages.PageSpeed ASC

Alexa 10K websites with PageSpeed score under 40

www.rui.jp - 9 PageSpeed Score
www.zocalo.com.mx - 10 PageSpeed Score
www.soku.com - 16 PageSpeed Score
www.firstrowus1.eu - 19 PageSpeed Score
www.firstrownow.eu - 20 PageSpeed Score
www.teamtreehouse.com - 20 PageSpeed Score
www.rs-online.com - 22 PageSpeed Score
www.sulia.com - 23 PageSpeed Score
www.avenues.info - 23 PageSpeed Score
www.begun.ru - 23 PageSpeed Score
www.zippyshare.com - 24 PageSpeed Score
www.dnaindia.com - 25 PageSpeed Score
www.fahrinfo-berlin.de - 25 PageSpeed Score
www.hotfile.com - 26 PageSpeed Score
www.jsfiddle.net - 26 PageSpeed Score
www.bancopopular.es - 26 PageSpeed Score
www.bakecaincontrii.com - 27 PageSpeed Score
www.flvrunner.com - 27 PageSpeed Score
www.1337x.org - 28 PageSpeed Score
www.dedecms.com - 28 PageSpeed Score
www.anoox.com - 28 PageSpeed Score
www.swedbank.lt - 28 PageSpeed Score
www.wunderground.com - 29 PageSpeed Score
www.the9gag.com - 29 PageSpeed Score
www.bnpparibas.net - 30 PageSpeed Score
www.inps.it - 30 PageSpeed Score
www.odn.ne.jp - 30 PageSpeed Score
www.mydocomo.com - 30 PageSpeed Score
www.bidvertiser.com - 31 PageSpeed Score
www.presstv.ir - 31 PageSpeed Score
www.mktmobi.com - 31 PageSpeed Score
www.gsis.gr - 31 PageSpeed Score
www.steamcommunity.com - 32 PageSpeed Score
www.subscribe.ru - 32 PageSpeed Score
www.ana.co.jp - 32 PageSpeed Score
www.4738.com - 32 PageSpeed Score
www.watch32.com - 32 PageSpeed Score
www.internetdownloadmanager.com - 33 PageSpeed Score
www.societegenerale.fr - 33 PageSpeed Score
www.service-public.fr - 33 PageSpeed Score
www.hackforums.net - 33 PageSpeed Score
www.serpbook.com - 33 PageSpeed Score
www.oxfordjournals.org - 33 PageSpeed Score
www.sxc.hu - 34 PageSpeed Score
www.softgozar.com - 34 PageSpeed Score
www.2checkout.com - 34 PageSpeed Score
www.cafef.vn - 34 PageSpeed Score
www.game2.cn - 34 PageSpeed Score
www.cic.fr - 34 PageSpeed Score
www.rqzao.com - 34 PageSpeed Score
www.girlfriendvideos.com - 34 PageSpeed Score
www.xdowns.com - 34 PageSpeed Score
www.freakshare.com - 35 PageSpeed Score
www.dardarkom.com - 35 PageSpeed Score
www.ems.com.cn - 35 PageSpeed Score
www.cwb.gov.tw - 35 PageSpeed Score
www.trk4.com - 35 PageSpeed Score
www.rahnama.com - 35 PageSpeed Score
www.bancoestado.cl - 35 PageSpeed Score
www.pc6.com - 35 PageSpeed Score
www.sportschau.de - 35 PageSpeed Score
www.unibytes.com - 35 PageSpeed Score
www.womai.com - 35 PageSpeed Score
www.pornhost.com - 35 PageSpeed Score
www.enstage.com - 35 PageSpeed Score
www.xtrasize.it - 35 PageSpeed Score
www.adv-adserver.com - 36 PageSpeed Score
www.thaiseoboard.com - 36 PageSpeed Score
www.ingdirect.es - 36 PageSpeed Score
www.centos.org - 36 PageSpeed Score
www.thesitewizard.com - 36 PageSpeed Score
www.canon.jp - 36 PageSpeed Score
www.indiapost.gov.in - 36 PageSpeed Score
www.khmer-news.org - 36 PageSpeed Score
www.jvzoo.com - 37 PageSpeed Score
www.bluedart.com - 37 PageSpeed Score
www.rbi.org.in - 37 PageSpeed Score
www.liex.ru - 37 PageSpeed Score
www.internetbookshop.it - 37 PageSpeed Score
www.baodatviet.vn - 37 PageSpeed Score
www.correios.com.br - 38 PageSpeed Score
www.sbisec.co.jp - 38 PageSpeed Score
www.tnt.com - 38 PageSpeed Score
www.bloggang.com - 38 PageSpeed Score
www.ox.ac.uk - 38 PageSpeed Score
www.live365.com - 38 PageSpeed Score
www.planalto.gov.br - 38 PageSpeed Score
www.valuecommerce.com - 38 PageSpeed Score
www.kingeshop.com - 38 PageSpeed Score
www.postgresql.org - 38 PageSpeed Score
www.registro.br - 39 PageSpeed Score
www.qq.cc - 39 PageSpeed Score
www.deutsche-bank.de - 39 PageSpeed Score
www.tre.it - 39 PageSpeed Score
www.hsbc.com.mx - 39 PageSpeed Score
www.warnerbros.com - 39 PageSpeed Score
www.mojang.com - 39 PageSpeed Score
www.virtualedge.com - 39 PageSpeed Score
www.setlinks.ru - 39 PageSpeed Score
www.mca.gov.in - 39 PageSpeed Score
www.tinychat.com - 39 PageSpeed Score
www.torrenty.org - 39 PageSpeed Score
www.s5themes.com - 39 PageSpeed Score

How about websites with a really low speed index?

SELECT url,SpeedIndex FROM pages WHERE rank<1000 AND SpeedIndex<1000 ORDER BY pages.SpeedIndex ASC

Popular Websites with a SpeedIndex under 1000

www.duckduckgo.com - 513 SpeedIndex Score
www.clkmon.com - 516 SpeedIndex Score
www.indeed.com - 588 SpeedIndex Score
www.google.cn - 600 SpeedIndex Score
www.wordreference.com - 617 SpeedIndex Score
www.search-results.com - 648 SpeedIndex Score
www.youtube-mp3.org - 659 SpeedIndex Score
www.backpage.com - 675 SpeedIndex Score
www.ezpowerads.com - 700 SpeedIndex Score
www.dmoz.org - 710 SpeedIndex Score
www.baidu.com - 711 SpeedIndex Score
www.zeobit.com - 754 SpeedIndex Score
www.blogfa.com - 786 SpeedIndex Score
www.chinabroadcast.cn - 790 SpeedIndex Score
www.optmd.com - 800 SpeedIndex Score
www.kayak.com - 819 SpeedIndex Score
www.intuit.com - 824 SpeedIndex Score
www.pixlr.com - 861 SpeedIndex Score
www.free-tv-video-online.me - 880 SpeedIndex Score
www.grooveshark.com - 886 SpeedIndex Score
www.gtmetrix.com - 892 SpeedIndex Score
www.torrentz.eu - 900 SpeedIndex Score
www.mpnrs.com - 900 SpeedIndex Score
www.4chan.org - 900 SpeedIndex Score
www.keepvid.com - 910 SpeedIndex Score
www.ups.com - 919 SpeedIndex Score
www.zoho.com - 926 SpeedIndex Score
www.google.kz - 932 SpeedIndex Score
www.examiner.com - 940 SpeedIndex Score
www.nairaland.com - 945 SpeedIndex Score
www.4dsply.com - 960 SpeedIndex Score
www.mercadolibre.com.mx - 993 SpeedIndex Score

What other website trends interest you?