Internet and Website Trends with HTTParchive
In my last web performance tools article I have mentioned Ilya Grigorik’s Bigqueries forum where people can share and discuss interesting queries on the HTTParchive. These queries can give us some insight into internet trends and the state of the web. Today, I wanted to share two of my favorite queries from that forum and follow up with some of my own queries to identify some interesting website trends. I have decided to execute my queries in MySQL with the latest imported HTTParchive dump. This is mostly because I am constantly running into limits with using the HTTPArchive database on Google BigQuery.
Content Delivery Networks on Top URLs
This query analyzes CDN values in HTTP Archive urls and I am mainly interested in websites that are in the Alexa top 1000. Here are the results (keep in mind that Google is up there mostly due to blogs on Blogger).
- No CDN solutions exists on 71.3% of sites
- Akamai is implemented on 14.7% of sites
- Google is implemented on 8.3% of sites
- Lxdns.com is implemented on 1.7% of sites
- CloudFlare is implemented on 1.4% of sites
- Edgecast is implemented on .5% of sites
- ChinaCache is implemented on .4% of sites
- Fastly is implemented on .4% of sites
- Level 3 is implemented on .4% of sites
- CDNetworks is implemented on .3% of sites
- LimeLight is implemented on .2% of sites
- CubeCDN is implemented on .1% of sites
- Amazon CloudFront is implemented on .1% of sites
Akamai clearly owns the CDN market but I really find it hard to believe that 71.3% of websites in the Alexa top 1K are not using any content delivery network.
Top Third-Party Domains
This query attempts to identify domains in the HTTParchive that are the top 3rd party dependencies while also providing the number of requests. Here are the top 40 results:
- google.com – 123182 requests
- fbcdn.net – 107494 requests
- doubleclick.net – 71125 requests
- google-analytics.com – 63812 requests
- facebook.com – 60707 requests
- googlesyndication.com – 55020 requests
- twitter.com – 52833 requests
- googleusercontent.com – 48093 requests
- addthis.com – 44251 requests
- googleapis.com – 37042 requests
- sharethis.com – 32742 requests
- gstatic.com – 32670 requests
- blogspot.com – 32063 requests
- ytimg.com – 31971 requests
- scorecardresearch.com – 22478 requests
- adnxs.com – 16942 requests
- cloudfront.net – 16159 requests
- amazonaws.com – 16056 requests
- youtube.com – 15538 requests
- gravatar.com – 12461 requests
- 2mdn.net – 11970 requests
- twimg.com – 10927 requests
- mathtag.com – 10412 requests
- quantserve.com – 10036 requests
- facebook.net – 9565 requests
- betrad.com – 9354 requests
- wp.com – 9339 requests
- truste.com – 9095 requests
- akamaihd.net – 8476 requests
- blogger.com – 8437 requests
- baidu.com – 8147 requests
- lijit.com – 7668 requests
- pinterest.com – 7437 requests
- yandex.ru – 7195 requests
- demdex.net – 6996 requests
- turn.com – 6871 requests
- adsafeprotected.com – 6726 requests
- yieldmanager.com – 6669 requests
- invitemedia.com – 6231 requests
- bluekai.com – 6040 requests
If you consider all the Google properties (google.com, doubleclick.net, google-analytics.com, googlesyndication.com, googleusercontent.com, googleapis.com, gstatic.com, blogspot.com, ytimg.com, youtube.com, 2mdn.net, blogger.com) it is pretty amazing how much of the web leverages Google. It’s not surprising to see Facebook (fbcdn.net, facebook.com, acebook.net), Twitter (twitter.com, twimg.com), and link sharing websites (addthis.com, gravatar.com) on top of this list as well. Amazon (cloudfront.net, amazonaws.com) and advertising companies (adnxs.com, mathtag.com, lijit.com) also make sense but I was a bit surprised to see scorecardresearch.com and turn.com in the top 40 list. Where you surprised by anything on this list?
There are many more awesome queries you can run on the HTTParchive and I decided to import the data into my local version of myQSL and play around with sorting some of the columns. I don’t really like the results from just a complete column sort as you get many unpopular websites so I often filter by Alexa rankings to segment the results. For example, I wanted to see what web pages have the highest amount of errors, but I only care about popular domains, so I limited the search to Alexa top 1000:
SELECT url,numErrors FROM pages WHERE rank<1000 ORDER BY pages.numErrors DESC
Popular Websites with more than 2 errors on page
- www.imageshack.us - 11 errors
- www.enet.com.cn - 10 errors
- www.hubspot.com - 10 errors
- www.techcrunch.com - 8 errors
- www.p5w.net - 7 errors
- www.chron.com - 6 errors
- www.elpais.com - 6 errors
- www.independent.co.uk - 5 errors
- www.peyvandha.ir - 5 errors
- www.directrev.com - 3 errors
- www.qtrax.com - 3 errors
- www.samsung.com - 3 errors
- www.intoday.in - 3 errors
- www.ci123.com - 3 errors
- www.yellowpages.com - 3 errors
- www.tutsplus.com - 3 errors
- www.sulekha.com - 3 errors
- www.today.com - 3 errors
- www.r7.com - 3 errors
- www.panet.co.il - 3 errors
How about popular websites that have a really bad pagespeed score? First I looked at the Alexa top 1K:
SELECT url,PageSpeed FROM pages WHERE rank<1000 AND PageSpeed<70 ORDER BY pages.PageSpeed ASC
Alexa 1K websites with PageSpeed score under 70
- www.soku.com - 16 PageSpeed Score
- www.zippyshare.com - 24 PageSpeed Score
- www.wunderground.com - 29 PageSpeed Score
- www.jvzoo.com - 37 PageSpeed Score
- www.correios.com.br - 38 PageSpeed Score
- www.zazzle.com - 40 PageSpeed Score
- www.ccb.com - 41 PageSpeed Score
- www.worldstarhiphop.com - 41 PageSpeed Score
- www.workercn.cn - 45 PageSpeed Score
- www.onlinesbi.com - 45 PageSpeed Score
- www.icbc.com.cn - 45 PageSpeed Score
- www.instagram.com - 47 PageSpeed Score
- www.directrev.com - 47 PageSpeed Score
- www.theverge.com - 47 PageSpeed Score
- www.blogfa.com - 48 PageSpeed Score
- www.wmmail.ru - 49 PageSpeed Score
- www.hdfcbank.com - 50 PageSpeed Score
- www.sakura.ne.jp - 50 PageSpeed Score
- www.acesse.com - 50 PageSpeed Score
- www.nih.gov - 51 PageSpeed Score
- www.caixa.gov.br - 51 PageSpeed Score
- www.mayoclinic.com - 51 PageSpeed Score
- www.naver.jp - 52 PageSpeed Score
- www.ocn.ne.jp - 53 PageSpeed Score
- www.ctrip.com - 54 PageSpeed Score
- www.p5w.net - 55 PageSpeed Score
- www.gc.ca - 55 PageSpeed Score
- www.bitshare.com - 55 PageSpeed Score
- www.novinky.cz - 55 PageSpeed Score
- www.mixi.jp - 55 PageSpeed Score
- www.xbox.com - 56 PageSpeed Score
- www.adf.ly - 57 PageSpeed Score
- www.hatena.ne.jp - 57 PageSpeed Score
- www.idealo.de - 57 PageSpeed Score
- www.persianblog.ir - 58 PageSpeed Score
- www.kdnet.net - 58 PageSpeed Score
- www.17ok.com - 58 PageSpeed Score
- www.webmoney.ru - 59 PageSpeed Score
- www.aljazeera.net - 59 PageSpeed Score
- www.prweb.com - 59 PageSpeed Score
- www.siteadvisor.com - 60 PageSpeed Score
- www.yxlady.com - 60 PageSpeed Score
- www.pr-cy.ru - 60 PageSpeed Score
- www.flippa.com - 60 PageSpeed Score
- www.optmd.com - 61 PageSpeed Score
- www.sape.ru - 61 PageSpeed Score
- www.rbc.ru - 61 PageSpeed Score
- www.adobe.com - 62 PageSpeed Score
- www.cloob.com - 62 PageSpeed Score
- www.cocolog-nifty.com - 62 PageSpeed Score
- www.qtrax.com - 63 PageSpeed Score
- www.wellsfargo.com - 63 PageSpeed Score
- www.clicksor.com - 63 PageSpeed Score
- www.oracle.com - 63 PageSpeed Score
- www.adclickxpress.com - 63 PageSpeed Score
- www.xhamstercams.com - 63 PageSpeed Score
- www.free.fr - 64 PageSpeed Score
- www.ig.com.br - 64 PageSpeed Score
- www.m2newmedia.com - 64 PageSpeed Score
- www.ad4game.com - 64 PageSpeed Score
- www.adnxs.com - 64 PageSpeed Score
- www.icicibank.com - 65 PageSpeed Score
- www.leo.org - 65 PageSpeed Score
- www.glispa.com - 65 PageSpeed Score
- www.pornhublive.com - 65 PageSpeed Score
- www.xyxy.net - 66 PageSpeed Score
- www.ucoz.ru - 66 PageSpeed Score
- www.cj.com - 66 PageSpeed Score
- www.webcrawler.com - 66 PageSpeed Score
- www.taleo.net - 66 PageSpeed Score
- www.itau.com.br - 66 PageSpeed Score
- www.ca.gov - 66 PageSpeed Score
- www.homeway.com.cn - 66 PageSpeed Score
- www.rednet.cn - 66 PageSpeed Score
- www.wmtransfer.com - 66 PageSpeed Score
- www.wikipedia.org - 67 PageSpeed Score
- www.china.com.cn - 67 PageSpeed Score
- www.goo.ne.jp - 67 PageSpeed Score
- www.jqw.com - 67 PageSpeed Score
- www.linkbucks.com - 67 PageSpeed Score
- www.lifehacker.com - 67 PageSpeed Score
- www.bloglovin.com - 67 PageSpeed Score
- www.evernote.com - 67 PageSpeed Score
- www.excite.co.jp - 67 PageSpeed Score
- www.w3.org - 68 PageSpeed Score
- www.java.com - 68 PageSpeed Score
- www.t-online.de - 68 PageSpeed Score
- www.wiktionary.org - 68 PageSpeed Score
- www.bet365.com - 69 PageSpeed Score
- www.chaturbate.com - 69 PageSpeed Score
- www.letitbit.net - 69 PageSpeed Score
- www.eonline.com - 69 PageSpeed Score
- www.yomiuri.co.jp - 69 PageSpeed Score
- www.makeuseof.com - 69 PageSpeed Score
- www.ucoz.com - 69 PageSpeed Score
And then I looked at Alexa 10K websites with a score under 50:
SELECT url,PageSpeed FROM pages WHERE rank<10000 AND PageSpeed<60 ORDER BY pages.PageSpeed ASC
Alexa 10K websites with PageSpeed score under 40
- www.rui.jp - 9 PageSpeed Score
- www.zocalo.com.mx - 10 PageSpeed Score
- www.soku.com - 16 PageSpeed Score
- www.firstrowus1.eu - 19 PageSpeed Score
- www.firstrownow.eu - 20 PageSpeed Score
- www.teamtreehouse.com - 20 PageSpeed Score
- www.rs-online.com - 22 PageSpeed Score
- www.sulia.com - 23 PageSpeed Score
- www.avenues.info - 23 PageSpeed Score
- www.begun.ru - 23 PageSpeed Score
- www.zippyshare.com - 24 PageSpeed Score
- www.dnaindia.com - 25 PageSpeed Score
- www.fahrinfo-berlin.de - 25 PageSpeed Score
- www.hotfile.com - 26 PageSpeed Score
- www.jsfiddle.net - 26 PageSpeed Score
- www.bancopopular.es - 26 PageSpeed Score
- www.bakecaincontrii.com - 27 PageSpeed Score
- www.flvrunner.com - 27 PageSpeed Score
- www.1337x.org - 28 PageSpeed Score
- www.dedecms.com - 28 PageSpeed Score
- www.anoox.com - 28 PageSpeed Score
- www.swedbank.lt - 28 PageSpeed Score
- www.wunderground.com - 29 PageSpeed Score
- www.the9gag.com - 29 PageSpeed Score
- www.bnpparibas.net - 30 PageSpeed Score
- www.inps.it - 30 PageSpeed Score
- www.odn.ne.jp - 30 PageSpeed Score
- www.mydocomo.com - 30 PageSpeed Score
- www.bidvertiser.com - 31 PageSpeed Score
- www.presstv.ir - 31 PageSpeed Score
- www.mktmobi.com - 31 PageSpeed Score
- www.gsis.gr - 31 PageSpeed Score
- www.steamcommunity.com - 32 PageSpeed Score
- www.subscribe.ru - 32 PageSpeed Score
- www.ana.co.jp - 32 PageSpeed Score
- www.4738.com - 32 PageSpeed Score
- www.watch32.com - 32 PageSpeed Score
- www.internetdownloadmanager.com - 33 PageSpeed Score
- www.societegenerale.fr - 33 PageSpeed Score
- www.service-public.fr - 33 PageSpeed Score
- www.hackforums.net - 33 PageSpeed Score
- www.serpbook.com - 33 PageSpeed Score
- www.oxfordjournals.org - 33 PageSpeed Score
- www.sxc.hu - 34 PageSpeed Score
- www.softgozar.com - 34 PageSpeed Score
- www.2checkout.com - 34 PageSpeed Score
- www.cafef.vn - 34 PageSpeed Score
- www.game2.cn - 34 PageSpeed Score
- www.cic.fr - 34 PageSpeed Score
- www.rqzao.com - 34 PageSpeed Score
- www.girlfriendvideos.com - 34 PageSpeed Score
- www.xdowns.com - 34 PageSpeed Score
- www.freakshare.com - 35 PageSpeed Score
- www.dardarkom.com - 35 PageSpeed Score
- www.ems.com.cn - 35 PageSpeed Score
- www.cwb.gov.tw - 35 PageSpeed Score
- www.trk4.com - 35 PageSpeed Score
- www.rahnama.com - 35 PageSpeed Score
- www.bancoestado.cl - 35 PageSpeed Score
- www.pc6.com - 35 PageSpeed Score
- www.sportschau.de - 35 PageSpeed Score
- www.unibytes.com - 35 PageSpeed Score
- www.womai.com - 35 PageSpeed Score
- www.pornhost.com - 35 PageSpeed Score
- www.enstage.com - 35 PageSpeed Score
- www.xtrasize.it - 35 PageSpeed Score
- www.adv-adserver.com - 36 PageSpeed Score
- www.thaiseoboard.com - 36 PageSpeed Score
- www.ingdirect.es - 36 PageSpeed Score
- www.centos.org - 36 PageSpeed Score
- www.thesitewizard.com - 36 PageSpeed Score
- www.canon.jp - 36 PageSpeed Score
- www.indiapost.gov.in - 36 PageSpeed Score
- www.khmer-news.org - 36 PageSpeed Score
- www.jvzoo.com - 37 PageSpeed Score
- www.bluedart.com - 37 PageSpeed Score
- www.rbi.org.in - 37 PageSpeed Score
- www.liex.ru - 37 PageSpeed Score
- www.internetbookshop.it - 37 PageSpeed Score
- www.baodatviet.vn - 37 PageSpeed Score
- www.correios.com.br - 38 PageSpeed Score
- www.sbisec.co.jp - 38 PageSpeed Score
- www.tnt.com - 38 PageSpeed Score
- www.bloggang.com - 38 PageSpeed Score
- www.ox.ac.uk - 38 PageSpeed Score
- www.live365.com - 38 PageSpeed Score
- www.planalto.gov.br - 38 PageSpeed Score
- www.valuecommerce.com - 38 PageSpeed Score
- www.kingeshop.com - 38 PageSpeed Score
- www.postgresql.org - 38 PageSpeed Score
- www.registro.br - 39 PageSpeed Score
- www.qq.cc - 39 PageSpeed Score
- www.deutsche-bank.de - 39 PageSpeed Score
- www.tre.it - 39 PageSpeed Score
- www.hsbc.com.mx - 39 PageSpeed Score
- www.warnerbros.com - 39 PageSpeed Score
- www.mojang.com - 39 PageSpeed Score
- www.virtualedge.com - 39 PageSpeed Score
- www.setlinks.ru - 39 PageSpeed Score
- www.mca.gov.in - 39 PageSpeed Score
- www.tinychat.com - 39 PageSpeed Score
- www.torrenty.org - 39 PageSpeed Score
- www.s5themes.com - 39 PageSpeed Score
How about websites with a really low speed index?
SELECT url,SpeedIndex FROM pages WHERE rank<1000 AND SpeedIndex<1000 ORDER BY pages.SpeedIndex ASC
Popular Websites with a SpeedIndex under 1000
- www.duckduckgo.com - 513 SpeedIndex Score
- www.clkmon.com - 516 SpeedIndex Score
- www.indeed.com - 588 SpeedIndex Score
- www.google.cn - 600 SpeedIndex Score
- www.wordreference.com - 617 SpeedIndex Score
- www.search-results.com - 648 SpeedIndex Score
- www.youtube-mp3.org - 659 SpeedIndex Score
- www.backpage.com - 675 SpeedIndex Score
- www.ezpowerads.com - 700 SpeedIndex Score
- www.dmoz.org - 710 SpeedIndex Score
- www.baidu.com - 711 SpeedIndex Score
- www.zeobit.com - 754 SpeedIndex Score
- www.blogfa.com - 786 SpeedIndex Score
- www.chinabroadcast.cn - 790 SpeedIndex Score
- www.optmd.com - 800 SpeedIndex Score
- www.kayak.com - 819 SpeedIndex Score
- www.intuit.com - 824 SpeedIndex Score
- www.pixlr.com - 861 SpeedIndex Score
- www.free-tv-video-online.me - 880 SpeedIndex Score
- www.grooveshark.com - 886 SpeedIndex Score
- www.gtmetrix.com - 892 SpeedIndex Score
- www.torrentz.eu - 900 SpeedIndex Score
- www.mpnrs.com - 900 SpeedIndex Score
- www.4chan.org - 900 SpeedIndex Score
- www.keepvid.com - 910 SpeedIndex Score
- www.ups.com - 919 SpeedIndex Score
- www.zoho.com - 926 SpeedIndex Score
- www.google.kz - 932 SpeedIndex Score
- www.examiner.com - 940 SpeedIndex Score
- www.nairaland.com - 945 SpeedIndex Score
- www.4dsply.com - 960 SpeedIndex Score
- www.mercadolibre.com.mx - 993 SpeedIndex Score
What other website trends interest you?