We’ve been using a tool called ‘crawl visualisations’ to reveal some significant issues with the structure of our website.\u00a0<\/span><\/p>\n Once a user is on our website, it’s very easy for them to get lost or to arrive at content dead-ends. <\/span>This leads to a poor user experience, meaning they’re less likely to return to our website in future.\u00a0<\/span><\/p>\n Search engines (e.g. Google, Bing etc) must crawl a LOT<\/em> of URLs to understand the site. Issues such as duplicate content and sitemap errors can confuse search engines and ultimately waste our crawl budget<\/a><\/span>, meaning they will not index or rank our pages, which in turn leads to users not finding our pages in their search results. Boo!<\/span><\/p>\n We’re going to deal with this as part of our OneWeb search engine optimisation (SEO) and larger strategy work. We’ve already made a start by sharing these visualisations and information about the work as part of our <\/span>challenge session<\/span><\/a> – but the data and findings were so compelling we want to share them with you too. <\/span><\/p>\n You\u2019ve probably heard us banging the \u2018user needs first\u2019 drum over the past year. Our website is not for us, it\u2019s for the people we seek to serve as a University. So that means focusing and understanding their needs first.<\/span><\/p>\n In other words, more about them, less about us.<\/span><\/p>\n As part of the preparation for the <\/span>challenge session<\/span><\/a>, we carried out a <\/span>crawl<\/span><\/a> (Search engines crawl websites to discover content and store it in databases) of the main domain <\/span>https:\/\/www.southampton.ac.uk<\/a>, using <\/span>Screaming Frog<\/span><\/a>. We then used the output from the crawl to create <\/span>force-directed graphs<\/span><\/a>; interactive visualisations of our website\u2019s architectures.<\/span><\/p>\n The crawler encountered 323,000 urls in total. Screaming Frog shorten each crawl map to the first 10,000 urls it finds, but even then the file is too large to reproduce online. Here are two screenshots instead, one showing a force-directed crawl diagram, the other a force-directed directory tree diagram.<\/span><\/p>\n The internal linking structure of https:\/\/www.southampton.ac.uk (first 10,000 urls encountered), created using a force-directed crawl diagram from Screaming Frog.\u00a0<\/span>KEY:\u00a0<\/strong><\/span>H: Home page, VOD: Virtual open day, P: Prospectuses, SS: Student services, RGP: Regulations, guides and policies, SM: Sitemap, PGT: Postgraduate taught course pages, PGR: Postgraduate research pages, UG: Undergraduate pages<\/span><\/p>\n The directory tree visualisation of https:\/\/www.southampton.ac.uk (first 10,000 urls encountered), created using a force-directed directory tree diagram from Screaming Frog.\u00a0<\/span>KEY:\u00a0<\/span>H: Home page, MOD: Modules, PS: Programme Specs, HU: Humanities, VOD: Virtual Open Day<\/span><\/p>\n Both visualisations\u2019 start URL was the home page and show the increasing crawl depth of the site. Crawl depth is the minimum number of clicks it takes to get from the home page to the destination URL.\u00a0<\/span><\/p>\n The green nodes are pages which are indexable, meaning that search engines (e.g. Google, Bing etc). can find the page and return it in search results.\u00a0<\/span><\/p>\n The red nodes are pages which are non-indexable by search engines. They may be non-indexable because the pages:<\/span><\/p>\n Therefore, they can identify areas of concern.\u00a0<\/span><\/p>\n The grey nodes show where a page has child pages, but the visualisation doesn\u2019t show them as it has reached the 10,000 url limit.<\/span><\/p>\n The lines represent the link between one URL and another, via the shortest path.<\/span><\/p>\n Crawl visualisations show the internal linking structure of a website. Internal links help to establish hierarchy within a website and pass value and authority around the site. Effectively, this is how search engines might crawl a site and rank the content within.<\/p>\n Directory tree visualisations are useful because they show the organisation of a website and how users might navigate a site.<\/p>\n Either way, they provide scale and perspective and can reveal underlying issues that are otherwise difficult to detect.<\/p>\n Our website is massive. These diagrams represent just 3.1% of the southampton.ac.uk domain.\u00a0<\/span>The crawl diagram shows that our website is very segmented, with undergraduate and postgraduate course pages operating as distinct and separate websites in themselves.\u00a0<\/span><\/p>\n Duplicate content and orphan pages (i.e. pages that aren\u2019t linked to any other page) are likely to be issues and the sitemap template seems to be throwing errors. Other potential problem areas include the regulations, guides and policies, prospectuses, and student services sections.<\/span><\/p>\n The directory tree diagram shows how deep our website is (10+ levels) and that the \u2018modules\u2019 section is orphaned \u2013 representing a missed opportunity to acquire and engage prospective students through organic search (Google, for instance, tends not to index or rank orphan pages). There are also possible issues here with programme-specification pages – and more worryingly – some level 2 pages around the home page. This is a key area which OneWeb will need to address.<\/span><\/p>\n Creating these diagrams has enabled us to visualise the website, albeit on a reduced scale, and produce a list of valuable action points to take forward into OneWeb.\u00a0<\/span><\/p>\nFocus on your users<\/h3>\n
![\"\"](\"https:\/\/www.southampton.ac.uk\/blog\/wp-content\/uploads\/sites\/27\/2019\/02\/force-directed-diagram-right-click.gif\")
\nAn example of an interactive force-directed crawl diagram from Screaming Frog. Credit: Screaming Frog<\/a><\/span><\/p>\n![\"\"](\"https:\/\/www.southampton.ac.uk\/blog\/wp-content\/uploads\/sites\/27\/2019\/02\/grey-nodes.gif\")
An example of an interactive force-directed crawl diagram from Screaming Frog. Right click on a node to focus here. Credit: Screaming Frog<\/a><\/span><\/p>\nMapping the data<\/h3>\n
Crawl visualisation<\/h4>\n
![\"Force-directed](\"https:\/\/www.southampton.ac.uk\/blog\/wp-content\/uploads\/sites\/27\/2019\/02\/crawl-vis.png\")
<\/p>\nDirectory tree visualisation<\/h4>\n
![\"\"](\"https:\/\/www.southampton.ac.uk\/blog\/wp-content\/uploads\/sites\/27\/2019\/02\/annotated-force-directed-site-architecture-visualisation-fullcrawl.260918.jpg\")
<\/p>\nHow to read a crawl visualisation<\/h3>\n
\n
Why visualisations are useful<\/h3>\n
What it all means<\/h3>\n
Next steps<\/h3>\n