The most important part of on-page SEO allows search engines to crawl and index your pages. On-page SEO is all about increasing your discoverability in search engines. You can’t do this without allowing these search engines to access your pages. This is where crawling, indexing, and ranking comes into play. Let’s take a look at how this process works and learn how to enable it on your webpages.
Search engines have 3 primary functions.
Crawl, or search webpages and look over the code and content for each URL. Index, or store and organize the information it finds from crawling. And rank, which allows search engines to provide the pieces of content that best match a person’s search query.
And how does Google find your content?
Basically, Google uses a huge set of computers to crawl billions of pages on the web. This crawler essentially begins with a list of webpage URLs generated from previous crawls and then augments those pages with the sitemap data that’s provided.
During the crawling process, the Googlebot looks for new sites, updates to existing pages, and any broken links. Once the crawling process is complete, all of the results are fed into Google’s index, and any new sites or updated content will be listed accordingly. While it processes the results, Google looks at information on your page such as title tags, meta description, alt tags, and more. Think of this process like a library. Books need to be identified, categorized, and placed in the right order on shelves. Crawlers are performing similar tasks as a librarian. They are organizing all of the contents on the internet into ways that make them easily accessible.
Can your pages already be found by Google?
The best place to start is to understand which pages on your website are already being indexed. If your page is being indexed, that means that Google is able to crawl your page and store its contents. The best way to do this is by using an advanced search operator. An advanced search operator sounds complicated, but it’s actually pretty simple. In the search engine, type “site:yourdomain.com” replacing the domain with your own. Example of an advanced search operator. So if we’re doing this for Nexorank, we’d type “site:https://nexorank.com” into Google. We see that Nexorank has 408 pages currently being indexed by Google. This isn’t going to be an exact number, but it will at least give you an idea of how many of your own pages are currently being indexed. If no content is indexed yet for a site, Google will let you know that your search did not match any results. If no content is found, your next step should be to create a sitemap that you can submit to Google.
A sitemap is a file of code that lives on your web server and lists all of the relevant URLs your website is carrying. Your sitemap helps search engine web crawlers understand how your website is built so they can evaluate and rank it more easily. XML sitemaps are designed specifically for search engines like Google. Google needs to be able to find webpages anchored within a website no matter how old or deeply nested they might be in that website’s domain. For this reason, an XML sitemap is a crucial component of a blog, where article pages are constantly bumped further back into the website’s archive as new content is published. Here’s an example of what a sitemap file might look like.
An XML sitemap includes 4 key elements. The URL location is the full URL to the webpage. The last modified date is when the page was last updated. Change frequency can be set to: Never, Yearly, Monthly, Weekly, Daily, Hourly, or Always. And page priority is a number between zero and one which indicates the relative priority of pages within your website.
To create a sitemap, you’ll need to work with multiple teams. Depending on the size of your organization, there may be many people involved in creating a sitemap. If you’re a small company or a team of one, keep what I’m about to say in mind too. It will give you context on what to think about when setting up your own sitemap. Whether you are a business of one or one thousand, there are a few perspectives you’ll want to keep in mind when setting up a sitemap: Marketing, Development, IT, and Legal. Let’s start with the marketing team’s role in creating your sitemap. Marketing is usually responsible for defining the structure of the site.
Which pages need to link to one another?
Should site visitors be able to get from the “About Us” page directly to the product page, for example? Whether they define the structure using a visual site map in PowerPoint or some other tool is up to them. Then you need to consider your developer. A developer will then build an XML sitemap file based on the website’s structure. Your IT team or agency usually gets involved, too, assuming you’re using your own servers or some servers that IT controls. The sitemap lives on a server — an IT expert will be able to get it there. Lastly, be sure
you check with your legal team. Legal will make sure the site doesn’t have outstanding copyright restrictions. Then you’ll want to define the navigation structure. The structure of your website plays a big role in your site’s SEO, so it’s important to plan your top-level navigation structure carefully. Pay attention to your website’s depth. The further away a page is from the original homepage URL of your site, the worse it is for that page’s SEO. What are best practices for top-level navigation, you might be asking? Unfortunately, it’s hard to give general advice here as best practices can vary significantly by industry and company type. But, you can take inspiration from other successful businesses and apply what works for them to your site.
For inspiration from companies similar to yours, then take a look at Crayon. Crayon offers competitive intelligence software. It has a huge library of guides and case studies that are easily filterable by industry. This keeps all of their resources close to their top-level homepage while also allowing pages to be easily navigated throughout their site. Check out high-traffic homepages in your industry to get a sense of their structure and get some ideas for yours. Once you’ve defined your homepage’s navigation structure, you’ll want to brainstorm and map out the pages that are two or three levels deeper into your website. This might be your “About Us” page, your team management page, your hiring page, your blog, and so on. Depending on the complexity of your website, you may only need two levels, or you may need up to four. And as you think out the deeper parts of your website, you may find you need to tweak the top-level navigation — that’s okay.
Now you’re ready to write the XML sitemap and submit it to search engines.
Once you’ve planned out how your website will be organized, you’ll create the XML sitemap, put it on your web server, and submit it to each individual search engine. Again, doing these steps depends on your technical ability and size of your organization. You can either do this yourself or work with a developer or agency. So why submit your sitemap. When you first publish your website, and each time you go through a significant site redesign, someone on your team will have to submit the sitemap to each search engine so that they can tell your homepage from your “About Us” page or your team management page. Unless you’re using a platform that automatically updates your sitemap, you’ll need to update the sitemap yourself and resubmit any time you make significant changes to your website. Otherwise, you’ll have to wait for search engines to identify, crawl, and index your new pages itself. That means if you’ve made a really significant change to your site’s structure, you might want to manually submit it.
Using the Nexorank CMS Hub makes this easier. Nexorank will automatically generate your XML sitemap file when you publish new pages and make changes to your site. Those of you using another CMS might be looking for an easy solution to create your XML sitemap.
There are a few free solutions, including Screaming Frog and XML-Sitemaps.com. Screaming Frog is a web crawler that allows you to assess your website’s on-page SEO. The company offers a tool to develop your own XML sitemap and strengthen your website’s on-page SEO in the process. To use Screaming Frog’s sitemap generator, you’ll first download the company’s SEO web crawler, which is free for crawling your first 500 URLs. Once your website is crawled, you can create an XML sitemap from it including every webpage that scores a “200” in the initial crawl. This ensures only your strongest pages are included in your new XML sitemap. The tool XMLSitemaps.com requires no registration or initial download to create a sitemap for your website. This tool is free to websites that carry up to 500 URLs. Once your sitemap has been created, you can either download it as an XML file or receive it via email if you need to transport it to a new computer or coworker. The paid version of XMLSitemaps.com allows you to crawl up to 1.5 million pages and create other forms of XML sitemaps — such as news, video, and image sitemaps — and submit your sitemap directly to a search engine from the sitemap generator.
First, sign into Google Search Console.
Search Console will be your dashboard for testing and submitting updated sitemaps to Google, so you can ensure Google is always aware of the latest pages published to your site. To get started, add in your domain into the blank field. This website should just be the domain name — the parent URL to which all of your other webpages belong. Then, click “continue.” Once your ownership is verified, click “go to property.” If your ownership isn’t verified, double check that you are logged into your correct Google account.
Now that you have your domain added to your Search Console account, the next step will be to add your sitemap. In the left-hand bar, under “index” click “sitemaps.” To submit a new sitemap, you’ll simply need to type in the URL where your sitemap lives. Adding a new sitemap requires you to add a string of text to the end of your website’s domain. Think of it like a tracking tag, allowing Google to examine all the activity that takes place inside your website. This sitemap tag is “sitemap.xml,” and you’ll want to add it to the end of your domain name. For example, if your domain is www.yourcompany.com, you’ll tag it like this: www.yourcompany.com/sitemap.xml.
Submit your sitemap and you’re all set.
Depending on how much page authority you have already accumulated on Google, it might take some time to see the status of your submitted sitemap. Give it time — Google will eventually accept it. Now let’s walkthrough how to submit a sitemap to Bing or Yahoo. Sign in to Bing Webmaster Tools. On the My Sites page, enter the URL for your company. Click “Add.” In the “Add a sitemap” field, enter http://www.yourdomain.com/sitemap.xml. Replace “yourdomain” with your company’s URL. Complete the rest of the required fields on the page, and click “Save.” Then you’re all set. Once you define and submit your sitemap the first time, chances are you’ll want to tweak and add pages to your website — and that’s completely fine. But keep in mind that if your website isn’t built on a platform that automatically generates a new sitemap and updates it on your web server when new pages are added, then every time you add a page — any page — to your website, that page will be missing from the sitemap that search engines see. Google crawls pages frequently, which means that you could get by without resubmitting your sitemap manually. Although the problem with this approach is that you’re reliant on Google’s timeframe to crawl and index your site content. Google might not crawl and index your new page as quickly as you’d like. To index your content the fastest way possible, you should resubmit your sitemap after publishing a new page. Having and maintaining your sitemap is crucial to being discoverable in search engines. Your sitemap will help search engines understand your content and get it served to the people you are trying to reach.
