Brian Harnish 00:00
Hello and welcome to Ask An SEO Episode 28: Advanced SEO Techniques That Work In 2022. Are you looking to stay ahead of the curve with your SEO strategy? Listen up, because you’ll need to know about the latest and greatest techniques that are sure to dominate the coming year. In this video, I’ll share with you some advanced SEO techniques that are guaranteed to help you rank higher in search engine results pages in 2022.

First, you want to keep your content fresh and relevant.

One of the most important things you can do to improve your SEO is to keep your website content fresh and relevant. This means regularly publishing new and original content that provides value to your audience. Not only will it help you improve your search engine ranking, but it will also keep people coming back to your site for more.

Next, you want to implement Schema Markup.

Schema Markup is a way to show Google exactly what your website is about. This will help them understand your website better and rank you higher in search engine result pages. How does schema help? By default, Google does not understand certain types of data by itself, instead, it must be marked up properly using a vocabulary such as that found on schema.org.

Leveraging and implementing the Schema Markup can help Google understand important data such as your business location, your NAP information on local. NAP stands for name, address and phone number, and information about when your event will take place, and much more.

Next is expert roundups.

Expert roundups are a great way to get links and insights from a variety of experts on a given topic. When done right, expert roundups can provide you with a wealth of information on a topic from people who know what they’re talking about. The benefits of expert roundups, besides links, are they can help give you visibility to people who may not otherwise see you as an expert. You have others in the industry backing you up on your area of expertise, and you have a method of showing people exactly how much you know.

In addition, this helps spread your influence and helps build brand new links back to your website.

Next, PR link building. PR link building is one of the most effective and under utilized techniques for getting links to your website. This involves reaching out to journalists and other members of the media who may be interested in writing about your company or your product. By pitching a story that is relevant to the audience,

you can often get links from high quality websites without having to resort to any shady tactics. In fact, PR link building can be such an effective method for getting links that many people consider to be a form of blackhat seo. However, this is not the case.

In fact, Google’s John Mueller stated that positive PR promotion is how link building is supposed to be done, and appears perfectly natural in Google’s eyes.

Use search analysis to uncover what your competition is writing about. If you want to stay ahead of the competition, you need to know what they’re doing. And one of the best ways to do that is by using search analysis to uncover what they’re writing about. By taking a look at your competitors’ website and seeing what topics they’re targeting in the content, you can get an idea of where their strengths and weaknesses lie. This information can then be used to help you shape your own content strategy accordingly.

So how do you go about performing search analysis on your competition? It’s actually quite simple. All you need is a tool called AHREFs. AHREFs is a powerful tool that allows you to see what your competition is up to. With it, you can see what content they’re targeting, what keywords they’re targeting, their ranking for, and much more.

In addition, you can also perform a content gap analysis, which will help you figure out exactly where your site has content, versus what your competitors don’t have and vice versa.

Next, you want to identify and boost pages that aren’t doing well. By taking a look at your website’s Analytics, you can see which pages are getting the most traffic and which ones are falling behind. Once you’ve identified these pages, you can then take steps to improve them. This may involve adding more relevant content, optimizing your page title and meta tags, or making other changes to ensure that they’re more visible to search engines. Boosting those pages can help increase your website’s overall traffic and improve your search engine Ranking.

Take advantage of video on YouTube and TikTok to really engage your users. By using engaging videos on platforms such as YouTube and TikTok, it’s possible to attract more attention to your brand and increase conversions.

First, let’s take a look at YouTube. YouTube is the world’s second largest search engine. It’s important to make sure that your videos are optimized for the platform. To do this, you need to ensure that your videos are interesting and informative. You also need to make sure that they’re properly tagged and titled so that they can easily be found by users who are looking for them.

In addition to YouTube, TikTok is another great platform for video content. TikTok allows you to create short, engaging videos that can be shared with other users on the platform. TikTok videos have increased to a limit of 10 minutes, but it’s important to make sure that your videos are more concise and to the point.

You also need to use hashtags and other strategies to ensure that your videos are visible to users who are searching for them. By using video on these platforms, you can really engage your users and increase your chances of getting links and conversion as a result.

You can use on page optimization tools like Surfer SEO and Frase.

On page optimization is one of the most important aspects of SEO. By optimizing your website content and structure, you can make it easy for your project engine to understand and rank your pages. This can help you improve your website’s visibility and help increase traffic to your site. There are a number of different on page optimization tools that you can use to help you with this process.

Two of the most popular tools are Surfer SEO and Frase. Surfer SEO is a tool that allows you to see how well your website is optimized for search engines. It provides information on things like how well your content is optimized, what keyword to use, what other competitors and search results are talking about and more.

Frase is the tool that allows you to find synonyms and related words for any keyword or phrase. That can be helpful when creating content for your website that allows you to use relevant keywords without overusing them. Frase also has some AI content generation built in and also allows you to create new headlines on the fly. Both of these tools are helpful in optimizing your website for search engines and improving your ranking in the SERPs.

Next, perform a comprehensive SEO audit.

Performing a comprehensive audit is a process that all businesses should go through periodically, in order to ensure that their website is performing well and meeting all of the necessary SEO requirements. A comprehensive SEO audit will examine all aspects of your website and identify any areas that need improvement. It will also provide recommendations on how to fix these issues.

In addition, the SEO audit will provide a roadmap to success so that your company did not get distracted by the minor details that should concern SEO professionals and web developers. Performing an SEO audit can help you improve your website’s ranking in the search engines, increase traffic to your site and boost your business’s overall online presence.

Next, create topical clusters.

In order to improve your website’s visibility and ranking in search engines, it’s important to create topical clusters. What are topical clusters? These are groups of related pages on a website that are all optimized for similar keywords around the same broader topic. By creating topical clusters, you can make it easier for search engines to understand your website and rank your pages accordingly. This can help you improve your website’s visibility and increase traffic to your site.

Now, there are a number of different ways to create topical clusters. One popular method is to use a content management system like WordPress. WordPress allows you to create custom taxonomies that can then be used for organizing your content. Another way to create topical clusters is by using tools like Surfer SEO or Frase. These tools have topic clustering aspects to their tools that will allow you to understand and figure out exactly what kind of topical cluster to use when you’re building out your website.

You can also use Semrush’s content topic research function in order to find appropriate topical clusters for your specific site.

Next, website architecture and siloing. One of the most important aspects of SEO is website architecture and siloing. So what exactly are these? Website architecture refers to the way in which a website is organized and structured. Siloing refers to the practice of separating your website content into specific, well defined categories (pages/links/URL structure, and so on). By organizing your website’s content in this fashion, you can make it easier for search engines to understand and rank your pages. This can help you improve your website’s visibility and increase traffic to your site.

There are a number of different ways to structure your website architecture and create silos. One popular method is to use a CMS like WordPress. WordPress allows you to create custom taxonomies that can be used for organizing your content. Now, there are two types of silos: virtual silos and physical silos. Virtual silos are silos that are created just because you link certain pages in a sequence. Physical silos where your pages are physically arranged within the silo, and linking pages are also arranged within that structure. Which is better? Both still work quite well. It all depends on how well you are able to complete the implementation.

All right, that’s it for today’s Ask an SEO Episode 28. This is Brian Harnish signing off. Please be sure to like and subscribe to our YouTube channel for a brand new episode every week.

Have a great day.

Brian Harnish
Hello, and welcome to Ask an SEO Episode 27. Today’s question is: which is better for URLs, dashes or underscores? Well, there are two popular ways to format URLs with dashes or underscores. Each has its own benefits and drawbacks, so which one you choose ultimately comes down to personal preference.

Dashes are generally considered to be more readable, both for humans and search engines. They’re also easier to type, since most keyboards have a dedicated dash key. On the downside, dashes can sometimes cause issues with server side scripts. And they’re not as commonly used as underscores, so you might run into compatibility problems down the line. Depending on what you use.

Underscores, on the other hand, are more commonly used in programming languages and are more likely to be compatible with server side scripts. They’re also less likely to cause issues with keywords, and search engines treat them as a word separator just like spaces.

However, underscores can be harder to read, both for humans and for search engines, and they’re not as easy to type on a standard keyboard.

In fact, in a recent podcast, Googlers reiterated the preference of hyphens in URLs over underscores. Google explained that they cannot easily segment an underscore and that’s why they are recommending dashes.

Alright, that’s it for today’s Ask an SEO episode 27.

This is Brian Harnish, signing off. Have a great day.

Brian Harnish 0:14
Hello, and welcome to Ask an SEO episode 26. Before we get started today, I just wanted to take a moment and say to our Ukrainian friends, allies and acquaintances that we support you. We stand in solidarity with you. And we cannot imagine everything you’re going through right now. And our hearts go out to you.

Okay, for episode 26 today the question is: what is the difference between HTTP and HTTPS? Alright, well, HTTP stands for Hypertext Transfer Protocol. This is the standard protocol that is used when any page is transferred over the world wide web. And does happen when it is transferred between the server and the client, which is you.

With HTTPS, that is the secure version of hypertext transfer protocol. And so what happened is that, by default, HTTP is not secure. Any data that is processed by the page can be seen by prying eyes, and people with nefarious intentions, such as hackers, now with a secure implementation with an SSL certificate, which is what’s required to get the HTTPS in your web browser right now.

Well, what happens is that this acts as an encryption device, on the actual page, so it will keep any data that you want hidden from prying eyes as a result. So Google also does have a relatively minor preference for HTTPS websites overall, too.

So in general, we’d always recommend using an HTTPS secure protocol whenever you’re creating a website, simply because the benefits of having one far outweigh not having it.
Alright, that’s it for today to Ask an SEO episode 26. Please be sure to like and subscribe to our YouTube channel for a brand new episode every week.

This is Brian Harnish, signing off. Have a great day!!

Brian Harnish 0:14
Hello, and welcome to Ask an SEO episode 25. Today’s question is: what is a competition analysis? Well, the competition analysis in SEO is where you are analyzing your organic competitors for the things that you can do in the search results to actually beat them.

So when you’re analyzing your competitors, you want to do things like look at the content, the length, the code, the overall website quality, and see exactly what they are doing in order to get where they are in those results, right?

So you would want to take a look at that and figure out exactly what you would want to do in order to beat them. Now, how do you beat them? That’s by becoming one of the most, I would say, it’s not necessarily about becoming the longest piece of content among your competitors, but more about the quality of your content and how it addresses the topic and how it addresses user intent in terms of that particular topic.

So that’s what you’re attempting to find out with competitor analysis: exactly what your competition is doing in order to get their site at the top of the search results.

Alright, that’s it for today’s Ask an SEO episode 25. This is Brian Harnish signing off.

Please be sure to like and subscribe to our YouTube channel for a brand new episode every week.

Have a great day.

Brian Harnish
Hello, and welcome to the first episode of Ask an SEO in 2022. Today, we’re going to talk about how to properly optimize your content pages according to guidelines within the program Surfer SEO, so let’s dive right in.

Alright, this is a blog post that we are working on optimizing with the brand-new Surfer SEO Chrome extension. So I’m going to go ahead and activate the Chrome extension here. We’re going to load Surfer. And this should be what actually shows up here. So what we are going to do is we want to create the content editor first for our particular keyword phrase, so we’re going to enter in “free keyword research tools.”

Or we can use “keyword research tools.” Let’s try that.

We’re gonna wait a bit for the actual content editor to be created.

And here we go, we’re gonna go ahead and click on this. It should physically be loaded. And this one is really already optimized for approximately 76 or so. And there’s a little bit more optimization we can do based on the actual guidelines here. So let’s kind of take a look at the guidelines. So the word count is really approximately about how you’re going to optimize the post, basically, for the word count. Headings will be how many headings you want to have in the document, and it should be approaching between 22 and 56. What it says here, for paragraphs there need to be at least 120. And images approximately 28 to 68 images, or so.

So that’s what we would be working with on in order to get the optimization of this post up to a 100 percent score in Surfer SEO. Alright, next, we want to take a look at the user intent behind the main keyword phase in the article, you know, just to make sure that they are aligned, and we aren’t shooting ourselves in the foot for anything.

So looking at keyword research tools in the web browser, we find that most of these articles here are informational, which is a good thing. So this really does pretty much align with the user intent of our article so we don’t have to change anything now.

So if we’re gonna optimize this based on our particular keyword phrases that are listed here, we want to nail down specifically mentions as well as how they’re mentioned, right? So if we take a look at this, we can see that it is currently used four times and we want to make sure that it actually goes into the nine to 17 times mark.

There are certain ways of doing this, for example. So maybe you can add a sentence or two or you can add another paragraph or whatever. This would help increase- this would help add further value in terms of word count and exactly what everything is physically covering right? So let’s take a look at this. We do want to be cautious as far as stuffing the article with these types of entities, so let’s be careful with that too. So with this particular article we could try and expand this a little bit, for example, like this.

I am just looking through the more specific keyword research tool suggestion to see where we can expand some of these opportunities.

Well that’s it for today’s episode of Ask an SEO. Please be sure to like and subscribe to our YouTube channel for a brand new episode every week. This is Brian Harnish signing off. Have a great day!

“I don’t see anything broken in the way Google indexes stuff at the moment. I do see us being critical about what we pick up for indexing though, as any search engine should.

It’s really hard to call a site authoritative after 30 articles, and especially if you’ve stopped publishing for a while, I can see how Google might be a bit more conservative with regards to indexing more. Over time, as we see that your site is more than just “30 ok posts”, and instead something we’re keen on sending as many users to as possible, then indexing will pick up. This isn’t something you can push through technical means though, it’s not the button-push before indexing that makes your site by far the best of its kind. Making your site “just as good / bad as others” is not compelling — you should really aim significantly higher, and not just with regards to the text on your pages, but in regards to everything across your site, and its embedding within the rest of the web.”

Brian (49:34)
I have a couple of questions. The first one on a Reddit post a few weeks ago, I believe you mentioned that having less than 30 pages would make Google consider the website less authoritative. Right. And I know that that leads to SEO professionals writing that you have to have a minimum of thirty articles to rank, etc. But my question concerns the other extreme: what about one page sites?

John (50:10)
I think you could make good one page sites. So from that point of view, I’m not too worried about that. I think the Reddit post, as far as I remember, was something along the lines of, I created 30 blog posts, and they’re really good. And therefore my website should be authoritative. And from my point of view, like you going off and creating 30 blog posts, does not automatically make your website authoritative. And especially for the higher, or the more critical topics. It’s something where you can’t just create 30 blog posts on a medical topic. And then say, like, I am a doctor, I’ve written 30 articles. So that that was kind of the direction I was, I was headed there.

And for a lot of websites, it’s not that you need to be seen as an authority, you, you essentially put your content out there. If you’re a small business, you’re selling something like, you don’t need to be an authority, and especially things like one page websites, they’re often very focused on this one thing. And you don’t need to be an authority to do that one thing to sell, I don’t know an ebook, or to give information about opening hours for a business. It’s like, it’s just information.

So from that point of view, having a one page website, I think it’s perfectly fine. With regards to starting out with a one page website, I think that’s fine. I would just think about, “Well, where do you want to go from there?” At some point, maybe you do want to create more pages and try to find a way that you don’t paint yourself into a corner by saying, Well, I have to put everything on one page all the time. But rather expand when you see that?

Hello and welcome to Ask an SEO Episode 21. Today’s question is what is holistic SEO? OK, so the word holistic can mean many different things, depending on who you ask about that. So in general, the term holistic refers to using a bunch of different types of disciplines related to SEO in order to achieve the desired result.

So when it comes to SEO, there are certain sub-disciplines within the field of SEO itself, right? So things like technical SEO, international SEO, mobile optimization. You can also have content writing and link building. All sorts of different fields within the SEO–within SEO itself.

So it’s really a matter of defining what you consider holistic. So when we consider holistic, let’s say that holistic means that it is using a whole bunch of different fields in order to create a better result, right?

So we would probably be using content marketing, Facebook ads, other types of paid ads, and advertising all sorts of those different fields combined within holistic SEO in order to achieve a desired result. So that’s really what holistic SEO refers to when someone says that they do a holistic SEO approach or their services are considered holistic SEO. When they

use the word holistic SEO, you might want to ask them exactly what is considered holistic in terms of their services and why they consider that holistic SEO. That’s it for today’s Ask an SEO Episode 21. This is Brian Harnish signing off.

Please be sure to like and subscribe to our YouTube channel for a brand new episode every week. Have a great day!

Hello and welcome to Ask an SEO episode 20. Today’s question is what is Schema.org structured data and how can it help your SEO? OK, Schema.org structured data is basically a way of beautifying up data on a web page in a way that Google can understand and parse and display that data in a more beautiful fashion across its rich results within the search results pages.

And there are two types of Schema.org structured data that we are going to concern ourselves with. The number one first type is actual markup, where you mark up the specific data on the page.

The second one is JSON-LD, and this is a JavaScript based Schema.org structured data implementation, and this is generally the preferred way to implement Schema.org structured data across websites. So the thing that we’re going to focus on today for the purposes of this tutorial is JSON-LD Structured data.

So let’s move on and dive right in. All right. This is a website that you will be using a lot in your schema implementation. It’s called Schema.org. When you first begin a schema implementation, you have to do research behind the specific Schema.org structured data that you want to implement, such as if you’re implementing something for a person, a business organization, or any other number of entities that you want to give Google more information and context about. That’s what you’re going to be doing here on Schema.org. So if we want to find out a specific Schema.org data type that we want to use…say we want to go to attorney, for example, and you can see that there are…there is one Schema.org type for attorney, but we’ll go ahead and click on this here. And this will load all of the information related to this specific data type.

And what you’ll end up doing is when you create that Schema.org microdata, you will put in this Schema.org/Attorney URL at the beginning of your structured data, and you will be able to customize and put in any specific type of data that are listed here depending on what you want to do, and this is really going to be the beginning of your research. Say if we want to do something that’s for a person, for example. This will bring up Schema.org type for person and basically any person on the website you would want to add schema for, you can create that specific Schema.org structured data and it will be formatted in such a way that Google will be able to parse it and display it on their featured snippet in the Google search results.

Okay, so here we have our schema that has been created and the thing that we have to be careful about when creating schema, is we always want to make sure that it’s validated through Google structured data testing tool and through this website. This is Google’s Rich Results test. It has replaced the structured data testing tool. And this is really nice because you can test the specific URL or you can test bits of a code.

And today we’re going to test our specific code right here. First things first, let’s take a look at the schema that we have created here for me, and we will put this in here and let Google test our code for us.

Ah, this is a strange result here, so this says no item detected. That means that it’s not really much of a rich result if Google can’t detect that. So let’s go back and we’re going to go ahead and test our other specific URL – I’m sorry – our other code that we created. So well, this is for a specific article on iloveseo.com. We’ll put…we’ll copy and paste the code into here. Go ahead and click on “test code” down here. And we see that this specific schema is actually valid.

There is one valid item detected and we can see that Google has passed all of the elements in this schema code. So we are good to go to put this code into our website.

All right. That’s it for today’s Ask an SEO Episode 20. Please be sure to like and subscribe to our YouTube channel for a brand new episode every week. This is Brian Harnish signing off. Have a great day.

SEO Horror Story Number One: Robots.txt Blocks Indexing

Brian Harnish 0:59
Hello, and welcome to a scary Halloween edition of Ask an SEO. Buwahahahahahaaaa.

Number one: robots.txt blocks indexing. This is a big one. So you have a site and somebody had accidentally set robots.txt to block all indexing and crawling from Google. So what do you do, you have to go back in and set robots.txt back to crawling and indexing.

SEO Horror Story Number Two: Traffic Drop After a Redesign

Brian Harnish 1:51
Number two: a traffic drop occurs after you have redesigned the website. This could not necessarily be due to the redesign. What could be happening is you might have accidentally triggered a Google algorithmic devaluation due to the low quality of your site, in which case, you will want to do a deep dive into what’s specifically causing that actual penalty.

SEO Horror Story Number Three: Stolen Content

Brian Harnish 2:27
Number three: your content has all been stolen and duplicated across a number of different domains by an unscrupulous spammer. Okay, well, assuming you have found all the domains, then you have information needed to contact the web host of those domains, then you can file a DMCA or Digital Millennium Copyright Act request with your web host asking that they take down content and show – at least that way – they should comply when, when you provide proof that you own that specific content.

SEO Horror Story Number Four: You Suffered a Google Penalty

Brian Harnish 3:13
Number four: you have suffered a manual penalty. Now the thing about manual penalties that can make them the worst is you don’t always necessarily know exactly what’s going to be causing it. And when you go through link requests and other things of that nature, then you might have to go through quite a number of those before Google will allow you to reconsider your site. So that is going to be a situation where if you have a manual penalty, you have to put in the time and effort that you put into getting those rankings. Basically the same amount of effort, or more, that you put into getting there in order to get out of that penalty. So really, it’s just a good idea to stick to practices that don’t violate Google Webmaster Guidelines.

SEO Horror Story Number Five: Your Client Wants Results Tomorrow

Brian Harnish 4:13
Story number five: you have a client who wants results tomorrow. Sadly, with SEO, results are not going to come that quickly. So what – unless you have an extremely uncompetitive keyword that had no searches, or whatever. So this is an area where you can overcome that by really just making sure that during the client discovery process that you actually explain the risks, the pitfalls, downfalls, whatever, of everything that you’re proposing and make sure that they know that things will take – will generally take a while. They’re not going to be an instant overnight sensation.

SEO Horror Story Number Six: You Didn’t Pay Your Web Host

Brian Harnish 5:03
Nightmare story number six: you forgot to pay your web host and your site’s down! Well, this is something that is going to be a situation where you just have to pay your web host, and your site will be back up. That’s pretty much the end of that.

SEO Horror Story Number Seven: You Didn’t Pay Your Web Designer

Brian Harnish 5:30
Story number seven: you forgot to pay your web designer, and they have gone into your site and somehow activated a page explaining payment. Okay, so really, yeah, the situation here is they should have been paid. And even if you disagree with the work or whatever, the fact is, that the work was done. And unless it was really a terrible job, there’s really not much you can do about that, right? So it’s really best to pay your web designer, and they can, they will likely go ahead and restore it, whatever. So really just…just pay the designer and get it over with.

SEO Horror Story Number Eight: Another SEO Pro Has Died

Brian Harnish 6:21
Story number eight: This one’s kind of more of a macabre one. Simply because it is referring to SEO professionals, our favorite ones, who have died in the last few years. One of the ones that was a big shock was, of course, Eric Ward. And then we have Erik Stafford and we have had a couple of other SEO professionals who have passed away before their time. It’s always sad, we always miss them. And this is our chance to really recognize what they have done for the industry and to say, thank you so much.

SEO Horror Story Number Nine: Social Media is Down

Brian Harnish 7:10
Okay, story number nine: you sign on in the morning to find that Facebook, Twitter, LinkedIn, whatever, are all down. This has happened several times over the past year. And it has always really been quite a nightmare each time simply because like, that’s what we do, right? We market our websites on social platforms and get them to influencers, and so on and so forth. So it’s always a shock, to go online and find that, hey, a main function of your job is down, what are you gonna do? Well, anyway, you can’t really prevent what, what happens there when, and all you can really do is just shrug your shoulders, and it happened, right. So that’s, there’s really no major steps on this or whatever. But if that’s a major part of your job, maybe go back and look at tasks you might have had on the backburner for a while, and kind of make sure that they’re completed whatever. Either way, if it’s just one of those things that we never hope happens again, right?

SEO Horror Story Number Ten: Staging Site Has Been Indexed

Brian Harnish 8:34
And finally, we come to story number ten: you find that your site has a staging site at one point in the design process, right? But, you find about several months later that the designer did not deindex it and they didn’t go in and check “discourage search engines from crawling the site” and robots.txt is left wide open. And as a result, Google has indexed all of your staging site pages in addition to your site. So yeah, that is a major issue. What you can technically end up doing is really just go to the designer, make sure that they take off those particular places in robots.txt and WordPress if you’re on WordPress, and just make sure that all the staging site is deindexed.

Brian Harnish 9:33
Well that’s it for today’s Halloween edition of Ask an SEO! Buwahahahahahaaaa!

We hope you enjoyed this very scary edition of SEO horror stories! Be sure to tune in next year.

Have a great day!!

Hello and welcome to Ask an SEO Episode 18. Today’s question is: what is a robots.txt file and how can I ask you just for FCO? All right. A robots.txt file is really just a simple file that sits at the root of your domain and allows you to block certain sections of your site from crawling.

That’s really its main function. So if you have anything that you really don’t want to be ranking for or you don’t want indexed or crawled by Google, then you would put that in the robots.txt file. Now there are a few considerations that Google takes into account for that type of file, so let’s take a look at these and dive right in.

All right. This is the Google Web Developer documentation for robots.txt files. And we’re going to go through this line by line as far as the specific examples are concerned. All right. Let’s hop to it.

So, the first thing that they show really are examples of valid robots.txt URLs. And this is pretty important to go over because the last thing you want to do is not – is to include specific URLs that are not specifically valid, according to these particular guidelines on Google.

Now, it’s important to note that for the most part, when you actually include the path on robots.txt, for example, you’re usually not going to include the full URL. You’re usually going to include a directory structure beginning at wherever you want Google to stop crawling and indexing, right?

And next they talk about the handling of errors and HTTP status codes. And, it’s also important to take this into consideration, because this is something that the…that Google is going to take into account when it’s crawl…when it’s crawling the site right, or any time that Google crawls the site and you have a specific error code that’s showing up with the error code as a result, then that’s going to impact your crawling as well. So it’s important to take these into account. If you don’t, you run the risk of really introducing errors that shouldn’t be there.

For the valid syntax that Google actually wants you to follow: you want to follow the syntax that is outlined here, right? So for a user-agent, allow, disallow and the sitemap directives – these are the most common fields that you’re going to be using in robots.txt. So you’re always most likely going to want to identify the specific user agent.
This is going to be the crawler that you’re going to allow to crawl through the directories on your site. Then you can specifically allow a path to be crawled, but really that it’s mostly redundant where Google is concerned because Google is going to crawl and index anyway.

That’s their default option. You don’t have to really include allow unless you want to be a little bit more anal about it, I guess, for whatever reason. But anyway, disallowing is going to be probably the most used command that you’re going to use in robots.txt.

And finally, you’re also going to want to declare where your sitemap specifically is on your server. This will allow Google to find and index these specific sitemaps on your server properly, rather than relying on random guesses as to where it’s going to be.

And it’s pretty important to follow the Google Web Developer docs because these are going to show you a lot of instances like this, for example, when you have groupings of lines and rules. It will show you formatting, specific rules, and how you want to group those lines in robots.txt.

So this can be a very handy manual to troubleshooting any possible issues that you might come across when it comes to figuring out exactly what’s causing the root of your robot.txt problem. And also, as well, you want to make sure that you follow some of the recommendations on URL matching, which is based on path values.

So any of these path values, for example, the asterisk, you’re going to designate zero or more instances, more instances of any valid character, and then the dollar sign would designate the end of the URL. These are certain placeholders within robots.txt that behave a certain way when you use them.

So if you don’t use them correctly, you can introduce errors in crawling and indexing that way. All right that’s it for today’s Ask an SEO Episode 18. This is Brian Harnish signing off. Please be sure to like and subscribe to our YouTube channel for a brand new episode every week.

Have a great day!