I have a database of all the towns / counties / postcodes in the UK - and using a plugin I am able to dynamically generate a page for each town. However, in the site map - there are only about 25 pages (typically the top level posts / pages)

Each page can be found / followed by links on the pages, but these are never added to the sitemap. How do I ensure all pages are added to the sitemap? - A similar example would be an e-commerce site - where you have one "page" which is dynamically populated with product specifics.

I'm a bit gutted really as creating the plugin as a non-wordpress / PHP developer took me a while to get exactly what I wanted with URL re-writing etc which works great (thanks to help from this site!) but now fallen at the last hurdle as there is no point in launching the site if none of the pages can be crawled and added to the site map.

Seems to be a problem specific to wordpress - as the same structure in pure PHP / ASP appears to work OK.

I have a database of all the towns / counties / postcodes in the UK - and using a plugin I am able to dynamically generate a page for each town. However, in the site map - there are only about 25 pages (typically the top level posts / pages)

Each page can be found / followed by links on the pages, but these are never added to the sitemap. How do I ensure all pages are added to the sitemap? - A similar example would be an e-commerce site - where you have one "page" which is dynamically populated with product specifics.

I'm a bit gutted really as creating the plugin as a non-wordpress / PHP developer took me a while to get exactly what I wanted with URL re-writing etc which works great (thanks to help from this site!) but now fallen at the last hurdle as there is no point in launching the site if none of the pages can be crawled and added to the site map.

Seems to be a problem specific to wordpress - as the same structure in pure PHP / ASP appears to work OK.

• seo
• sitemap

• I'm confused, you created a custom WordPress plugin to generate your sitemap. It isn't working as you expect, but it's 'a problem specific to WordPress'? – Steve Commented Sep 1, 2019 at 15:15
• Hi - no - I developed a plugin to dynamically generate pages based on locations in the UK. When crawled, none of the pages are indexed - even though there are links navigable from the front page. creating the same structure of links - using PHP or ASP.NET instead of my wordpress plugin - all the links are discovered. – Fozzy279 Commented Sep 1, 2019 at 18:30

2 Answers 2

First off, WordPress itself is pretty good about SEO and structure right out of the box, there are countless WordPress websites that rank in the top 10 if not at number one for a given search. There is nothing at all in your OP that would lead me to believe WordPress is the problem.

Second I think you are confused about what a sitemap is. From your OP and comments, it sounds like you are actually talking about what pages are indexed via some search engine(s).

What is a sitemap?

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

from sitemaps

Sitemaps are very helpful in getting a site and it's content indexed, but you have to implement the sitemap within your website and follow the steps to submit it to Google or other search engines. In the case of Google, you need to use Google Search Console.

List indexed pages.

To list all of the pages that a search engine has indexed for a given website, go to that search engine and enter the below replacing the example with the domain name you're searching for. I know that bing and Google support this, but all search engines may not.

site:example

Troubleshoot crawl and indexing errors.

To determine why a given page or pages have not been indexed you'll need to use Google's Search Console or the individual search engines tool to determine what pages have been scanned and what technical errors may have occurred that are preventing them from being indexed. Keep in mind that these results are typically technical results and errors such as a 404 encountered during the scan.

Content and SEO.

Even if a search engine has scanned your entire website and encountered no technical errors at all, that doesn't mean that your pages will be indexed by the search engine.

Google and others use a variety of rules to determine if the content of the pages are meaningful and worthy of being indexed. So a perfectly served up page with limited content, poorly structured content, or content that Google doesn't see as relevant still won't be added to the index.

Troubleshoot your website.

Review your access logs, error logs, .htaccess file, and PHP logs. These will often indicate that you have a problem with your PHP, Apache, or .htaccess configuration.

Rewrite Rules

You mention rewrite rules in your OP and I can tell you from my experience that they can be very tricky and problematic. I've seen rules that appear to work perfectly except when they don't, so double check your rules and verify that you're getting the behavior you want.

FYI I found the problem - my dynamically generated pages did not have a trailing "/" t the end of the URL. Once I added this, the site was crawled as expected. Thanks for all your comments.