Mage

Back-dated Posts Complex Solution

The Mage Jedi consists of specially configured xenu-wild tool and two more components.

With this method there is no need to disable any eBay, Amazon, CJ or any other merchant links in order to run it.
It doesn't care if you use affmage-ebay or phpbay.
You can keep even Sociable plugin activated.
It is much faster and more efficient than any other Xenu method described in mage forum.

1) You have to download and install on your PC special version of Xenu Link Sleuth.
The official version is 1.3.5 from Dec 2009.
I'm using XenuWild ver 1.3.6 beta, you can download it from the above document posted by Tilman, but go to Point 36 and get the wild version. See setup instructions below.

2) Install Dagon Design Sitemap Generator plugin for WP.
You can get it from WordPress downloads as well. Setup instructions below.

3) Download small file map.php, unzip it, and upload to root of your domain. You can rename it, if you want.

Don't go crazy with this method (or any method) trying to load (unpack) content from hundreds or thousands of posts in one run.

Use Dagon map generator to create smaller maps of up to about 200 posts per category and run it. Then create next category map and run it again.

This is to avoid timeouts and not to kill your server. On optimized VPS server you could go a little bit further, but still split very large maps to smaller ones, per category.

Tip - don't use Translation and you will have better content, faster loading/unpacking and no timeouts.

Suggested usage - create map with up to 200 posts (in one or two categories) to be unpacked in one run, in order to avoid too much stress on the server and content services.
Don't try to run to unpack 3,000 back-dated posts in one run!

Usage of the Jedi Mage solution for "content unpacking" in old, back-dated auto-generated posts or pages.

In Xenu-wild enter your URL.
If you use www.domain.com/, so you have to enter www. accordingly.

Also map.php should be done by category, preferably to not exceed 200 back-dated posts per run,

http://yourdomain.com/map.php

Check settings and options as per instructions below.
Click OK and that's it.

Mage Jedi Xenu-wild solution does not scan any other links on your website.
It does not scan or click on any links inside your posts either.
You are scanning only yourdomain.com/map.php, not your website.
Your recursion level must be set to "1", not "99".
You better save that "1" in your Xenu1.XEN file, otherwise it will go to 99 again.

Here is how to set up Dagon Sitemap Gen for this purpose.
Items per page could be set to 0 (unlimited) but it is better to limit it to 200 unpacked posts.

In Dagon sitemap generator, you can exclude categories you don't want a map be generated for.
So if you have 10 categories, then Exclude 9 and build map and run it through xenu-wild.
So your map will be for just one category.
To exclude categories from the map, you must enter category_ID (not category name), separated by commas.
If you select "newest first" your map will contain 200 most recent posts in the category, if you select "oldest first" - you get the oldest 200 posts listed. So this is the trick to have 400 posts per category created in 2 maps to be scanned by xenu-wild.
And repeat it so you have maps with one or 2 categories only, therefore more manageable number of posts to be triggered/unpacked.

Mike from Master the Mage created a video tutorial on sitemaps for xenu-wild.

If you don't have your posts categorized (they should be) I have an easy solution here as well.

For the old sites with many back-dated posts without content, just run xenu-wild with Dagon maps for 200 posts first, then later run it for 400 posts, then later run it for 600 posts, etc, until all 3,000 are done.

The posts which have content already unpacked go fast, the last 200 without content is the problem, so that's why it's better to not go higher than 200 of unpacked posts in each run. Then 30 min later you can run another round until everything is done.

When you are done with unpacking old, back-dated posts, then select Pages and Posts to be displayed and set number of items to unlimited (0). So now you can use this HTML sitemap together with XML sitemap, as Matt explains in the video below, to get your pages indexed faster.

There is additional SEO benefit of having both sitemaps, XML and HTML. For those who don't understand that, here is a tutorial from Matt Cutts himself.

Setup instructions for Xenu-wild.

I use 8 Parallel Threads, you can vary it, depends on your server/hosting.
Be sure Maximum level is set to 1.

You better save that "1" in your Xenu1.XEN file, otherwise it will go to 99 again and it will be scanning millions of files and the sky will fell down on you.