16. Changelog

This page lists what changed in each version of WP Content Crawler.

Important

Please backup your database before upgrading or downgrading the plugin

16.1. v1.14.0 - 29 October 2023

New
Improvement
  • Turns URLs shown in the test results into clickable links
  • Adds gpt-3.5-turbo-16k and gpt-3.5-turbo-instruct models to OpenAI GPT Short Code creator
  • Adds Updated, Deleted and Other URL types to the Clear URLs Tool
Fix
  • WooCommerce Section is not shown in the multi-site environment when WooCommerce is not activated for the entire network but only for a single site.
  • Certain prompts used in [openai-gpt] short code cause an infinite loop, making PHP stop execution
  • Decode HTML entities of the JSON values found via Options Box
  • Contents found by Post Content Selectors must not be wrapped with a p element
  • Values of “0” found by CSS selectors must not be ignored
  • Remove empty HTML elements and comments setting causes a PHP fatal error for HTML elements containing too many nodes
Other
  • Tested and works with WordPress 6.3, 6.2, 6.1, 6.0, 5.9, 5.8, 5.6, 5.5, 5.4, 5.3, 5.2, with PHP 8.2, 8.1, 8.0, 7.4, 7.3, with Chrome, Firefox, Opera, with WooCommerce 8.2, 8.1, 7.9, 7.8, 7.2, 7.1, 7.0, 6.9, 6.1, 5.2, 5.1, 4.9, 4.5
  • Updates third party libraries
  • Minor fixes and improvements
  • Code quality improvements

16.2. v1.13.1 - 7 April 2023

Fix
  • Custom and generative short codes used in the Post Date Selectors setting’s options box should be applied
  • Remove the buttons of the short codes that are not available for use from the Templates Tab of the Post Date Selectors setting’s options box
  • File names cannot be retrieved from the Content-Disposition header in WordPress 6.2
  • When translating with Amazon Translate API, too many requests error occurs
  • Translation does not work for long texts

16.3. v1.13.0 - 26 March 2023

New
  • Adds [openai-gpt] short code. This is used to generate text by using OpenAI API, including ChatGPT. See OpenAI GPT Short Code for more information.
  • Adds DeepL Translate API
  • Adds Create command to the Element subject. This command is used to create HTML elements and insert them relative to other elements. See Inserting text/HTML into the middle of the post content for more information.
  • Adds Field template command to the Post subject. This is used to apply a template to all the selected transformable fields.
  • Adds Unwrap command to the Element subject. This is used to remove an element’s tags, while keeping its content.
Improvement
Fix
  • Symbolic-linking the uploads directory causes the file URLs to be invalid
  • Cookie values must not be URL-encoded. Instead, they must be sent as-is.
  • Disable SSL verification? setting’s value is not used when running a proxy test in the general settings page.
Other
  • Tested and works with WordPress 6.1, 6.0, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.0, with PHP 8.2, 8.1, 8.0, 7.4, 7.3, with Chrome, Firefox, Opera, with WooCommerce 7.5, 7.4, 7.3, 7.2, 7.1, 7.0, 6.9, 6.5, 6.1, 5.2, 5.1, 4.9, 4.5
  • Updates third party libraries.
  • Minor fixes and improvements
  • Code quality improvements

16.4. v1.12.0 - 15 May 2022

Important

This version requires at least PHP 7.3

New
Improvements

  • For the paginated posts, adds the “next page” separator as a Gutenberg block definition. By this way, the themes that render the page navigation via blocks (such as Twenty Twenty-Two theme) can display the page navigation.

  • Displays a notice in Dashboard Page page when the scheduling is not active

  • Updates the languages available for Amazon Translate API. More than 40 new languages are added.

  • Updates the regions available for Amazon Translate API. More than 10 regions are added.

  • Updates the default user agent. The existing HTTP User Agent setting is not updated. The default user agent is

    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36

  • Marks Site URL and Category URLs settings as required, by adding a red star next to their labels

  • Displays the options of category selects in hierarchical format. Also, the names of the categories are not escaped. This fixes the issue of HTML special chars used in the category names not being displayed properly.

Fix
  • Trying to remove an element multiple times causes a fatal error
  • The dismiss button of the notices do not work
  • Notifications should be placed correctly in the page
  • When the browser is re-opened in mobile devices, the auto-starting guides should not start after cancellation
  • The secure URL for TurkceSpin API should be used. Also, the error message retrieved from the API should be displayed.
  • When saving files, the file name is not retrieved from Content-Disposition header
Requirements
  • Requires at least PHP 7.3
Other
  • Tested and works with WordPress 6.0, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, with PHP 8.1, 8.0, 7.4, 7.3, with Chrome, Firefox, Opera, with WooCommerce 6.5, 6.4, 6.3, 6.2, 6.0, 5.7, 5.5, 5.0, 4.9, 4.5
  • Minor fixes and improvements
  • Code quality improvements

16.5. v1.11.0 - 11 March 2021

New
  • Filters. With filters, you can do things conditionally, i.e. you can say “if this happens, then do this”, where the condition part is optional. The filters have 30 Condition Commands and 25 Action Commands. Via the filters’ Action Commands, these are possible:
    • Make a text uppercase, lowercase, title case, snake case, kebab case, camel case, studly case, slug, and make its first letter uppercase
    • Find and replace in a text, such as post title or product attribute value
    • Clear a text. For example, you can clear a tag’s text to remove it if a certain condition is met.
    • Limit the number of characters and words in a text or HTML code
    • Remove empty HTML elements in a text
    • Remove links targeting specific domains or not targeting specific domains in a text or element
    • Add something before or after a text
    • Remove an element from the source code retrieved from the target site
    • Clone an element
    • Calculate the numeric values. For example, you can change the price of a product depending on a condition.
    • Stop crawling a post and delete it from your site if a certain condition is met. For example, you delete the post if the post is not found in the target site when recrawling. Another example is to delete a post if the title or content has a specific word that you do not want in your site.
    • Set author of the post depending on a condition. For example, you can change the author of the post if the post has a specific category.
    • Set post status, Add tags, Add categories
    • Send email notifications. For example, if a condition is satisfied, you can change the post status to pending review and send an email notification to yourself so that you can review and publish it.
Improvement
  • Import/export Cookies. You can copy the cookie header from the network tab of the developer tools of your browser and import it. See Importing all cookies for more information.
  • Post Status option contains all the post statuses registered to WordPress.
  • Options Box opens much faster
  • Error message displayed when a setting is not configured as expected now displays the path to to problematic setting so that you can understand which setting has to be fixed.
Fix
  • Test results of a setting that has a File Options Box should not modify the result if the options box does not have any options configured.
  • WooCommerce options are shown in multi-site installations as well
  • wpcc/post/source-code/after_retrieve action is not triggered when crawling a category page. Instead, wpcc/category/source-code/after_retrieve action is triggered.
Other
  • Tested and works with WordPress 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9, with PHP 8.0, 7.4, 7.3, 7.2, with Chrome, Firefox, Opera, with WooCommerce 5.1, 5.0, 4.9, 4.8, 4.7, 4.6, 4.5, 4.4, 4.3, 4.2, 4.1, 4.0, 3.9, 3.8, 3.6, 3.3.
  • Minor fixes and improvements

16.6. v1.10.2 - 31 January 2021

Requirements
  • Requires PHP’s fileinfo extension. According to this extension’s official installation page, “this extension is enabled by default as of PHP 5.3.0”. This means that this extension is already available when the minimum PHP version requirement of the plugin is satisfied. See Requirements for more information.
Fix
  • Support PHP 8.0
  • Add cookies when downloading files
  • Define file paths relative to the plugin’s root path. With this, it should be possible to use the plugin with custom wp-content or plugins directory paths.
  • Disable comments when Allow Comments setting is unchecked
  • Improve guessing file name and extension when saving files
  • Do not use nested ternary operators without explicit parentheses, since this is deprecated
  • Remove undefined function dd()
Other
  • Tested and works with WordPress 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9, with PHP 8.0, 7.4, 7.3, 7.2, with Chrome, Firefox, Opera, with WooCommerce 4.9, 4.8, 4.7, 4.6, 4.5, 4.4, 4.3, 4.2, 4.1, 4.0, 3.9, 3.8, 3.7, 3.6, 3.3.

16.7. v1.10.1 - 25 August 2020

Fix
  • Make it compatible with the major changes made in WordPress 5.5. WordPress 5.5 does not include jQuery Migrate, which lets deprecated jQuery functions be used for backward compatibility. This caused the plugin’s some JavaScript features to not work properly. This version removes the usage of any deprecated jQuery functions. Additionally, as instructed by WordPress’s official Test jQuery Updates plugin, the plugin has been thoroughly tested with all of the 4 specified combinations of jQuery, jQuery Migrate, and jQuery UI. This means that the plugin should now be compatible as well with the jQuery changes that will be made in the future WordPress versions.
  • Abandon the usage of PHP features deprecated in PHP 7.4.
  • Tooltip must not block the underlying UI elements even if it is not visible.
Other
  • Tested and works with WordPress 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9, with PHP 7.4, 7.3, 7.2, with Chrome, Firefox, Opera, with WooCommerce 4.4, 4.3, 4.2, 4.1, 4.0, 3.9, 3.8, 3.7, 3.6, 3.5.

16.8. v1.10.0 - 13 February 2020

New
  • Guides. Interactive guides show you step-by-step how to use the plugin to achieve certain things such as automatically crawling posts, using custom general settings, saving lazy-loading images, saving categories from target post pages, and many more, a total of 20 guides. You can start the guides any time you want. You can even start a guide from any step you want. Guides can be reached by clicking to Guides button shown at the bottom right corner of the pages of the plugin.
  • TürkçeSpin API, a spinning service used to spin Turkish texts, is added.
Fix
  • Some parts of the texts are not translated.
  • Category ID is saved as term name when taxonomy isn’t hierarchical
  • CSS selectors of some elements cannot be found via Visual Inspector in some cases where one of their parent elements contains more than one class.
  • Partial translations fr_FR and ko_KR are removed. The plugin currently has only Turkish translations.
Other
  • Tested and works with WordPress 5.3, 5.2, 5.1, 5.0, 4.9, with PHP 7.4, 7.3, 7.2, with Chrome, Firefox, Opera, Safari, Edge, and with WooCommerce 3.9, 3.8, 3.7, 3.6, 3.5.
  • Updates third party libraries.
  • Other small fixes and improvements

16.9. v1.9.0 - 13 December 2019

Important

This version requires at least PHP 7.2

New
  • Spinning (See: Spinning Tab). You can spin the crawled post’s content and other details by using Spin Rewriter (See: Spin Rewriter API Options). Note that this is a paid service. You can select what parts of the post should be spun, as well (See: Spinnable Fields).
  • It is now much easier to find CSS selectors. Visual Inspector can now find CSS selectors in four different modes named as unique, similar, similar (specific) and contains. See Selection behavior for more information.
  • Translate with Amazon Translate (See: Amazon Translate API Options) and Yandex Translate (See: Yandex Translate API Options) APIs.
  • Select what types of information should be translated (See: Translatable Fields).
  • Remove all scripts, empty HTML tags, and HTML comments from all parts of the post by simply checking a few checkboxes. The settings are available under Quick Fixes Section section (See Remove empty HTML elements and comments and Remove scripts settings).
  • It is now possible to reset the general settings of the plugin. Just click to Reset General Settings button in General Settings Page.
  • Save all pages of the target post as a single page (See: Save as single page?)
  • Permissions to limit which user roles can view pages and update settings of the plugin (See: Permissions). The permissions contain wpcc in their name. So, you can look for those in your user role editor plugin.
  • Detailed documentation that explains features of the plugin with images, videos, and examples and includes several quick tutorials guiding you on how to do certain things.
Improvement
Fix
  • Active tabs are not restored in General Settings and Tools pages.
  • Plugin cannot get the content when there is an invalid URL in the target page’s source code.
  • URLs are not resolved properly in Visual Inspector.
  • The last post URL found in a category page has a featured image although it should not.
  • Infinite loop occurring when the next page URL of the last page of a post points to the first page of the post.
  • Translation test results are not shown when the test text is too long.
Other
  • Tested and works with WordPress 5.3, 5.2, 5.1, 5.0, 4.9, with PHP 7.4, 7.3, 7.2, with Chrome, Firefox, Opera, Safari, Edge, and with WooCommerce 3.8, 3.7, 3.6, 3.5.
  • Updates third party libraries.
  • Other small fixes and improvements
  • Requires at least PHP 7.2, which is the oldest actively-supported version. See: Supported Versions at PHP.net.

16.10. v1.8.0 - 1 January 2019

New
  • Save WooCommerce product details much more easily by using the options specifically defined for WooCommerce products. To see the options under Post tab of site settings, just select the post type as “product” either in General Settings or by defining custom general settings. The options are available for WooCommerce versions 3.3, 3.4, and the latest one, 3.5.
  • Save categories.
  • Save post slugs.
  • Save taxonomy values.
  • Save the posts into custom post categories. You can define custom post category taxonomies in general settings so that you can select the custom post categories when saving a post.
  • Options Box. For the settings that have Options Box button, you can define several settings for each item found by given CSS selectors. Options Box contains find-replace, calculation, and templating options. You can also take notes. It also allows you to use JSON values in calculations and templates.
  • Rename, copy, and move saved files. You can also define title, description, caption, and alt texts for the saved media files by using templates in which you can use any short code. It is also possible to give random names to the saved files.
  • Recent tests in Site Tester page. You can now repeat your previous tests easily.
  • Caching responses of test URLs in site settings page. You can now configure the settings faster and send less number of requests to the target site.
  • Replace iframe and script HTML elements with short codes by just checking a checkbox. The options are available under Templates tab.
  • Quick save button in the site settings. Now you can save the site settings faster (much faster).
  • You can use delimiters and modifiers for regular expressions in find-replace options.
  • Sets srcset attribute values of img elements of the saved image files in the templates when different sizes of the images are available.
  • Warnings. When there is an error, you will get a warning showing the details of the warning/error.
  • Saves “alt” and “title” values of media items when they are saved as attachments.
  • Adds “wpcc/post/settings/meta-key-defaults” filter that you can use to set default values of site settings.
  • Convert character encoding to UTF8 when target page’s HTML has a different encoding. You can enable the option under General Settings > Advanced.
  • You can now navigate between tabs and settings much more easily when you activate fixing tabs and content navigation under Main tab of site settings.
  • Adds “find and replace in raw response HTML” option for post and category settings. Using this, you can fix HTML errors that prevent the plugin from being able to parse the HTML code.
  • Adds “wpcc/bot/response-content” filter that can be used to manipulate raw response content.
Improvement
  • Manual crawling tool has been redesigned. Now, you can manually crawl multiple URLs or insert post URLs to the database so that they can be crawled later. You can also perform parallel crawling. Moreover, you can recrawl the posts directly from the manual crawling tool.
  • When testing your settings in the site settings page, all manipulations defined in your settings will be applied. By this way, you can conduct more robust tests and figure out the cause of a misbehavior more easily.
  • Short code buttons now contain custom short codes defined by you.
  • The files that are saved when testing are now deleted from the file system after the test.
  • Shows all types of saved posts, including custom post types, in the dashboard.
  • Uses the HTTP user agent defined in the settings when saving media.
Fix
  • You can now enter cookies without decoding them. Just copy and paste the values retrieved from your browser.
  • In “Find and replace in custom meta” option, only one replacement was applied to each meta key. Now, all replacements will be applied sequentially.
  • Images having “&” symbol in their URLs are not saved properly.
  • Invalid chars coming after the file’s extension in the file’s URL (such as png:s) cause the files not to be saved with the right extension.
  • When there were no API keys for translation services and the translation was active, a fatal error were shown. Now, it is handled silently.
  • Scroll animation does not work.
  • When testing find and replace settings for custom short codes, all test data options are required although one of them is enough to perform the test.
  • The plugin causes the text editor in “Edit Page” page to be double.
  • Relative URLs should be resolved automatically.
Other
  • Tested and works with PHP 5.6, 7.0, 7.1, and 7.2 and with Chrome, Firefox, Opera, Safari, and Edge.
  • Updates limits and API versions of Google’s and Microsoft’s translation services.
  • Updates third party libraries.
  • Other small fixes and improvements.
  • No longer supports Internet Explorer.

16.11. v1.7.0 - 22 October 2017

New
  • Translate posts automatically by using Google Cloud Translation API or Microsoft Translator Text API.
  • Randomize proxies. By checking this option, you can make the plugin randomly order the proxies you entered.
  • Over 50 filters and actions are added. If you are a developer, you can now use these to extend the plugin however you like.
Fix
  • The proxies were used when there was an error getting the target page’s source code. Now, they are always used, even when testing.
  • Plugin’s pages were not shown properly with PHP 7.1.
Other
  • UI and UX improvements.

16.12. v1.6.0 - 4 March 2017

New
  • Date selectors.
  • Add/remove minutes to/from the post date. You can schedule post publishing by this way.
  • Scheduled post delete.
  • Duplicate post checking via URL, title and/or content.
  • More HTML manipulation options: exchange element attributes, remove element attributes, find and replace in element attributes, manipulate HTML of an element.
  • Find and replace in custom short code and custom post meta content
Improvement
  • More counts are shown in site listing.
  • Save all images in the post content by checking a single checkbox.
  • Reorder settings that can have multiple values.
  • If the main template is empty, it will be considered as it contains [wcc-main-content] shortcode in it.
  • An option to always use UTF8 encoding.
  • Load general settings with a button when you are overwriting them for a site.
  • Settings are grouped and reordered for better navigation.
  • Auto refresh the dashboard every few seconds.
  • Track CRON events and the next sites that will be processed by the CRON events in the dashboard.
  • Better notifications for the required settings when performing a test.
  • Auto find for next page URL, post date and post title in DEV tools.
  • Remove elements by using a CSS selector in DEV tools. This can be used to remove blocking elements to better select the elements you want.
Fix
  • Sometimes thumbnail images and post URLs did not match when category pages were crawled.
  • When importing site settings, form validation should not be performed.
Other
  • Small bug fixes and improvements.

16.13. v1.5.1 - 7 February 2017

New
  • Dashboard. See what’s going on behind the scenes.
Other
  • Bug fixes and improvements.

16.14. v1.4.1 - 27 January 2017

Fix
  • URLs in the queue should be saved uniformly according to their categories.

16.15. v1.4.0 - 26 January 2017

New
  • Post recrawling. Recrawl posts to update them regularly.
  • Proxy tester. Test if your proxies work correctly.
  • Cookies. Attach cookies to every request that is made to the target site.
Other
  • Removes Lodash.
  • Small bug fixes and improvements.

16.16. v1.3.0 - 14 January 2017

New
  • Visual Inspector
Fix
  • Assets are not loaded on Windows servers.
  • “General settings” link on plugins page does not work.
  • Plugin does not crawl all active sites when there are more than 10 active sites.

16.17. v1.2.0 - 30 August 2016

New
  • You can now use proxy.
  • Set connection timeout in seconds.
  • Post title and excerpt templates in which you can use custom short codes.
  • Find and replace in custom short code data.
  • Maximum number of categories that can be added automatically via CSS selectors to the category map increased.
  • Add custom post meta without a selector.
  • You can set how many times URL collection and post crawling events should run each time for a site. For instance, you can save 3 posts every minute, or run URL collection 5 times every 2 minute.
  • You can collect post URLs in reverse order for each category page.
  • Remove links from all short code data. This will not touch the links manually added to the templates.
  • Notifications. You can now set CSS selectors whose values should not be empty for category and post pages. When an empty value is found by using those selectors, you can get an email notification.
Fix
  • Downloaded file’s name does not have a proper file extension if the file on the target site is generated dynamically.
  • Crawling stops if there is a request exception.
  • Crawling stops if target page’s HTML could not be retrieved.