Welcome to WP Content Crawler’s documentation
- 1. Getting Started
- 2. Quick Start
- 3. Guides
- 3.1. Creating site settings to crawl a site
- 3.2. Saving posts automatically
- 3.3. Updating posts automatically
- 3.4. Translating posts automatically
- 3.5. Spinning posts automatically
- 3.6. Deleting posts automatically
- 3.7. Using cookies
- 3.8. Importing all cookies of a site
- 3.9. Adding request headers
- 3.10. Importing request headers from browser
- 3.11. Increasing speed of tests in site settings page
- 3.12. Using custom general settings for a site
- 3.13. Adding category URLs automatically
- 3.14. Duplicating site settings
- 3.15. Manually saving posts
- 3.16. Manually updating posts
- 3.17. Tracking saved posts
- 3.18. Getting notified when selectors cannot find a value
- 3.19. Testing site settings
- 3.20. Collecting post URLs from multiple pages of a category
- 3.21. Saving multi-page posts
- 3.22. Saving list-type posts
- 3.23. Saving post meta (custom fields)
- 3.24. Saving taxonomy values
- 3.25. Creating post categories automatically
- 3.26. Saving posts as custom post types
- 3.27. Disabling fixed navigation and tabs in site settings page
- 3.28. Saving post title
- 3.29. Saving post excerpt
- 3.30. Saving post content
- 3.31. Saving post tags
- 3.32. Saving post permalink (slug)
- 3.33. Saving post date
- 3.34. Saving the featured image of a post
- 3.35. Saving images in post content
- 3.36. Saving lazy-loaded images
- 3.37. Defining custom short codes to use anything in templates
- 3.38. Saving WooCommerce products
- 3.39. Saving featured images from category pages
- 3.40. Defining how posts are marked as duplicate
- 3.41. Adding/removing things to/from post content
- 3.42. Adding/removing things to/from post title
- 3.43. Adding/removing things to/from post excerpt
- 3.44. Saving posts as draft or pending (defining post status)
- 3.45. Setting the author of the posts
- 3.46. Removing the links in post content
- 3.47. Showing
iframe
s in post content - 3.48. Showing
script
s in post content - 3.49. Creating a gallery of images or files in post content
- 3.50. Taking notes about site settings
- 3.51. Changing something in every page of every site
- 3.52. Removing an HTML element from post content
- 3.53. Dealing with character encoding problems
- 3.54. Using proxies
- 3.55. Adding/removing/changing things in target page
- 3.56. Limiting maximum tags posts can have
- 3.57. Setting password to posts
- 3.58. Setting HTTP User Agent and HTTP Accept values
- 3.59. Manually adding post URLs
- 3.60. Deleting post URLs of a site
- 3.61. Deleting a post when it is not found when recrawling
- 3.62. Assigning a featured image from media library
- 3.63. Disabling SSL verification for a site
- 3.64. Inserting text/HTML into the middle of the post content
- 3.65. Using ChatGPT
- 3.66. Prompt chaining with ChatGPT
- 3.67. Structuring text with ChatGPT
- 4. Video Tutorials
- 5. Site Listing Page
- 6. Site Settings Page
- 6.1. Creating a New Site
- 6.2. Saving The Settings
- 6.3. Tabs
- 6.3.1. Main Tab
- 6.3.2. Category Tab
- 6.3.3. Post Tab
- 6.3.3.1. Post
- 6.3.3.2. Category Section
- 6.3.3.3. WooCommerce Section
- 6.3.3.4. Date Section
- 6.3.3.5. Meta Section
- 6.3.3.6. Featured Image Section
- 6.3.3.7. Images Section
- 6.3.3.7.1. Save all images in the post content?
- 6.3.3.7.2. Save images as media?
- 6.3.3.7.3. Save images as gallery?
- 6.3.3.7.4. Gallery Image URL Selectors
- 6.3.3.7.5. Save images as WooCommerce product gallery?
- 6.3.3.7.6. Image URL Selectors
- 6.3.3.7.7. Test Image URL
- 6.3.3.7.8. Find and replace in image URLs
- 6.3.3.8. Custom Short Codes Section
- 6.3.3.9. List Type Posts Section
- 6.3.3.10. Pagination Section
- 6.3.3.11. Post Meta Section
- 6.3.3.12. Taxonomies Section
- 6.3.3.13. Manipulate HTML Section
- 6.3.3.14. Unnecessary Elements Section
- 6.3.3.15. Filters Section
- 6.3.3.16. Notifications Section
- 6.3.3.17. Other Section
- 6.3.4. Templates Tab
- 6.3.4.1. Templates
- 6.3.4.2. Item Templates Section
- 6.3.4.3. Quick Fixes Section
- 6.3.4.4. Unnecessary Elements Section
- 6.3.4.5. Manipulate HTML Section
- 6.3.4.5.1. Find and Replace Test Code
- 6.3.4.5.2. Find and replace in post’s content
- 6.3.4.5.3. Find and replace in custom short code contents
- 6.3.4.5.4. Find and replace in post’s title
- 6.3.4.5.5. Find and replace in post’s excerpt
- 6.3.4.5.6. Find and replace in post’s each tag
- 6.3.4.5.7. Find and replace in meta keywords
- 6.3.4.5.8. Find and replace in meta description
- 6.3.5. Filters Tab
- 6.3.6. Notes Tab
- 6.3.7. Settings Tab
- 6.3.8. Import/Export Tab
- 6.4. Visual Inspector
- 6.4.1. Back
- 6.4.2. Forward
- 6.4.3. Refresh
- 6.4.4. Address bar
- 6.4.5. Go
- 6.4.6. Show sidebar
- 6.4.7. Use CSS selector
- 6.4.8. CSS selector
- 6.4.9. Test CSS selector
- 6.4.10. Clear highlights
- 6.4.11. Show alternative CSS selectors
- 6.4.12. Remove elements
- 6.4.13. Toggle hover select
- 6.4.14. Target HTML tag
- 6.4.15. Selection behavior
- 6.4.16. Test type
- 6.4.17. Manipulations
- 6.4.18. Use immediately
- 6.4.19. Remove scripts
- 6.4.20. Remove styles
- 6.4.21. URL History
- 6.4.22. Delete URL history
- 6.4.23. Alternative CSS selectors
- 6.4.24. All used selectors
- 6.4.25. Show/hide section
- 6.4.26. Close sidebar
- 6.4.27. WP Content Crawler Variables
- 6.5. Config Helper
- 6.6. Lifecycle of Events
- 7. General Settings Page
- 7.1. Scheduling Tab
- 7.1.1. Scheduling
- 7.1.1.1. Scheduling is active?
- 7.1.1.2. Post URL Collection Interval
- 7.1.1.3. Post Crawl Interval
- 7.1.1.4. Maximum number of pages to crawl per category
- 7.1.1.5. Maximum number of pages to check to find new post URLs
- 7.1.1.6. Run count for URL collection event
- 7.1.1.7. Run count for post crawling event
- 7.1.2. Recrawling Section
- 7.1.3. Deleting Section
- 7.1.1. Scheduling
- 7.2. Post Tab
- 7.3. Translation Tab
- 7.4. Spinning Tab
- 7.4.1. Spinning
- 7.4.2. Spin Rewriter API Options
- 7.4.2.1. Email address
- 7.4.2.2. API Key
- 7.4.2.3. Confidence Level
- 7.4.2.4. Auto Protected Terms
- 7.4.2.5. Nested Spintax
- 7.4.2.6. Auto Sentences
- 7.4.2.7. Auto Paragraphs
- 7.4.2.8. Auto New Paragraphs
- 7.4.2.9. Auto Sentence Trees
- 7.4.2.10. Use Only Synonyms
- 7.4.2.11. Reorder Paragraphs
- 7.4.2.12. Text with spintax?
- 7.4.2.13. Test Spinning Options
- 7.4.2.14. Check API Usage Statistics
- 7.4.3. Türkçe Spin API Options
- 7.5. APIs Tab
- 7.6. SEO Tab
- 7.7. Notifications Tab
- 7.8. Advanced Tab
- 7.1. Scheduling Tab
- 8. Tester Page
- 9. Dashboard Page
- 10. Tools Page
- 11. License Settings Page
- 12. Common Options
- 12.1. Convert JSON to HTML
- 12.2. Filter Setting
- 12.2.1. Filter
- 12.2.2.
if
part (or condition part) - 12.2.3.
then
part (or action part) - 12.2.4. Condition block
- 12.2.5. Command container
- 12.2.6. Subject
- 12.2.7. Property
- 12.2.8. Command
- 12.2.9. Event of the
if
part - 12.2.10. Event of the
then
part - 12.2.11. Condition operator
- 12.2.12. Collapse filter
- 12.2.13. Enable/disable filter
- 12.2.14. Side-by-side view
- 12.2.15. Move filter up
- 12.2.16. Move filter down
- 12.2.17. Clone filter
- 12.2.18. Delete filter
- 12.2.19. Delete condition block
- 12.2.20. Move command up
- 12.2.21. Move command down
- 12.2.22. Move command out
- 12.2.23. Move command in
- 12.2.24. Clone command
- 12.2.25. Move command
- 12.2.26. Delete command
- 12.2.27. Add condition command
- 12.2.28. Add condition block
- 12.2.29. Add action command
- 12.2.30. Test subject input
- 12.2.31. Test command
- 12.2.32. Filter title/description
- 12.2.33. Command options
- 12.2.34. Summary of the
if
part - 12.2.35. Summary of the
then
part - 12.2.36. Expand/collapse command option summaries
- 12.2.37. Command summary
- 12.2.38. Enable/disable filter setting
- 12.2.39. Add new
- 12.2.40. Export
- 12.2.41. Import
- 12.2.42. Export filter
- 12.3. Filter Commands
- 12.3.1. Command Subjects
- 12.3.2. Subject Properties
- 12.3.2.1. Value
- 12.3.2.2. Character length
- 12.3.2.3. Word length
- 12.3.2.4. Numeric value
- 12.3.2.5. Element attribute value
- 12.3.2.6. Element numeric attribute value
- 12.3.2.7. Count
- 12.3.2.8. Attribute value
- 12.3.2.9. Numeric attribute value
- 12.3.2.10. HTML
- 12.3.2.11. Text
- 12.3.2.12. Numeric text
- 12.3.2.13. Tag name
- 12.3.2.14. Attribute value as JSON
- 12.3.2.15. Text as JSON
- 12.3.2.16. JSON attribute’s text value
- 12.3.2.17. JSON attribute’s numeric value
- 12.3.2.18. JSON attribute’s date value
- 12.3.2.19. Error
- 12.3.2.20. HTTP status code
- 12.3.2.21. Error text
- 12.3.3. Condition Commands
- 12.3.3.1. Greater than
- 12.3.3.2. Less than
- 12.3.3.3. Greater than or equal to
- 12.3.3.4. Less than or equal to
- 12.3.3.5. Equal to
- 12.3.3.6. Not equal to
- 12.3.3.7. Starts with
- 12.3.3.8. Does not start with
- 12.3.3.9. Ends with
- 12.3.3.10. Does not end with
- 12.3.3.11. Contains
- 12.3.3.12. Does not contain
- 12.3.3.13. Is uppercase
- 12.3.3.14. Is not uppercase
- 12.3.3.15. Is lowercase
- 12.3.3.16. Is not lowercase
- 12.3.3.17. Matches regex
- 12.3.3.18. Does not match regex
- 12.3.3.19. Exists
- 12.3.3.20. Does not exist
- 12.3.3.21. Is older than
- 12.3.3.22. Is older than fixed date
- 12.3.3.23. Is newer than
- 12.3.3.24. Is newer than fixed date
- 12.3.3.25. Is crawling
- 12.3.3.26. Is recrawling
- 12.3.3.27. Is first page
- 12.3.3.28. Is error
- 12.3.3.29. Is HTML error
- 12.3.3.30. Is request error
- 12.3.3.31. Exists (for errors)
- 12.3.3.32. Does not exist (for errors)
- 12.3.4. Action Commands
- 12.3.4.1. Assign
- 12.3.4.2. Clear
- 12.3.4.3. Find and replace
- 12.3.4.4. Make upper case
- 12.3.4.5. Make lower case
- 12.3.4.6. Make title case
- 12.3.4.7. Make snake case
- 12.3.4.8. Make kebab case
- 12.3.4.9. Make camel case
- 12.3.4.10. Make studly case
- 12.3.4.11. Make first letter upper case
- 12.3.4.12. Make slug
- 12.3.4.13. Limit words
- 12.3.4.14. Limit characters
- 12.3.4.15. Remove empty HTML elements
- 12.3.4.16. Remove links
- 12.3.4.17. Template
- 12.3.4.18. Exchange attributes
- 12.3.4.19. Remove
- 12.3.4.20. Remove attributes
- 12.3.4.21. Remove data attributes
- 12.3.4.22. Simplify
- 12.3.4.23. Clone
- 12.3.4.24. Create
- 12.3.4.25. Move
- 12.3.4.26. Unwrap
- 12.3.4.27. Calculate
- 12.3.4.28. Add minutes
- 12.3.4.29. Set to fixed date
- 12.3.4.30. Set to now
- 12.3.4.31. Set scheduling
- 12.3.4.32. Set recrawling
- 12.3.4.33. Set post deleting
- 12.3.4.34. Send email notification
- 12.3.4.35. Stop
- 12.3.4.36. Stop and delete the post
- 12.3.4.37. Wait
- 12.3.4.38. Set author
- 12.3.4.39. Set featured image
- 12.3.4.40. Set post status
- 12.3.4.41. Add tags
- 12.3.4.42. Add categories
- 12.3.4.43. Field template
- 12.3.4.44. Translate
- 12.3.4.45. Add post URLs
- 12.3.4.46. Translate element attributes
- 12.3.4.47. Convert to HTML
- 12.3.4.48. Convert to HTML automatically
- 12.3.4.49. Make
- 12.3.4.50. Remove
- 12.4. Find and Replace Setting
- 12.4.1. Examples
- 12.4.1.1. Find and replace a word in a sentence in a case-sensitive way
- 12.4.1.2. Remove a word from a sentence
- 12.4.1.3. Find and replace a word in a sentence in a case-insensitive way
- 12.4.1.4. Find and replace by using a regular expression
- 12.4.1.5. Find and use something from the found text in the replacement
- 12.4.1.6. Replace
data-src
attribute’s value withsrc
attribute’s value
- 12.4.1. Examples
- 12.5. JSON path
- 12.6. Manipulate HTML Section
- 12.6.1. Test code for find-and-replaces in HTML
- 12.6.2. Find and replace in raw HTML
- 12.6.3. Find and replace in HTML at first load
- 12.6.4. Find and replace in element attributes
- 12.6.5. Exchange element attributes
- 12.6.6. Remove element attributes
- 12.6.7. Find and replace in element HTML
- 12.6.8. Convert JSON to HTML
- 12.6.9. Convert JSON to HTML automatically?
- 12.7. Options Box
- 12.8. Selector and Attribute Setting
- 12.9. Short Codes
- 12.10. Transformable Fields
- 12.11. Unnecessary Elements Section
- 13. Permissions
- 14. Frequently Asked Questions
- 14.1. Where is my license key?
- 14.2. Can I get content from X site?
- 14.3. Can I save custom fields/taxonomies of a plugin/theme?
- 14.4. How to change the domain registered to my license
- 14.5. In how many domains can I use my license?
- 14.6. I cannot crawl sites that use SSL (https)
- 14.7.
embed
,script
, andiframe
codes are not shown properly, or not shown at all - 14.8. How to change PHP version and enable
mbstring
extension? - 14.9. How to remove links from images
- 14.10. Automatic crawling works only when someone browses my site
- 14.11. I installed the plugin on a test site and I want to use the plugin with the same license code on another site. Can I do it?
- 14.12. How to send a support request
- 14.13. What do I do when admin and FTP login credentials are asked from me?
- 14.14. The plugin does not work
- 15. Troubleshooting
- 15.1. Checking if CRON jobs work and fixing CRON problems
- 15.2. Fixing error messages like “Parse error: syntax error…”
- 15.3. Fixing automatic crawling problems
- 15.4. Fixing invisible tab contents in setting pages
- 15.5. Fixing unsaved settings problem
- 15.6. Fixing empty plugin pages problem
- 15.7. Fixing the problem of empty pages shown after saving the settings
- 15.8. Fixing the problem of error pages shown after saving the settings
- 15.9. Fixing automatic crawling problems caused by the theme Newspaper 8 (or later)
- 15.10. Fixing cURL-related automatic crawling problems
- 15.11. Fixing duplicate post issues
- 15.12. Fixing character encoding problems
- 15.13. Manual Crawling Tool’s buttons do not work
- 15.14. Troubleshooting proxy issues
- 15.15. Fixing the problems caused by other plugins/themes
- 15.16. Debugging errors
- 16. Changelog
- 16.1. v1.15.0 - 7 September 2024
- 16.2. v1.14.0 - 29 October 2023
- 16.3. v1.13.1 - 7 April 2023
- 16.4. v1.13.0 - 26 March 2023
- 16.5. v1.12.0 - 15 May 2022
- 16.6. v1.11.0 - 11 March 2021
- 16.7. v1.10.2 - 31 January 2021
- 16.8. v1.10.1 - 25 August 2020
- 16.9. v1.10.0 - 13 February 2020
- 16.10. v1.9.0 - 13 December 2019
- 16.11. v1.8.0 - 1 January 2019
- 16.12. v1.7.0 - 22 October 2017
- 16.13. v1.6.0 - 4 March 2017
- 16.14. v1.5.1 - 7 February 2017
- 16.15. v1.4.1 - 27 January 2017
- 16.16. v1.4.0 - 26 January 2017
- 16.17. v1.3.0 - 14 January 2017
- 16.18. v1.2.0 - 30 August 2016