Welcome to WP Content Crawler’s documentation
- 1. Getting Started
- 2. Quick Start
- 3. Guides
- 3.1. Creating site settings to crawl a site
- 3.2. Saving posts automatically
- 3.3. Updating posts automatically
- 3.4. Translating posts automatically
- 3.5. Spinning posts automatically
- 3.6. Deleting posts automatically
- 3.7. Using cookies
- 3.8. Importing all cookies of a site
- 3.9. Adding request headers
- 3.10. Importing request headers from browser
- 3.11. Increasing speed of tests in site settings page
- 3.12. Using custom general settings for a site
- 3.13. Adding category URLs automatically
- 3.14. Duplicating site settings
- 3.15. Manually saving posts
- 3.16. Manually updating posts
- 3.17. Tracking saved posts
- 3.18. Getting notified when selectors cannot find a value
- 3.19. Testing site settings
- 3.20. Collecting post URLs from multiple pages of a category
- 3.21. Saving multi-page posts
- 3.22. Saving list-type posts
- 3.23. Saving post meta (custom fields)
- 3.24. Saving taxonomy values
- 3.25. Creating post categories automatically
- 3.26. Saving posts as custom post types
- 3.27. Disabling fixed navigation and tabs in site settings page
- 3.28. Saving post title
- 3.29. Saving post excerpt
- 3.30. Saving post content
- 3.31. Saving post tags
- 3.32. Saving post permalink (slug)
- 3.33. Saving post date
- 3.34. Saving the featured image of a post
- 3.35. Saving images in post content
- 3.36. Saving lazy-loaded images
- 3.37. Defining custom short codes to use anything in templates
- 3.38. Saving WooCommerce products
- 3.39. Saving featured images from category pages
- 3.40. Defining how posts are marked as duplicate
- 3.41. Adding/removing things to/from post content
- 3.42. Adding/removing things to/from post title
- 3.43. Adding/removing things to/from post excerpt
- 3.44. Saving posts as draft or pending (defining post status)
- 3.45. Setting the author of the posts
- 3.46. Removing the links in post content
- 3.47. Showing
iframe
s in post content - 3.48. Showing
script
s in post content - 3.49. Creating a gallery of images or files in post content
- 3.50. Taking notes about site settings
- 3.51. Changing something in every page of every site
- 3.52. Removing an HTML element from post content
- 3.53. Dealing with character encoding problems
- 3.54. Using proxies
- 3.55. Adding/removing/changing things in target page
- 3.56. Limiting maximum tags posts can have
- 3.57. Setting password to posts
- 3.58. Setting HTTP User Agent and HTTP Accept values
- 3.59. Manually adding post URLs
- 3.60. Deleting post URLs of a site
- 3.61. Deleting a post when it is not found when recrawling
- 3.62. Assigning a featured image from media library
- 3.63. Disabling SSL verification for a site
- 4. Video Tutorials
- 5. Site Listing Page
- 6. Site Settings Page
- 6.1. Creating a New Site
- 6.2. Saving The Settings
- 6.3. Tabs
- 6.3.1. Main Tab
- 6.3.1.1. Main
- 6.3.1.1.1. Site URL
- 6.3.1.1.2. Active for scheduling?
- 6.3.1.1.3. Active for recrawling?
- 6.3.1.1.4. Active for post deleting?
- 6.3.1.1.5. Active for post translation?
- 6.3.1.1.6. Translatable Fields
- 6.3.1.1.7. Active for post spinning?
- 6.3.1.1.8. Spinnable Fields
- 6.3.1.1.9. Check duplicate posts via
- 6.3.1.1.10. Use custom general settings?
- 6.3.1.2. Request Section
- 6.3.1.3. Settings Page Section
- 6.3.1.1. Main
- 6.3.2. Category Tab
- 6.3.3. Post Tab
- 6.3.3.1. Post
- 6.3.3.2. Category Section
- 6.3.3.3. WooCommerce Section
- 6.3.3.4. Date Section
- 6.3.3.5. Meta Section
- 6.3.3.6. Featured Image Section
- 6.3.3.7. Images Section
- 6.3.3.7.1. Save all images in the post content?
- 6.3.3.7.2. Save images as media?
- 6.3.3.7.3. Save images as gallery?
- 6.3.3.7.4. Gallery Image URL Selectors
- 6.3.3.7.5. Save images as WooCommerce product gallery?
- 6.3.3.7.6. Image URL Selectors
- 6.3.3.7.7. Test Image URL
- 6.3.3.7.8. Find and replace in image URLs
- 6.3.3.8. Custom Short Codes Section
- 6.3.3.9. List Type Posts Section
- 6.3.3.10. Pagination Section
- 6.3.3.11. Post Meta Section
- 6.3.3.12. Taxonomies Section
- 6.3.3.13. Manipulate HTML Section
- 6.3.3.14. Unnecessary Elements Section
- 6.3.3.15. Filters Section
- 6.3.3.16. Notifications Section
- 6.3.3.17. Other Section
- 6.3.4. Templates Tab
- 6.3.4.1. Templates
- 6.3.4.2. Quick Fixes Section
- 6.3.4.3. Unnecessary Elements Section
- 6.3.4.4. Manipulate HTML Section
- 6.3.4.4.1. Find and Replace Test Code
- 6.3.4.4.2. Find and replace in post’s content
- 6.3.4.4.3. Find and replace in custom short code contents
- 6.3.4.4.4. Find and replace in post’s title
- 6.3.4.4.5. Find and replace in post’s excerpt
- 6.3.4.4.6. Find and replace in post’s each tag
- 6.3.4.4.7. Find and replace in meta keywords
- 6.3.4.4.8. Find and replace in meta description
- 6.3.5. Filters Tab
- 6.3.6. Notes Tab
- 6.3.7. Settings Tab
- 6.3.8. Import/Export Tab
- 6.3.1. Main Tab
- 6.4. Visual Inspector
- 6.4.1. Back
- 6.4.2. Forward
- 6.4.3. Refresh
- 6.4.4. Address bar
- 6.4.5. Go
- 6.4.6. Show sidebar
- 6.4.7. Use CSS selector
- 6.4.8. CSS selector
- 6.4.9. Test CSS selector
- 6.4.10. Clear highlights
- 6.4.11. Show alternative CSS selectors
- 6.4.12. Remove elements
- 6.4.13. Toggle hover select
- 6.4.14. Target HTML tag
- 6.4.15. Selection behavior
- 6.4.16. Test type
- 6.4.17. Manipulations
- 6.4.18. Use immediately
- 6.4.19. Remove scripts
- 6.4.20. Remove styles
- 6.4.21. URL History
- 6.4.22. Delete URL history
- 6.4.23. Alternative CSS selectors
- 6.4.24. All used selectors
- 6.4.25. Show/hide section
- 6.4.26. Close sidebar
- 6.5. Lifecycle of Events
- 7. General Settings Page
- 7.1. Scheduling Tab
- 7.1.1. Scheduling
- 7.1.1.1. Scheduling is active?
- 7.1.1.2. Post URL Collection Interval
- 7.1.1.3. Post Crawl Interval
- 7.1.1.4. Maximum number of pages to crawl per category
- 7.1.1.5. Maximum number of pages to check to find new post URLs
- 7.1.1.6. Run count for URL collection event
- 7.1.1.7. Run count for post crawling event
- 7.1.2. Recrawling Section
- 7.1.3. Deleting Section
- 7.1.1. Scheduling
- 7.2. Post Tab
- 7.3. Translation Tab
- 7.4. Spinning Tab
- 7.4.1. Spinning
- 7.4.2. Spin Rewriter API Options
- 7.4.2.1. Email address
- 7.4.2.2. API Key
- 7.4.2.3. Confidence Level
- 7.4.2.4. Auto Protected Terms
- 7.4.2.5. Nested Spintax
- 7.4.2.6. Auto Sentences
- 7.4.2.7. Auto Paragraphs
- 7.4.2.8. Auto New Paragraphs
- 7.4.2.9. Auto Sentence Trees
- 7.4.2.10. Use Only Synonyms
- 7.4.2.11. Reorder Paragraphs
- 7.4.2.12. Text with spintax?
- 7.4.2.13. Test Spinning Options
- 7.4.2.14. Check API Usage Statistics
- 7.4.3. Türkçe Spin API Options
- 7.5. SEO Tab
- 7.6. Notifications Tab
- 7.7. Advanced Tab
- 7.1. Scheduling Tab
- 8. Tester Page
- 9. Dashboard Page
- 10. Tools Page
- 11. License Settings Page
- 12. Common Options
- 12.1. Selector and Attribute Setting
- 12.2. Find and Replace Setting
- 12.2.1. Examples
- 12.2.1.1. Find and replace a word in a sentence in a case-sensitive way
- 12.2.1.2. Remove a word from a sentence
- 12.2.1.3. Find and replace a word in a sentence in a case-insensitive way
- 12.2.1.4. Find and replace by using a regular expression
- 12.2.1.5. Find and use something from the found text in the replacement
- 12.2.1.6. Replace
data-src
attribute’s value withsrc
attribute’s value
- 12.2.1. Examples
- 12.3. Options Box
- 12.4. Manipulate HTML Section
- 12.5. Unnecessary Elements Section
- 12.6. Filter Setting
- 12.6.1. Filter
- 12.6.2.
if
part (or condition part) - 12.6.3.
then
part (or action part) - 12.6.4. Condition block
- 12.6.5. Command container
- 12.6.6. Subject
- 12.6.7. Property
- 12.6.8. Command
- 12.6.9. Event of the
if
part - 12.6.10. Event of the
then
part - 12.6.11. Condition operator
- 12.6.12. Collapse filter
- 12.6.13. Enable/disable filter
- 12.6.14. Side-by-side view
- 12.6.15. Move filter up
- 12.6.16. Move filter down
- 12.6.17. Clone filter
- 12.6.18. Delete filter
- 12.6.19. Delete condition block
- 12.6.20. Move command up
- 12.6.21. Move command down
- 12.6.22. Move command out
- 12.6.23. Move command in
- 12.6.24. Clone command
- 12.6.25. Move command
- 12.6.26. Delete command
- 12.6.27. Add condition command
- 12.6.28. Add condition block
- 12.6.29. Add action command
- 12.6.30. Test subject input
- 12.6.31. Test command
- 12.6.32. Filter title/description
- 12.6.33. Command options
- 12.6.34. Summary of the
if
part - 12.6.35. Summary of the
then
part - 12.6.36. Expand/collapse command option summaries
- 12.6.37. Command summary
- 12.6.38. Enable/disable filter setting
- 12.6.39. Add new
- 12.6.40. Export
- 12.6.41. Import
- 12.7. Filter Commands
- 12.7.1. Command Subjects
- 12.7.2. Subject Properties
- 12.7.2.1. Value
- 12.7.2.2. Character length
- 12.7.2.3. Word length
- 12.7.2.4. Numeric value
- 12.7.2.5. Element attribute value
- 12.7.2.6. Element numeric attribute value
- 12.7.2.7. Count
- 12.7.2.8. Attribute value
- 12.7.2.9. Numeric attribute value
- 12.7.2.10. HTML
- 12.7.2.11. Text
- 12.7.2.12. Numeric text
- 12.7.2.13. Tag name
- 12.7.2.14. Error
- 12.7.2.15. HTTP status code
- 12.7.3. Condition Commands
- 12.7.3.1. Greater than
- 12.7.3.2. Less than
- 12.7.3.3. Greater than or equal to
- 12.7.3.4. Less than or equal to
- 12.7.3.5. Equal to
- 12.7.3.6. Not equal to
- 12.7.3.7. Starts with
- 12.7.3.8. Does not start with
- 12.7.3.9. Ends with
- 12.7.3.10. Does not end with
- 12.7.3.11. Contains
- 12.7.3.12. Does not contain
- 12.7.3.13. Is uppercase
- 12.7.3.14. Is not uppercase
- 12.7.3.15. Is lowercase
- 12.7.3.16. Is not lowercase
- 12.7.3.17. Matches regex
- 12.7.3.18. Does not match regex
- 12.7.3.19. Exists
- 12.7.3.20. Does not exist
- 12.7.3.21. Is older than
- 12.7.3.22. Is older than fixed date
- 12.7.3.23. Is newer than
- 12.7.3.24. Is newer than fixed date
- 12.7.3.25. Is crawling
- 12.7.3.26. Is recrawling
- 12.7.3.27. Is first page
- 12.7.3.28. Is error
- 12.7.3.29. Is HTML error
- 12.7.3.30. Is request error
- 12.7.4. Action Commands
- 12.7.4.1. Clear
- 12.7.4.2. Find and replace
- 12.7.4.3. Make upper case
- 12.7.4.4. Make lower case
- 12.7.4.5. Make title case
- 12.7.4.6. Make snake case
- 12.7.4.7. Make kebab case
- 12.7.4.8. Make camel case
- 12.7.4.9. Make studly case
- 12.7.4.10. Make first letter upper case
- 12.7.4.11. Make slug
- 12.7.4.12. Limit words
- 12.7.4.13. Limit characters
- 12.7.4.14. Remove empty HTML elements
- 12.7.4.15. Remove links
- 12.7.4.16. Template
- 12.7.4.17. Exchange attributes
- 12.7.4.18. Remove
- 12.7.4.19. Remove attributes
- 12.7.4.20. Clone
- 12.7.4.21. Calculate
- 12.7.4.22. Send email notification
- 12.7.4.23. Stop
- 12.7.4.24. Stop and delete the post
- 12.7.4.25. Set author
- 12.7.4.26. Set featured image
- 12.7.4.27. Set post status
- 12.7.4.28. Add tags
- 12.7.4.29. Add categories
- 12.8. Transformable Fields
- 13. Permissions
- 14. Frequently Asked Questions
- 14.1. Where is my license key?
- 14.2. Can I get content from X site?
- 14.3. Can I save custom fields/taxonomies of a plugin/theme?
- 14.4. How to change the domain registered to my license
- 14.5. In how many domains can I use my license?
- 14.6. I cannot crawl sites that use SSL (https)
- 14.7.
embed
,script
, andiframe
codes are not shown properly, or not shown at all - 14.8. How to change PHP version and enable
mbstring
extension? - 14.9. How to remove links from images
- 14.10. Automatic crawling works only when someone browses my site
- 14.11. I installed the plugin on a test site and I want to use the plugin with the same license code on another site. Can I do it?
- 14.12. How to send a support request
- 14.13. What do I do when admin and FTP login credentials are asked from me?
- 14.14. The plugin does not work
- 15. Troubleshooting
- 15.1. Checking if CRON jobs work and fixing CRON problems
- 15.2. Fixing error messages like “Parse error: syntax error…”
- 15.3. Fixing automatic crawling problems
- 15.4. Fixing invisible tab contents in setting pages
- 15.5. Fixing unsaved settings problem
- 15.6. Fixing empty plugin pages problem
- 15.7. Fixing the problem of empty pages shown after saving the settings
- 15.8. Fixing the problem of error pages shown after saving the settings
- 15.9. Fixing automatic crawling problems caused by the theme Newspaper 8 (or later)
- 15.10. Fixing cURL-related automatic crawling problems
- 15.11. Fixing duplicate post issues
- 15.12. Fixing character encoding problems
- 15.13. Manual Crawling Tool’s buttons do not work
- 15.14. Troubleshooting proxy issues
- 15.15. Fixing the problems caused by other plugins/themes
- 15.16. Debugging errors
- 16. Changelog
- 16.1. v1.12.0 - 15 May 2022
- 16.2. v1.11.0 - 11 March 2021
- 16.3. v1.10.2 - 31 January 2021
- 16.4. v1.10.1 - 25 August 2020
- 16.5. v1.10.0 - 13 February 2020
- 16.6. v1.9.0 - 13 December 2019
- 16.7. v1.8.0 - 1 January 2019
- 16.8. v1.7.0 - 22 October 2017
- 16.9. v1.6.0 - 4 March 2017
- 16.10. v1.5.1 - 7 February 2017
- 16.11. v1.4.1 - 27 January 2017
- 16.12. v1.4.0 - 26 January 2017
- 16.13. v1.3.0 - 14 January 2017
- 16.14. v1.2.0 - 30 August 2016