9.1. Manual Crawling Tool
This tool let’s you crawl one or more post URLs. If you want, you can also save the post URLs into the database so that the plugin will crawl them using your scheduling settings (See: Scheduling Tab).
The inputs of this tool are explained as follows.
Important
You should provide at least one post URL.
- Site
Select a site whose settings should be saved to save posts.
Note
Only the published sites are shown.
- Category
Select a category into which the posts will be saved.
- Post URLs (optional)
Enter the URL of the post that you want to crawl. You can enter many URLs by writing each of them into a new line.
Note
Make sure you entered full URLs. In other words, they should start with
http.- Post and Featured Image URLs (optional)
This setting lets you define post URLs with their featured image URLs. If you configured the settings under Featured Image Section of Post Tab, then you do not need to use this setting, because the featured images will be retrieved from the post. In that case you can just use Post URLs setting. This setting is just for the case where the featured images are not available inside the post page. For example, if you configured the settings under Featured Images Section of Category Tab and you did not configure the settings under Featured Image Section of Post Tab then you should use this setting, in case you want the featured images to be saved as well.
This setting has two input fields as explained below:
- Post URL
- Enter the URL of the post that should be saved
- Featured image URL
- Enter the URL of the featured image that belongs to the post URL. The plugin will save the post and assign this image to the post as its featured image.
Note
Make sure you entered full URLs. In other words, they should start with
http.- Retrieve post URLs from these category URLs (optional)
Enter the URLs of the categories from which the post URLs should be collected. This setting automatically finds post URLs that are available in the given category pages and crawls them so that you do not enter them manually. You can use this setting if you want to save all of the posts that are available inside a category page. The post URLs will be found by using the settings you defined under Category Tab.
Note
Make sure you entered full URLs. In other words, they should start with
http.- Pause after crawling this number of posts (optional)
- Enter a number. The tool will pause crawling after this number of posts are crawled. This
option is valid only when crawling now. Entering
0or leaving this empty means unlimited posts. - Maximum parallel crawling count (optional)
Enter a number. This number of posts will be crawled at the same time. You can use this setting to increase the speed of crawling. The default number is
1.Let’s say a single post is saved in
3seconds and you entered10post URLs. When this setting is set to1, the posts will be finished crawling after about30seconds. If you set this to10, crawling will be finished in about3seconds, depending on the response time of the target site.- Clear entered URLs after I click submit button (optional)
- When you check this, the URLs you have entered into the inputs will be cleared after you click one of the submit buttons.
After you configured the inputs, simply click one of the submit buttons. The submit buttons are explained as follows.
- Crawl now
The URLs you entered will be crawled one by one, as soon as you click this.
Important
Your browser needs to stay open until all URLs are finished being crawled.
- Add to the database
- The URLs you entered will be added to the database. They will be crawled using your scheduling settings (See: Scheduling Tab).
The following video shows how this tool is used.
9.1.1. URLs for Manual Crawling Section
After you click to Crawl now button, you can follow the status of the URLs in this section. All of the URLs will be shown here. After the URLs are crawled, the link of the posts from your site will be displayed. You can see how many posts are in the queue, how many of them are being saved, and pause/continue crawling.
You can click to button to pause the crawling of the URLs in the queue. Note that the URLs that are in progress will not be stopped. The button will pause crawling for the URLs that are in queue, not being crawled. After you click button, it will be turned into button. You can click to button to continue the crawling of the URLs that are in the queue.
The URLs are shown as rows. A URL row displays the following information about the URL:
- Status
- The status of the URL as already crawled, being crawled or not crawled.
- Site
- The name of the site whose settings are used to crawl the URL
- Category
- The category into which the post will be saved into in your site
- Image
- Featured image of the post. If a featured image is not provided when adding the URL, then no image will be shown here.
- Post URL
- URL of the post. You can click to this URL to go to the target page.
Each URL row shown in this section has the following buttons:
- Clicking to this button will remove the URL row from this section. It does not delete the post. It just removes the URL row.
- Clicking this button will recrawl the URL.
After a URL is crawled, the results of crawling are shown just below the URL row. If there were errors, you can see them in this part, as well. You can also see the saved post’s link so that you can click to it and check if it was saved as you want.
Tip
You can click to the row that shows the URL to toggle the visibility of the results of that URL.
Tip
The URLs that are currently being saved are shown with a green background.
At the top of the URL rows, there are the following links:
- Show all results
- Clicking this link displays the results of all URL rows.
- Hide all results
- Clicking this link hides the results of all URL rows.
- Remove all
- Clicking this link removes all of the URLs from the list. The posts will not be deleted. Only the rows showing the URLs will be deleted.
At the top of this section, a list of URLs that are currently in progress of being crawled is shown. Also, you can see how many of the total URLs are already crawled.