With this Utility you can download and save interlinked pages automatically. Starting from one or several initial pages the links are followed and stored in the cache archive.
Start of Utility
Please, start the mirroring with the symbol in the main menu. You get the configuration dialog: Mirror
You also can alternatively start the mirroring with the script MM3-Utility in the folder MM3-WebAssistantProfessional/script/.
You can save a mirroring configuration in a Set. At first use the set default. With New you create a new set. Furthermore you can Rename or Delete a set with a pop-up menu.
General
Start Page
Starting from one or several initial pages the links are followed, pages are downloaded and stored in the cache archive. Enter the URL for every start page in a new line.
POST parameter
If you need the option Post, then you indicate a blank and then the parameters (key/value list) behind the URL.
Note
There are restrictions at the link following of dynamic elements like Java Skript and Flash.
Sequence
If you indicate several start pages, then these are downloaded at first (depth 1).
After this all links of the start pages are downloaded (depth 2) and so on.
Start pages can alternatively be downloaded also sequentially.
Surf Set
The parameters of the Surf Set are used with the exception of Marker and Prefetch.
An important parameter is the re-use time of files in the cache archive. Files aren't downloaded newly from the internet within their re-use time.
Cancel
Please note that you need with higher depth a larger proportion of bandwidth and hard disk space. Often this is not foreseeable, how many HTML pages are actually tracked and downloaded. Depending on application you should limit the links to be followed regarding to the depth. You can in addition use a filter.
You can indicate the following abort conditional:
- Depth
- Link depth of following links
- HTML pages
- Maximum number of HTML pages to be downloaded.
- Volume
- Maximum size of the date volume to be downloaded.
- Time
- Maximum time for the duration of link following in minutes.
Waiting time
Some servers break off downloading files automatically. The files are requested in much faster consequence as this is carried out at the surfing. You therefore insert a waiting time, at which you let best vary these between a minimal and maximum waiting time.
Report
You can create a report on the followed links. You can select between a report with all links or only with the links at which a file was downloaded.
Log
All links followed up are logged per default.
| Log with | In a HTML page … |
|---|---|
| Contained resources | all contained resources |
| Not followed links | all not followed links |
In the summary you can log links not followed in addition.
Note
Pages and resource occurring repeatedly are logged only once.
Following Links
If too many pages are downloaded at a mirroring, you can prevent this with a filter so. You can define in a filter: domain, directory, file name, file or MIME types. With the additional parameter filterState the effect of the filter is controlled by the link following.
| Follow | Following or excluding link |
|---|---|
| Yes | Follow: If the link includes the character pattern of the filter. |
| No | Exclude: If the link includes the character pattern of the filter. |
| --- | Disable: Filter isn't use. |
If both, Follow and Exclude, meet a link, then the link will be excluded and the page won't be downloaded.
The detailed structure of the filter is described: Filter for Following Links
| Follow | Domain | Path | File | Typ |
|---|---|---|---|---|
| Yes | / | |||
| No | / | pdf application/pdf | ||
| No | /private/ |
Only pages are downloaded from the domain Proxy-Offline-Browser.com, however no pdf documents and pages from the folder /private/ and its subdirectories.
Note
Please select, whether the link has to be followed, excluded or enquired generally.
With Enquire, you can indicated when mirroring interactively whether is a link to follow or to exclude.
With Generate filters you can generate the required filters from your start pages (URLs). An additional pattern still can specify the filters or you can manually or with interaction postprocess the filters.
If e.g. you liked to exclude images in a page, you must use the Filter for Advertising.
Log
Log all pages which were excluded from the link following. You hereby can check the setting of the filter.
Cookie
Cookies can be accepted or blocked depending on filter setting. This supports the protection of data privacy. You also can prevent the representation of history (breadcrumb trail) on a HTML page with that, however.
| Accept | Accepting or blocking cookie |
|---|---|
| Yes | Accept: If the URL includes the character pattern of the filter. |
| No | Block: If the URL includes the character pattern of the filter. |
| --- | Disable: Filter isn't use. |
If both, Accept and Bock, meet a cookie, then the cookie will be blocked.
The detailed structure of the filter is described: Filter for Cookie
| Accept | Domain | Path |
|---|---|---|
| No | / |
All Cookies are blocked.
Log
All accepted cookies are logged. You hereby can check the setting of the filter.
Authentication
For a password protected web site the user name and the password can be saved. Depending on web site the login is carried out by two different procedures.
Request by the HTTP header
If the login by the header of the HTTP protocol is carried out, then you can enter user name and password here.
| Login | Access data for a web site |
|---|---|
| Yes | Login: Logging in for this URL with the access data. |
| No | Block: Not logging in for this URL any more. |
| --- | Disable: Filter isn't use. |
You can in addition indicate whether the password shall be saved at the call of a protected page.
Passwords can be protected with a master password. The master password has to be entered at every start of the Utility Mirror.
Start of the mirror
After you have filled out the required configuration or selected a set then you start the mirroring. In a dialog the mirroring is logged. A summary is displayed at the end of the mirroring or at the cancel. A report can be displayed in the browser.
Display of the reports
You reach the reports from the local sitemap (or direct by URL: http://127.0.0.1:8080/mirror/). You select the desired report from a list. A report contains the followed links of a mirroring.
You can adjust the display:
- Number of followed links per report page.
- Sorting sequence of the followed links.
- Representation of the report page
You can surf in a report.