How to download a websites source files

How to download a websites source files

how to download a websites source files

www.youtube.com › watch. Right-click on the page you want to view the. Simple cloud-based Website Downloader with nothing to install or configure - Download the complete source code of any website (including all assets).

The purpose: How to download a websites source files

GOPRO QUIK WINDOWS 10 DOWNLOAD RAW 527
STENCIL D FREE DOWNLOAD 244
DOWNLOAD GIF FILES SAVED IN GOOGLE MAPS HISTORY 952
DOWNLOAD PDF 5348-700 COLEMAN LANTERN MANUAL 993
ULEAD VIDEO STUDIO 8.0 SE DVD FREE DOWNLOAD 804

How to download the source code (JS,CSS and images) of a website through its URL (Web Scraping) with Node.js

Disclaimer

In no case shall we (Our Code World) or the developer of this module be liable for direct, indirect, special or other consequential damages resulting from the use of this module or from the web pages downloaded with this module. Use it at your own risk.

And with our disclaimer we don't talk about that you computer will explode by using the module that we'll use to copy a website. We only warn that this script should not be used for illegal activities (like, fake a website and expose it in another web domain), but learning more about Node.js and web development. 

Having said that, have you ever seen an awesome website with some awesome gadget or widget that you absolutely want to have or learn how to do but you don'f find an open source library that does that? Because as first step, that's what should do as first, look for an open source library that creates that awesome gadget and if it exists, implement it in your own project. If you don't find it, then you can use the chrome dev tools to inspect the element and see superficially how it works and how you could create it by yourself. However if you're not so lucky or you don't have the skills to copy a feature through the dev tools, then you still have a change to do it.

What would be better than having the entire code that creates the awesome widget and edit it as you want (thing that will help you to understand how the widget works). That is precisely what you're going to learn in this article, how to download an entire website through its URL with Node.js using a web scraper. Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

Requirements

To download all the resources from a website, we are going to use the website-scraper module. This module allows you to download an entire website (or single webpages) to a local directory (including all the resources css, images, js, fonts etc.). 

Install the module in your project executing the following command in the terminal:

Note

Dynamic websites (where content is loaded by js) may be saved not correctly because website-scraper doesn't execute js, it only parses http responses for html and css files.

Visit the official Github repository for more information here.

1. Download a single page

The scrape function returns a Promise that makes requests to all the providen urls and saves all files found with sources to directory. The resources will be organized into folders according to the type of resources (css, images or scripts) inside the providen directory path. The following script will download the homepage of the node.js website:

Save the previous script in a js file (script.js) and then execute it with node using . Once the script finishes, the content of the folder will be:

And the file from a web browser will look like:

All the scripts, style sheets were downloaded and the website works like a charm. Notice that the only error shown in the console, is due to the analytics script from Google that you should obviously remove from the code manually.

2. Download multiple pages

If you're downloading multiple pages of a website, you should provide them simultaneously in the same script, scraper is smart enough to know that a resource shouldn't be downloaded twice (but only if the resource has been already downloaded from the same website in another page) and it will download all the markup files but not the resources that already exist.

In this example, we are going to download 3 pages of the node.js website (index, about and blog) specified in the property. The content will be saved in the node-website folder (where the script is executed), if it doesn't exists it will be created. To be more organized, we are going to sort out every type of resources manually in different folders respectively (images, javascript, css and fonts). The sources property specifies with an array of objects to load, specifies selectors and attribute values to select files for loading.

This script is useful if you want specifically some web pages:

3. Recursive downloads

Imagine that you don't need only specific web pages from a website, but all the pages of it. A way to do it, is to use the previous script and specify manually every URL of the website that you can get to download it, however this can be counterproductive because it will take a lot of time and you will probably overlook some URLs. That's why Scraper offers the recursive download feature that allows you to follow all the links from a page and the links from that page and so on. Obviously, that would lead to a very very long (and almost infinite) loop that you can limit with the max allowed depth ( property):

The previous script should download more pages:

Filter external URLs

As expected in any kind of website, there will be external URLs that don't belong to the website that you want to copy. To prevent that those pages are downloaded too, you can filter it only if the URL matches with the one you use:

That should decrease in our example the number of downloaded pages:

4. Download an entire website

Note

This task require much time, so be patient.

If you want to download an entire website, you can use the recursive download module and increase the max allowed depth to a reasonable number (in this example not so reasonable with 50, but whatever): 

Final recommendations

If the CSS or JS code of the website are minified (and probably all of them will be), we recommend you to use a beautifier mode for the language (cssbeautify for css or js-beautify for Javascript) in order to pretty print and make the code more readable (not in the same way that the original code does, but acceptable). 

Happy coding !

Источник: https://ourcodeworld.com/articles/read/374/how-to-download-the-source-code-js-css-and-images-of-a-website-through-its-url-web-scraping-with-node-js
how to download a websites source files

How to download a websites source files - confirm. agree

How to download a websites source files

3 thoughts to “How to download a websites source files”

Leave a Reply

Your email address will not be published. Required fields are marked *