Wget

Wget
Name	Wget
Developer	GNU Project / community
Released	1995
Programming language	C (programming language)
Operating system	Unix-like; Microsoft Windows; macOS
Genre	Command-line interface
License	GNU General Public License

Wget is a free, command-line network downloader originally released in 1995 and maintained within the GNU Project ecosystem. It retrieves files via Hypertext Transfer Protocol and File Transfer Protocol protocols and supports robust non-interactive operation suitable for batch processing, mirroring, and automated tasks on systems such as Linux, FreeBSD, and Microsoft Windows. Widely used by administrators, archivists, and developers, it integrates into workflows alongside tools like cron (software), systemd, and Git.

History

Wget was created in 1995 by Hrvoje Nikšić while contributing to the GNU Project and emerged during an era shaped by projects such as the World Wide Web Consortium, the rise of Apache HTTP Server, and the expansion of Linux distributions like Debian and Red Hat Linux. Its development reflects influences from network utilities including ftp (protocol), telnet, and cURL, and it has been packaged for operating systems such as NetBSD, OpenBSD, and Solaris (operating system). Over time, stewardship involved contributors associated with communities around GNU Savannah, SourceForge, and later GitHub, aligning releases with the practices of projects like Perl (programming language) and Python (programming language) ecosystems.

Features

Wget implements retrieval via Hypertext Transfer Protocol and File Transfer Protocol and supports recursive retrieval for mirroring sites, following conventions akin to those used by Robots Exclusion Standard and employed by projects such as the Internet Archive and Project Gutenberg. It offers persistent retries on network failure comparable to resilience goals in TCP/IP stacks and supports HTTP/1.1 features, proxies common to Squid (software) and Apache Traffic Server, and authentication methods used by Kerberos (protocol) and OAuth 2.0 integrations in other systems. Wget’s recursion, timestamping, and directory structure preservation align with mirror strategies used by rsync and Mirror (computing) operations. Platforms often integrate it with package managers such as APT (software) and Homebrew (package manager).

Usage

Typical invocation patterns resemble utilities like tar (computing), grep, and sed (Unix), enabling scripting in environments utilizing Bash (Unix shell), Zsh, and PowerShell on Windows PowerShell. Administrators schedule jobs with cron (software), systemd, and Windows Task Scheduler to automate downloads for backup workflows similar to those using Bacula or rsnapshot. Use cases include mirroring websites for offline access paralleling initiatives by OpenStreetMap contributors, collecting datasets for research projects affiliated with institutions like MIT or Stanford University, and batch-fetching assets for continuous integration systems such as Jenkins and Travis CI.

Options and Configuration

Wget exposes command-line options akin to flags in GCC and Make (software), with parameters for recursion depth, acceptance/rejection lists, and header customization similar to features provided by curl (tool). Configuration can be set in files in user environments like /etc and home directories comparable to dotfiles for Vim (text editor) and Emacs. Options support proxy configuration used by Squid (software), certificate handling interoperable with OpenSSL and GnuTLS, and conditional requests leveraging ETag and Last-Modified semantics as practiced by web servers such as Nginx and Apache HTTP Server.

Security and Limitations

Security considerations mirror concerns addressed by projects like OpenSSL and Let's Encrypt; Wget’s handling of TLS/SSL, certificate verification, and hostnames interacts with system libraries such as OpenSSL and GnuTLS. Limitations include lack of native support for modern browser engines like Chromium and Firefox, so it cannot execute JavaScript or process dynamic pages generated by frameworks such as React (JavaScript library), Angular (web framework), or Vue.js. This contrasts with headless browsers like Puppeteer or Selenium (software), which render client-side content. Rate limiting and polite crawling practices overlap with norms established by Robots Exclusion Standard and entities like Internet Archive and W3C.

Development and Licensing

Wget is distributed under the GNU General Public License and developed within the GNU Project community, with contributions tracked via platforms historically including SourceForge and GNU Savannah, and later repositories inspired by models used at GitHub and GitLab. Its C codebase interacts with libraries such as libidn for internationalized domain names and OpenSSL for transport security, and it follows release practices similar to those of longstanding projects like Emacs and GCC. Commercial distributions and vendors such as Red Hat and Canonical (company) include packages, and academic courses at institutions like University of California, Berkeley and Carnegie Mellon University reference it when teaching networking and systems administration.

Category:Free software Category:Command-line software Category:GNU Project