Web search engines

Web search engines
Mplungjan · CC BY-SA 4.0 · source
Name	Web search engines
Developer	Various organizations
Released	1990s–present
Programming languages	Various
Operating system	Cross-platform

Web search engines are software systems designed to carry out searches for information on the World Wide Web and related networks. They provide interfaces for users to submit queries and return ranked lists of documents, images, videos, and other resources indexed from the Internet. Major projects, companies, universities, and standards bodies have shaped their development and deployment across commercial, academic, and governmental contexts.

History

The early development of web search functionality involved projects and organizations such as CERN, Stanford University, MIT, Oxford University, and companies like Yahoo! and AltaVista that emerged in the 1990s. Key milestones include index and crawler systems from teams at Digital Equipment Corporation, academic research at University of California, Berkeley, and algorithmic breakthroughs associated with researchers at Stanford University and SUN Microsystems labs. Commercialization accelerated with entrants including Microsoft, Google, Ask Jeeves, and Lycos, while regulatory and policy responses from entities like the European Commission, Federal Trade Commission, and national parliaments influenced market structure. Later expansions involved contributions from platforms and organizations such as Amazon (company), Apple Inc., Facebook, Twitter, and initiatives at Wikimedia Foundation to integrate structured data and open content.

Architecture and Operation

The typical architecture combines front-end services developed by companies like Google, Microsoft, Baidu, and Yandex with back-end infrastructure from cloud providers such as Amazon Web Services, Google Cloud, Microsoft Azure, and data-center operators like Equinix. Components include user interfaces inspired by research at Bell Labs and Xerox PARC, query parsers and natural language systems influenced by work at IBM Research, and storage layers using technologies from Apache Software Foundation projects like Apache Hadoop and Apache Lucene. Network operations rely on protocols and standards from organizations such as the Internet Engineering Task Force, World Wide Web Consortium, and national registries like ICANN and IANA.

Search Algorithms and Ranking

Ranking algorithms evolved through contributions from academics at Stanford University, Cornell University, Massachusetts Institute of Technology, and companies like Google (PageRank), Microsoft Research (learning-to-rank), and teams at Yahoo! and Alibaba Group. Techniques incorporate probabilistic models from researchers associated with Bell Labs and AT&T Laboratories, machine learning methods influenced by work at University of Toronto and Carnegie Mellon University, and deep learning architectures researched at Google DeepMind, OpenAI, and Facebook AI Research. Evaluation and benchmarking draw on conferences and organizations such as SIGIR, ACL, NeurIPS, and ICML, while personalization and localization intersect with services from Apple Maps, Google Maps, and global platforms like Bing.

Indexing and Crawling

Large-scale crawling and indexing systems trace lineage to projects at Internet Archive, AltaVista, and research groups at University of California, Santa Cruz and University of California, Berkeley. Crawlers must negotiate access and standards from institutions like The Library of Congress, National Archives, and international bodies such as the United Nations Educational, Scientific and Cultural Organization. Data pipelines rely on storage, compression, and retrieval technologies developed in collaboration with industry partners including Oracle Corporation, IBM, and open-source communities around Linux Foundation projects. Metadata and schema efforts connect to initiatives by W3C and knowledge graph work from Wikidata and DBpedia.

Privacy, Ethics, and Regulation

Privacy debates involve stakeholders such as European Commission, United States Department of Justice, Federal Communications Commission, and advocacy groups like Electronic Frontier Foundation and Privacy International. Ethical considerations reference frameworks and reports from institutions such as Harvard University, Yale University, Stanford Law School, and standards bodies like ISO and IEEE. High-profile legal cases and regulatory interventions include actions involving Google LLC, Microsoft Corporation, Meta Platforms, Inc., and probes by competition authorities in jurisdictions such as United Kingdom, Germany, France, and China. Civil society and NGOs including Amnesty International and Human Rights Watch have engaged on content moderation, algorithmic bias, and surveillance issues.

Market and Economics

The market structure has been shaped by major firms such as Google, Microsoft, Baidu, Yandex, and regional players like Naver and Seznam.cz, with advertising ecosystems involving DoubleClick, AdSense, AdWords, Amazon Advertising, and exchanges coordinated by firms in the Interactive Advertising Bureau. Economic analyses reference work by scholars at London School of Economics, Harvard Business School, Stanford Graduate School of Business, and institutions like OECD and World Bank regarding competition, platforms, and network effects. Mergers, acquisitions, and antitrust matters have featured companies such as Yahoo!, AOL, Verizon Communications, and investment activities by firms including SoftBank and Sequoia Capital.

Category:Search engines