Merge branch 'rolling' into handle-reqwest-errors

.github/CODE_OF_CONDUCT.md

@@ -0,0 +1,85 @@
+# Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community and as such we strongly believe in **give respect and take respect** policy.
+
+# What We Strive At 
+
+* **Be patient.**
+* **Be welcoming and be friendly**: We strive to be a community that welcomes, supports and remain friendly to people of all backgrounds and identities. This includes, but is not limited to members of any race, ethnicity, culture, national origin, colour, immigration status, social and economic class, educational level, sex, sexual orientation, gender identity and expression, age, size, family status, political belief, religion, and mental and physical ability.
+* **Be considerate**: Your work will be used by other people, and you in turn will depend on the work of others. Any decision you take will affect users and colleagues, and you should take those consequences into account when making decisions. Remember that we're a world-wide community, so you might not be communicating in someone else's primary language.
+* **Be respectful**:  Not all of us will agree all the time, but disagreement is no excuse for poor behavior and poor manners. We might all experience some frustration now and then, but we cannot allow that frustration to turn into a personal attack. It’s important to remember that a community where people feel uncomfortable or threatened is not a productive one.
+* **Be careful in the words that you choose**: we are a community of professionals, and we conduct ourselves professionally. Be kind to others. Do not insult or put down other participants. Harassment and other exclusionary behavior aren't acceptable. This includes, but is not limited to:
+  * Violent threats or language directed against another person.
+  * Discriminatory jokes and language.
+  * Using political talks and political orientated views. 
+  * Posting sexually explicit or violent material.
+  * Posting (or threatening to post) other people's personally identifying information ("doxing").
+  * Personal insults, especially those using racist or sexist terms.
+  * Unwelcome sexual attention.
+  * Advocating for, or encouraging, any of the above behavior.
+  * Humor is acceptable but should not be done to harass, demean or to insult others.
+  * Repeated harassment of others. In general, if someone asks you to stop, then stop.
+  * Using overtly sexual aliases or other nicknames that might detract from a friendly, safe and welcoming environment for all.
+  * Using bad words or cursing.
+* **When we disagree, try to understand why**: Disagreements, both social and technical, happen all the time. It is important that we resolve disagreements and differing views constructively. Remember that we’re different. The strength of our community comes from its diversity, people from a wide range of backgrounds. Different people have different perspectives on issues. Being unable to understand why someone holds a viewpoint doesn’t mean that they’re wrong. Don’t forget that it is human to err and blaming each other doesn’t get us anywhere. Instead, focus on helping to resolve issues and learning from mistakes.
+* **Treat everyone equally:** we are community of mature people and maturity in the way we act, behave and treat others and as such we treat others as our brothers and sisters. 
+
+# Scope 
+
+The scope of the conduct of code is not limited to one individual or few individuals. This code of conduct applies for all be it the maintainer, the developer or any role assigned to the project or involved in any other way to the project. This code of conduct is not to provide privelages of one over the other. Any failure to enforce, act, using the code of conduct for your benefit or to evade certain situations or findings way to evade this conduct or a failure to provide a safe environment for others under the umberalla of this code of conduct will be considered a clear act of violation.
+
+# Violation 
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+1. **Correction**
+
+**Community Impact:** Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence:** A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+2. **Warning**
+
+**Community Impact:** A violation through a single incident or series of actions.
+
+**Consequence:** A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+3. **Temporary Ban**
+
+**Community Impact:** A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence:** A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+4. **Permanent Ban**
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+
+# Reporting Issues
+
+If you experience or witness unacceptable behavior—or have any other concerns—please report it by contacting the community leader via [mustafadhuleb53@gmail.com](mustafadhuleb53@gmail.com). All reports will be handled with discretion. In your report please include:
+
+- Your contact information.
+- Names (real, nicknames, or pseudonyms) of any individuals involved. If there are additional witnesses, please
+include them as well. Your account of what occurred, and if you believe the incident is ongoing. If there is a publicly available record (e.g. a mailing list archive or a public IRC logger), please include a link.
+- Any additional information that may be helpful.
+
+Anyone asked to stop unacceptable behavior is expected to comply immediately. If an individual engages in unacceptable behavior, We will act according to the [violations rules](#violation) as stated above.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html, [Rust Code of Conduct](https://www.rust-lang.org/policies/code-of-conduct) and [Twitter's Code of Conduct](https://github.com/twitter/.github/blob/main/code-of-conduct.md?plain=1)
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

.github/workflows/contributors.yml

@@ -0,0 +1,47 @@
+name: Contributors List
+
+on:
+  workflow_dispatch:
+  
+  schedule:
+    - cron: "0 1 * * *"
+
+jobs:
+  contributors:
+    permissions:
+      contents: write
+      pull-requests: write
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@8f4b7f84864484a7bf31766abe9204da3cbe65b3 # v3.5.0
+        with:
+          fetch-depth: 0
+          ref: ${{ github.event.repository.default_branch }}
+          
+      - name: Update contributors list
+        uses: wow-actions/contributors-list@b9e91f91a51a55460fdcae64daad0cb8122cdd53 # v1.1.0
+        with:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          svgPath: images/contributors_list.svg
+          round: true
+          includeBots: false
+          noCommit: true
+      
+      - name: Commit & PR
+        uses: peter-evans/create-pull-request@38e0b6e68b4c852a5500a94740f0e535e0d7ba54 # v4.2.4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          add-paths: .github/assets/CONTRIBUTORS.svg
+          commit-message: 'chore: update contributors-list'
+          committer: GitHub <noreply@github.com>
+          author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
+          signoff: false
+          branch: workflow/update-contributors-list
+          base: main
+          delete-branch: true
+          title: 'chore: update contributors-list'
+          body: |
+            Automated update to `images/contributors_list.svg`

.github/workflows/hello.yml

@@ -0,0 +1,18 @@
+name: Welcome first time contributors 
+
+on:
+  pull_request_target:
+    types:
+      - opened
+ 
+jobs:
+  welcome:
+    name: Welcome
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/first-interaction@v1
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          pr-message: |-
+            Congrats on making your first Pull Request and thanks for taking the time to improve Websurfx! ❤️!
+            Say hello by joining the conversation in our [Discord](https://discord.gg/SWnda7Mw5u)

.github/workflows/labels.yml

@@ -0,0 +1,22 @@
+name: Import open source standard labels
+
+on:
+  push:
+    branches:  
+      - rolling 
+
+jobs:
+  labels:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '14'
+    - uses: EddieHubCommunity/gh-action-open-source-labels@main
+      with:
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        owner-name: ${{ github.repository_owner }}
+        repository-name: ${{ github.event.repository.name }}
+        force: false 

.github/workflows/rust.yml

@@ -21,6 +21,20 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+    - run: rustup toolchain install stable --profile minimal
+    - uses: Swatinem/rust-cache@v2
+      with:
+        prefix-key: ""
+        shared-key: ""
+        key: ""
+        env-vars: ""
+        workspaces: ""
+        cache-directories: ""
+        cache-targets: ""
+        cache-on-failure: ""
+        cache-all-crates: ""
+        save-if: ""    
+    - uses: actions/checkout@v3
     - run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}
     - name: Build
       run: cargo build --verbose

.github/workflows/rust_format.yml

@@ -0,0 +1,35 @@
+name: Rust format and clippy checks
+on:
+  push:
+    branches:
+      - "**"
+  pull_request:
+    branches:
+      - "rolling" 
+
+jobs:
+  check:
+    name: Rust project
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Install minimal stable with clippy and rustfmt
+        uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: stable
+          components: rustfmt, clippy
+      - name: Format
+        uses: actions-rs/cargo@v1
+        with:
+          command: fmt
+          args: -- --check
+      - name: Clippy
+        uses: actions-rs/cargo@v1
+        with:
+          command: clippy
+          args: --all-features --all-targets --all
+      - name: Run cargo check
+        uses: actions-rs/cargo@v1
+        with:
+          command: check

.github/workflows/stale.yml

@@ -0,0 +1,27 @@
+# This workflow warns and then closes issues and PRs that have had no activity for a specified amount of time.
+#
+# You can adjust the behavior by modifying this file.
+# For more information, see:
+# https://github.com/actions/stale
+name: Mark stale issues and pull requests
+
+on:
+  schedule:
+  - cron: '30 1 * * *'
+
+jobs:
+  stale:
+
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+
+    steps:
+    - uses: actions/stale@v5
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        stale-issue-message: 'Stale issue message'
+        stale-pr-message: 'Stale pull request message'
+        stale-issue-label: 'no-issue-activity'
+        stale-pr-label: 'no-pr-activity'

.gitignore

@@ -1 +1,3 @@
 /target
+
+dump.rdb

CONTRIBUTING.md

@@ -1,34 +1,54 @@
-# Things to Consider Before Contributing
+# What You Can Contribute To?
 
-## Knowledge Required
+## Documentation/Wiki
 
--   Rust basics
--   Actix-web crate basics
--   Tokio crate and async/await
--   Reqwest crate basics
--   Serde and serde~json~ crate basics
--   fake~useragent~ crate basics
--   pyo3/hlua/rlua crates basics
+Found a typo, or something that isn't as clear as it could be? Maybe I've missed something off altogether, or you hit a roadblock that took you a while to figure out. Edit the [docs](./docs/) to add to or improve the documentation. This will help future users get Websurfx up and running more easily.
+
+## Readme
+
+Found a typo, or the Readme is not very clear as it should be?. Consider Submitting a Pull request to the [Readme](https://github.com/neon-mmd/websurfx/blob/master/README.md) to add to or improve the Readme. This will help future users to better understand the project more clearly.
+
+## Help Improve Github Actions
+
+Know how to fix or improve a github action?. Consider Submitting a Pull request to help make automation and testing better.
+
+## Source Code
+
+You should know atleast one of the things below to start contributing:
+
+- Rust basics
+- Actix-web crate basics
+- Tokio crate and async/await
+- Reqwest crate basics
+- Serde and serde_json crate basics
+- Scraper crate basics
+- Frontend (handlebars, css and js).
+- Fake useragent crate basics
+- pyo3/hlua/rlua crates basics
+
+## Report a Bug/Issue
+
+If you've found a bug, then please consider raising it as an issue [here](https://github.com/neon-mmd/websurfx/issues). This will help me know if there's something that needs fixing. Try and include as much detail as possible, such as your environment, steps to reproduce, any console output and maybe an example screenshot or recording if necessary.
+
+## Spread the word
+
+Websurfx is still a relatively young project, and as such not many people know of it. It would be great to see more users, and so it would be awesome if you could consider sharing with your friends or on social platforms.
 
 ## Guidelines
 
--   Please be patient.
+- Please be patient.
+- Treat everyone with respect -- \"give respect and take respect.\"
+- Document your code properly with Rust coding conventions in mind.
+- Provide a brief description of the changes you made in the pull request.
+- Provide an appropriate header for the pull request.
 
--   Treat everyone with respect -- \"give respect and take respect.\"
+## Join the discussion
 
--   Document your code properly with Rust coding conventions in mind.
+We have a [Discord](https://discord.gg/SWnda7Mw5u) channel, feel free to join and share your ideas and ask questions about the project, we would be glad to hear you out.
 
--   Provide a brief description of the changes you made in the pull
-    request.
+# Where To Contribute?
 
--   Provide an appropriate header for the pull request.
+The _rolling branch_ is where we intend all contributions should go.
 
-    **NOTE:** The rolling branch is where all contributions should go.
-    In other words, it is the working branch for this project.
 
-We appreciate any contributions and suggestions to help improve the
-Websurfx project. Please keep in mind the above requirements and
-guidelines before submitting a pull request and also if you have any
-doubts/concerns/questions about the project, its source code or anything
-related to the project than feel free to ask by opening an
-\[issue\](<https://github.com/neon-mmd/websurfx/issues>).
+We appreciate any contributions whether be of any size or topic and suggestions to help improve the Websurfx project. Please keep in mind the above requirements and guidelines before submitting a pull request and also if you have any doubts/concerns/questions about the project, its source code or anything related to the project than feel free to ask by opening an [issue](https://github.com/neon-mmd/websurfx/issues) or by asking us on our [Discord](https://discord.gg/SWnda7Mw5u) channel.

Cargo.lock

@@ -3325,6 +3325,7 @@ dependencies = [
  "handlebars",
  "log",
  "md5",
+ "once_cell",
  "rand 0.8.5",
  "redis",
  "reqwest 0.11.18",

Cargo.toml

@@ -21,3 +21,4 @@ rlua = {version="*"}
 redis = {version="*"}
 md5 = {version="*"}
 rand={version="*"}
+once_cell = {version="*"}

README.md

@@ -1,66 +1,47 @@
-<h1 align="center">Websurfx</h1>
+<h1 align="center">
+  <img src="./images/websurfx_logo.png" alt="websurfx logo" align="center" />
+</h1>
 <p align="center">
-  <i>
-    A modern-looking, lightning-fast, privacy-respecting, secure
-    <a href="https://en.wikipedia.org/wiki/Metasearch_engine"
-      >meta search engine</a
-    >
-    (pronounced as websurface or web-surface /wɛbˈsɜːrfəs/.) written in Rust. It
-    provides a fast and secure search experience while respecting user
-    privacy.</i
-  >
-  <br />
-  <br />
-  <img align="center" width="500px" src="./images/intro.png" />
-  <br />
-  <br />
-  <b align="center"><a href="README.org">Readme</a></b> |
-  <b><a href="https://discord.gg/k4GH2TnZ">Discord</a></b> |
+  <b align="center"><a href="README.md">Readme</a></b> |
+  <b><a href="https://discord.gg/SWnda7Mw5u">Discord</a></b> |
   <b><a href="https://github.com/neon-mmd/websurfx">GitHub</a></b> |
-  <b><a href="https://github.com/neon-mmd/websurfx/wiki">Documentation</a></b>
+  <b><a href="./docs/README.md">Documentation</a></b>
   <br /><br />
-  <a href="./LICENSE">
-    <img
-      alt="GitHub"
-      src="https://img.shields.io/github/license/neon-mmd/websurfx?style=flat-square"
-    />
-  </a>
-  <a href="https://github.com/neon-mmd/websurfx/stargazers">
-    <img
-      alt="GitHub Repo stars"
-      src="https://img.shields.io/github/stars/neon-mmd/websurfx?style=flat-square"
-    />
-  </a>
-  <a href="https://github.com/neon-mmd/websurfx/forks">
-    <img
-      alt="GitHub forks"
-      src="https://img.shields.io/github/forks/neon-mmd/websurfx?style=flat-square"
-    />
-  </a>
   <a href="#">
     <img
       alt="GitHub code size in bytes"
       src="https://img.shields.io/github/languages/code-size/neon-mmd/websurfx?style=flat-square"
     />
   </a>
-  <a href="https://github.com/neon-mmd/websurfx/issues">
+  <a href="https://github.com/neon-mmd/websurfx/actions">
     <img
-      alt="GitHub issues"
-      src="https://img.shields.io/github/issues/neon-mmd/websurfx?style=flat-square"
+      alt="GitHub Workflow Status"
+      src="https://img.shields.io/github/actions/workflow/status/neon-mmd/websurfx/rust.yml?style=flat-square"
     />
   </a>
-  <a href="https://github.com/neon-mmd/websurfx/pulls">
-    <img
-      alt="GitHub pull requests"
-      src="https://img.shields.io/github/issues-pr/neon-mmd/websurfx?style=flat-square"
+  <a href=""
+    ><img
+      alt="Maintenance"
+      src="https://img.shields.io/maintenance/yes/2023?style=flat-square"
     />
   </a>
-  <a href="https://github.com/neon-mmd/websurfx/actions">
+  <a href="https://gitpod.io/#https://github.com/neon-mmd/websurfx">
     <img
-      alt="GitHub Workflow Status"
-      src="https://img.shields.io/github/actions/workflow/status/neon-mmd/websurfx/rust.yml?style=flat-square"
+      alt="Gitpod"
+      src="https://img.shields.io/badge/Gitpod-Ready--to--Code-blue?logo=gitpod"
     />
   </a>
+  <br />
+  <br />
+  <i>
+    A modern-looking, lightning-fast, privacy-respecting, secure
+    <a href="https://en.wikipedia.org/wiki/Metasearch_engine"
+      >meta search engine</a
+    >
+    (pronounced as websurface or web-surface /wɛbˈsɜːrfəs/.) written in Rust. It
+    provides a quick and secure search experience while maintaining user
+    privacy.</i
+  >
 </p>
 
 <details>
@@ -68,8 +49,8 @@
   <p>
 
 - **Getting Started**
-    - [🌈 Features](#features-)
     - [🔭 Preview](#preview-)
+    - [🚀 Features](#features-)
     - [🛠️ Installation and Testing](#installation-and-testing-)
     - [🔧 Configuration](#configuration-)
 - **Feature Overview**
@@ -89,17 +70,6 @@
   </p>
 </details>
 
-# Features 🌈 
-
-- ⛔ Proper NSFW blocking
-- 🎨 High level customizability with 9 colorchemes provided by default with a simple theme, also supporting creation of your custom themes and colorschemes very quickly and easily
-- 🔍 Advanced image search
-- 🔐 Fast, private and secure
-- 🆓 100% free and open source
-- 🌈 and lots more...
-
-**[⬆️ Back to Top](#websurfx)**
-
 # Preview 🔭 
 
 ## Main Page
@@ -114,31 +84,33 @@
 
 <img align="center" src="./images/404_error_page.png" />
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
-# Installation and Testing 🛠️ 
+# Features 🚀 
 
-> For full setup instructions, see: [**Installation**]()
+- 🎨 High level of customizability with nine color schemes provided by default with a simple theme, also supporting the creation of your custom themes and colorschemes very quickly and easily
+- 🔐 Fast, private, and secure
+- 🆓 100% free and open source
+- 💨 Ad-free and clean results
+- 🌟 and lots more...
 
-To get started with Websurfx, clone the repository, edit the config file which is located in the `websurfx`{.verbatim} directory and install redis server by following the instructions located [here](https://redis.io/docs/getting-started/) and then run the websurfx server and redis server using the following commands:
+**[⬆️ Back to Top](#--)**
 
-``` shell
-git clone https://github.com/neon-mmd/websurfx.git
-cd websurfx
-cargo build
-redis-server -p 8082 &
-./target/debug/websurfx
-```
+# Installation and Testing 🛠️ 
+
+> For full setup instructions, see: [**Installation**](./docs/installation.md)
+
+Before you can start building `websurfx`, you will need to have `Cargo` installed on your system. You can find the installation instructions [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
 
-If you want to use the rolling/edge branch, run the following commands instead:
+To get started with Websurfx, clone the repository, edit the config file, which is located in the `websurfx/` directory, and install the Redis server by following the instructions located [here](https://redis.io/docs/getting-started/) and then run the websurfx server and redis server using the following commands:
 
 ``` shell
 git clone https://github.com/neon-mmd/websurfx.git
 cd websurfx
-git checkout rolling
-cargo build
-redis-server -p 8082 &
-./target/debug/websurfx
+git checkout stable
+cargo build -r
+redis-server --port 8082 &
+./target/release/websurfx
 ```
 
 Once you have started the server, open your preferred web browser and navigate to <http://127.0.0.1:8080> to start using Websurfx.
@@ -146,91 +118,91 @@ Once you have started the server, open your preferred web browser and navigate t
 > **Warning**
 > Please be aware that the project is still in the testing phase and is not ready for production use.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Configuration 🔧 
 
-> For full configuration instructions, see: [**Configuration**]()
+> For full configuration instructions, see: [**Configuration**](./docs/configuration.md)
 
 Websurfx is configured through the config.lua file, located at `websurfx/config.lua`.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Theming 🎨 
 
-> For full theming and customization instructions, see: [**Theming**]()
+> For full theming and customization instructions, see: [**Theming**](./docs/theming.md)
 
-Websurfx comes with several themes and colorschemes by default which you can apply and edit through the config file. Support for custom themes and colorschemes using css and develop your own unique-looking website.
+Websurfx comes with several themes and color schemes by default, which you can apply and edit through the config file. Supports custom themes and color schemes using CSS, allowing you to develop your own unique-looking website.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Multi-Language Support 🌍
 
 > **Note**
-> Currently, we do not support other languages but in future we would start accepting contributions regarding language support because we believe that language should not be a barrier for entry.  
+> Currently, we do not support other languages, but in the future, we will start accepting contributions regarding language support because we believe that language should not be a barrier to entry.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # System Requirements 📊
 
-At present, we only support x86_64 architecture systems but will love to have contributions to extend to other architectures as well.
+At present, we only support x86_64 architecture systems, but we would love to have contributions that extend to other architectures as well.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # FAQ (Frequently Asked Questions) 🗨️
 
 ## Why Websurfx?
 
-The main goal of the Websurfx project is to provide a fast, secure, and privacy-focused [meta search engine](https://en.wikipedia.org/wiki/Metasearch_engine). While there are many meta search engines available, they do not always guarantee the security of their search engine, which is essential for ensuring privacy. For example, memory vulnerabilities can leak private or sensitive information, which is never good. Websurfx is written in Rust, which guarantees memory safety and eliminates such problems. Many meta search engines also lack key features such as advanced image search, which is required by many graphic designers, content creators, and others. Websurfx aims to provide these features and others, such as proper NSFW blocking, to improve the user experience.
+The primary purpose of the Websurfx project is to create a fast, secure, and privacy-focused meta-search engine. While there are numerous meta-search engines available, not all of them guarantee the security of their search engine, which is critical for maintaining privacy. Memory flaws, for example, can expose private or sensitive information, which is never a good thing. Also, there is the added problem of Spam, ads, and unorganic results which most engines don't have the full-proof answer to it till now but with Websurfx I finally put a full stop to this problem, also, Rust is used to write Websurfx, which ensures memory safety and removes such issues. Many meta-search engines also lack important features like advanced picture search, which is required by many graphic designers, content providers, and others. Websurfx attempts to improve the user experience by providing these and other features, such as proper NSFW blocking and Micro-apps or Quick results (like providing a calculator, currency exchanges, etc in the search results).
 
 ## Why AGPLv3?
 
-Websurfx is released under the **AGPLv3** license to ensure that the source code remains open and transparent. This helps to prevent the inclusion of spyware, telemetry, or other malicious code in the project. **AGPLv3** is a strong copyleft license that ensures the source code of the software remains open and available to everyone, including any modifications or improvements made to the code.
+Websurfx is distributed under the **AGPLv3** license to keep the source code open and transparent. This helps to keep malware, telemetry, and other dangerous programs out of the project. **AGPLv3** is a strong copyleft license that ensures the software's source code, including any modifications or improvements made to the code, remains open and available to everyone.
 
 ## Why Rust?
 
-Rust was chosen as the programming language for Websurfx due to its memory safety features, which can help prevent vulnerabilities and make the codebase more secure. Rust is also faster than C++, which helps to make Websurfx fast and responsive. In addition, Rust\'s ownership and borrowing system allows for safe concurrency and thread safety in the codebase.
+Rust was chosen as the programming language for Websurfx because of its memory safety features, which can help prevent vulnerabilities and make the codebase more secure. Rust is also faster than C++, which contributes to Websurfx's speed and responsiveness. Furthermore, the Rust ownership and borrowing system enables secure concurrency and thread safety in the program.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # More Contributers Wanted 📣 
 
-We are looking for more willing contributors to help grow this project.For more information on how you can contribute, check out the [project board](https://github.com/neon-mmd/websurfx/projects?query=is%3Aopen) and the [CONTRIBUTING.org](CONTRIBUTING.org) file for guidelines and rules for making contributions.
+We are looking for more willing contributors to help grow this project. For more information on how you can contribute, check out the [project board](https://github.com/neon-mmd/websurfx/projects?query=is%3Aopen) and the [CONTRIBUTING.md](CONTRIBUTING.md) file for guidelines and rules for making contributions.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Supporting Websurfx 💖
 
 > For full details and other ways you can help out, see: [**Contributing**]()
 
-If you're using Dashy and would like to help support its development, then that would be awesome! Contributions of any type, any size, are always very much appreciated, and we will appropriately credit you for your effort.
+If you use Websurfx and would like to contribute to its development, that would be fantastic! Contributions of any size or type are always welcome, and we will properly acknowledge your efforts.
 
 Several areas that we need a bit of help with at the moment are:
-- **Better and more colorchemes** - Help fix colorchemes and add other famous colorchemes.
-- **Improve evasion code for bot detection** - Help improve code related to evade ip blocking and emulate human behaviours located in everyone engine file.
+- **Better and more color schemes**: Help fix color schemes and add other famous color schemes.
+- **Improve evasion code for bot detection** - Help improve code related to evading IP blocking and emulating human behaviors located in everyone's engine file.
 - **Logo** - Help create a logo for the project and website.
-- **Docker Support** - Help write a docker compose file for the project.
-- Submit a PR to add a new feature, fix a bug, update the docs, add a theme, widget or something else.
+- **Docker Support** - Help write a Docker Compose file for the project.
+- Submit a PR to add a new feature, fix a bug, update the docs, add a theme, widget, or something else.
 - Star Websurfx on GitHub.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Documentation 📘
 
 > **Note**
-> We are willing to have any contribution regarding [documentation](https://github.com/neon-mmd/websurfx/wiki) as this helps everyone using this project. 
+> We welcome any contributions to the [documentation](./docs/) as this will benefit everyone who uses this project. 
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Roadmap 🛣️
 
 > Coming soon!! 🙂.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Contributing 🙋 
 
-Contributions are welcome from anyone. It doesn\'t matter who you are; you can still contribute to the project in your way.
+Contributions are welcome from anyone. It doesn\'t matter who you are; you can still contribute to the project in your own way.
 
 ## Not a developer but still want to contribute?
 
@@ -238,15 +210,15 @@ Check out this [video](https://youtu.be/FccdqCucVSI) by Mr. Nick on how to contr
 
 ## Developer
 
-If you are a developer, have a look at the [CONTRIBUTING.org](CONTRIBUTING.org) document for more information.
+If you are a developer, have a look at the [CONTRIBUTING.org](CONTRIBUTING.md) document for more information.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # License 📜 
 
-Websurfx is available under the [AGPLv3](LICENSE) license.
+Websurfx is licensed under the [AGPLv3](LICENSE) license.
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 # Credits 🤝 
 
@@ -260,7 +232,7 @@ We would like to thank the following people for their contributions and support:
   <br />
 </p>
 
-**[⬆️ Back to Top](#websurfx)**
+**[⬆️ Back to Top](#--)**
 
 ---
 

docs/README.md

@@ -0,0 +1,17 @@
+<h1 align="center"><img src="../images/websurfx_docs_image.png" alt="Websurfx Docs" align="center"></h1>
+
+# General
+
+- [Introduction](./introduction.md)
+- [**FAQ**](./faq.md)
+
+# Users
+
+- [Installation](./installation.md)
+- [Configuration](./configuration.md)
+- [Theming](./theming.md)
+
+# Developers
+
+- [**Contribute**](https://github.com/neon-mmd/websurfx/blob/master/CONTRIBUTING.md)
+- [**Coding style**](https://rust-lang.github.io/api-guidelines/naming.html)

docs/configuration.md

@@ -0,0 +1,58 @@
+# Configuration
+
+## Installed From Source
+
+If you have built `websurfx` from source then the configuration file will be located under project directory (codebase) at `websurfx/`
+
+> **Note**
+> If you have built websurfx with unstable/rolling/edge branch then you can copy the configuration file from `websurfx/config.lua` located under project directory (codebase) to `~/.config/websurfx/` and make the changes there and rerun the websurfx server. _This is only available from unstable/rolling/edge version_.
+
+## Installed From Package
+
+If you have installed `websurfx` using the package manager of your Linux distro then the default configuration file will be located at `/etc/xdg/websurfx/`. You can copy the default config to `~/.config/websurfx/` and make the changes there and rerun the websurfx server.
+
+Some of the configuration options provided in the file are stated below. These are subdivided into the following categories:
+
+- General
+- Server
+- Website
+- Cache
+
+# General
+
+- **logging:** An option to enable or disable logs.
+- **debug:** An option to enable or disable debug mode.
+
+## Server
+
+- **port:** Port number on which server should be launched.
+- **binding_ip_addr:** IP address on the which server should be launched.
+- **production_use:** Whether to use production mode or not (in other words this option should be used if it is to be used to host it on the server to provide a service to a large number of users). If production_use is set to true. There will be a random delay before sending the request to the search engines, this is to prevent DDoSing the upstream search engines from a large number of simultaneous requests. This is newly added option and hence is only available in the **edge version**.
+
+## Website
+
+- **colorscheme:** The colorscheme name which should be used for the website theme (the name should be in accordance to the colorscheme file name present in `public/static/colorschemes` folder).
+
+> By Default we provide 9 colorschemes to choose from these are:
+>
+> 1. catppuccin-mocha
+> 2. dracula
+> 3. monokai
+> 4. nord
+> 5. oceanic-next
+> 6. solarized-dark
+> 7. solarized-light
+> 8. tomorrow-night
+> 9. gruvbox-dark
+
+- **theme:** The theme name which should be used for the website (again, the name should be in accordance to the theme file name present in `public/static/themes` folder).
+
+> By Default we provide 1 theme to choose from these are:
+>
+> 1. simple
+
+## Cache
+
+- **redis_connection_url:** Redis connection url address on which the client should connect on.
+
+[⬅️  Go back to Home](./README.md)

docs/faq.md

@@ -0,0 +1,15 @@
+# General Questions
+
+## Why Websurfx?
+
+The primary purpose of the Websurfx project is to create a fast, secure, and privacy-focused [meta-search engine](https://en.wikipedia.org/wiki/Metasearch_engine). While there are numerous meta-search engines available, not all of them guarantee the security of their search engine, which is critical for maintaining privacy. Memory flaws, for example, can expose private or sensitive information, which is never a good thing. Also, there is the added problem of Spam, ads, and unorganic results which most engines don't have the full-proof answer to it till now but with Websurfx I finally put a full stop to this problem, also, Rust is used to write Websurfx, which ensures memory safety and removes such issues. Many meta-search engines also lack important features like advanced picture search, which is required by many graphic designers, content providers, and others. Websurfx attempts to improve the user experience by providing these and other features, such as proper NSFW blocking and Micro-apps or Quick results (like providing a calculator, currency exchanges, etc in the search results).
+
+## Why AGPLv3?
+
+Websurfx is released under the AGPLv3 license to ensure that the source code remains open and transparent. This helps to prevent the inclusion of spyware, telemetry, or other malicious code in the project. AGPLv3 is a strong copyleft license that ensures the source code of the software remains open and available to everyone, including any modifications or improvements made to the code.
+
+## Why Rust?
+
+Rust was chosen as the programming language for Websurfx due to its memory safety features, which can help prevent vulnerabilities and make the codebase more secure. Rust is also faster than C++, which helps to make Websurfx fast and responsive. In addition, Rust's ownership and borrowing system allows for safe concurrency and thread safety in the codebase.
+
+[⬅️  Go back to Home](./README.md)

docs/installation.md

@@ -0,0 +1,127 @@
+# Install From Package
+
+## Arch Linux
+
+You can install `Websurfx` through the [Aur](https://aur.archlinux.org/packages/websurfx-git), Currently we only support `Rolling/Edge` version. You can install the rolling/edge version by running the following command (using [paru](https://github.com/Morganamilo/paru)):
+
+```bash
+paru -S websurfx-edge-git
+```
+
+After installing it you can run the websurfx server by running the following commands:
+
+``` bash
+redis-server --port 8082 &
+websurfx
+```
+
+Once you have started the server, open your preferred web browser and navigate to http://127.0.0.1:8080/ to start using Websurfx.
+
+If you want to change the port or the ip or any other configuration setting checkout the [configuration docs](./configuration.md).
+
+## Other Distros 
+
+The package is currently not available on other Linux distros. With contribution and support it can be made available on other distros as well 🙂.
+
+# Install From Source
+
+Before you can start building `websurfx`, you will need to have `Cargo` installed on your system. You can find the installation instructions [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
+
+## Stable
+
+To get started with Websurfx, clone the repository, edit the config file which is located in the `websurfx` directory and install redis server by following the instructions located [here](https://redis.io/docs/getting-started/) and then build and run the websurfx server by running the following commands:
+
+```shell
+git clone https://github.com/neon-mmd/websurfx.git
+cd websurfx
+git checkout stable
+cargo build -r
+redis-server --port 8082 &
+./target/release/websurfx
+```
+
+Once you have started the server, open your preferred web browser and navigate to http://127.0.0.1:8080/ to start using Websurfx.
+
+If you want to change the port or the ip or any other configuration setting checkout the [configuration docs](./configuration.md).
+
+## Rolling/Edge/Unstable
+
+If you want to use the rolling/edge branch, run the following commands instead:
+
+```shell
+git clone https://github.com/neon-mmd/websurfx.git
+cd websurfx
+cargo build -r
+redis-server --port 8082 &
+./target/release/websurfx
+```
+
+Once you have started the server, open your preferred web browser and navigate to http://127.0.0.1:8080/ to start using Websurfx.
+
+If you want to change the port or the ip or any other configuration setting checkout the [configuration docs](./configuration.md).
+
+# Docker Deployment
+
+Before you start, you will need [Docker](https://docs.docker.com/get-docker/) installed on your system first.
+
+## Unstable/Edge/Rolling
+
+First clone the the repository by running the following command:
+
+```bash
+git clone https://github.com/neon-mmd/websurfx.git
+cd websurfx
+```
+
+After that edit the config.lua file located under `websurfx` directory. In the config file you will specifically need to change to values which is `binding_ip_addr` and `redis_connection_url` which should make the config look something like this:
+
+```lua
+-- ### General ###
+logging = true -- an option to enable or disable logs.
+debug = false -- an option to enable or disable debug mode.
+
+-- ### Server ###
+port = "8080" -- port on which server should be launched
+binding_ip_addr = "0.0.0.0" --ip address on the which server should be launched.
+production_use = false -- whether to use production mode or not (in other words this option should be used if it is to be used to host it on the server to provide a service to a large number of users)
+-- if production_use is set to true
+-- There will be a random delay before sending the request to the search engines, this is to prevent DDoSing the upstream search engines from a large number of simultaneous requests.
+
+-- ### Website ###
+-- The different colorschemes provided are:
+-- {{
+-- catppuccin-mocha
+-- dracula
+-- monokai
+-- nord
+-- oceanic-next
+-- solarized-dark
+-- solarized-light
+-- tomorrow-night
+-- }}
+colorscheme = "catppuccin-mocha" -- the colorscheme name which should be used for the website theme
+theme = "simple" -- the theme name which should be used for the website
+
+-- ### Caching ###
+redis_connection_url = "redis://redis:6379" -- redis connection url address on which the client should connect on.
+```
+
+After this run the following command to deploy the app:
+
+```bash
+docker compose up -d --build
+```
+
+This will take around 5-10 mins for first deployment, afterwards the docker build stages will be cached so it will be faster to be build from next time onwards. After the above step finishes launch your preferred browser and then navigate to `http://<ip_address_of_the_device>:<whatever_port_you_provided_in_the_config>`.
+
+## Stable
+
+For the stable version, follow the same steps as above (as mentioned for the unstable/rolling/edge version) with an addition of one command which has to be performed after cloning and changing directory into the repository which makes the cloning step as follows:
+
+```bash
+git clone https://github.com/neon-mmd/websurfx.git
+cd websurfx
+git checkout stable
+```
+
+[⬅️  Go back to Home](./README.md)

docs/introduction.md

@@ -0,0 +1,13 @@
+# Introduction
+
+A modern-looking, lightning-fast, privacy-respecting, secure [meta search engine](https://en.wikipedia.org/wiki/Metasearch_engine) (pronounced as websurface or web-surface /wɛbˈsɜːrfəs/.) written in Rust. It provides a fast and secure search experience while respecting user privacy.
+
+# Motivation
+
+Most meta search engines tend to be slow, lack high level of customization and missing many features and all of them like security as they are written in unsafe languages like python, javascript, etc which tend to open a wide variety of vulnerabilities which can also sometimes pose a threat to privacy as sometimes this can be exploited and can be used to leveraged to leak out sensitive information which is never good. 
+
+# Solution 
+
+Websurfx is a project which seeks to provide privacy, security, speed and all the features which the user wants.    
+
+[⬅️  Go back to Home](./README.md)

docs/theming.md

@@ -0,0 +1,365 @@
+# Colorschemes
+
+## Built-in
+
+By default `websurfx` comes with 9 colorschemes to choose from which can be easily chosen using the config file. To how to change colorschemes please view the [Configuration](https://github.com/neon-mmd/websurfx/wiki/configuration) section of the wiki.
+
+## Custom
+
+Creating coloschemes is as easy as it gets it requires the user to have a theme file name with the colorscheme in which every space should be replaced with a `-` (dash) and it should end with a `.css` file extension. After creating the file you need to add the following code with the `colors` you want:
+
+``` css
+:root{
+  --bg: <background color>;
+  --fg: <foreground color (text color)>;
+  --1: <color 1>;
+  --2: <color 2>;
+  --3: <color 3>;
+  --4: <color 4>;
+  --5: <color 5>;
+  --6: <color 6>;
+  --7: <color 7>;
+}
+```
+
+> **Note**
+> Please infer the theme file located under `public/static/themes` to better understand where each color is being used.
+
+**Example of `catppuccin-mocha` colorscheme:**
+
+``` css
+:root {
+  --bg: #1e1e2e;
+  --fg: #cdd6f4;
+  --1: #45475a;
+  --2: #f38ba8;
+  --3: #a6e3a1;
+  --4: #f9e2af;
+  --5: #89b4fa;
+  --6: #f5c2e7;
+  --7: #ffffff;
+}
+```
+
+# Themes
+
+## Built-in
+
+By default `websurfx` comes with 1 theme to choose from which can be easily chosen using the config file. To how to change themes please view the [Configuration](https://github.com/neon-mmd/websurfx/wiki/configuration) section of the wiki.
+
+## Custom 
+
+To write custom color scheme, it requires the user to have some knowledge of `css stylesheets`. 
+
+**Here is an example of `simple theme` (which we provide by default with the app) which will give the user a better idea on how to create a custom theme using it as a template:**
+
+### General
+``` css
+* {
+  padding: 0;
+  margin: 0;
+  box-sizing: border-box;
+}
+
+html {
+  font-size: 62.5%;
+}
+
+body {
+  display: flex;
+  flex-direction: column;
+  justify-content: space-between;
+  align-items: center;
+  height: 100vh;
+  background: var(--1);
+}
+```
+### Styles for the index page
+``` css
+.search-container {
+  display: flex;
+  flex-direction: column;
+  gap: 5rem;
+  justify-content: center;
+  align-items: center;
+}
+
+.search-container div {
+  display: flex;
+}
+```
+### Styles for the search box and search button
+``` css
+.search_bar {
+  display: flex;
+}
+
+.search_bar input {
+  padding: 1rem;
+  width: 50rem;
+  height: 3rem;
+  outline: none;
+  border: none;
+  box-shadow: rgba(0, 0, 0, 1);
+  background: var(--fg);
+}
+
+.search_bar button {
+  padding: 1rem;
+  border-radius: 0;
+  height: 3rem;
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  outline: none;
+  border: none;
+  gap: 0;
+  background: var(--bg);
+  color: var(--3);
+  font-weight: 600;
+  letter-spacing: 0.1rem;
+}
+
+.search_bar button:active,
+.search_bar button:hover {
+  filter: brightness(1.2);
+}
+```
+### Styles for the footer and header
+``` css
+header {
+  background: var(--bg);
+  width: 100%;
+  display: flex;
+  justify-content: right;
+  align-items: center;
+  padding: 1rem;
+}
+
+header ul,
+footer ul {
+  list-style: none;
+  display: flex;
+  justify-content: space-around;
+  align-items: center;
+  font-size: 1.5rem;
+  gap: 2rem;
+}
+
+header ul li a,
+footer ul li a,
+header ul li a:visited,
+footer ul li a:visited {
+  text-decoration: none;
+  color: var(--2);
+  text-transform: capitalize;
+  letter-spacing: 0.1rem;
+}
+
+header ul li a {
+  font-weight: 600;
+}
+
+header ul li a:hover,
+footer ul li a:hover {
+  color: var(--5);
+}
+
+footer div span {
+  font-size: 1.5rem;
+  color: var(--4);
+}
+
+footer div {
+  display: flex;
+  gap: 1rem;
+}
+
+footer {
+  background: var(--bg);
+  width: 100%;
+  padding: 1rem;
+  display: flex;
+  flex-direction: column;
+  justify-content: center;
+  align-items: center;
+}
+```
+### Styles for the search page
+``` css
+.results {
+  width: 90%;
+  display: flex;
+  flex-direction: column;
+  justify-content: space-around;
+}
+
+.results .search_bar {
+  margin: 1rem 0;
+}
+
+.results_aggregated {
+  display: flex;
+  flex-direction: column;
+  justify-content: space-between;
+  margin: 2rem 0;
+}
+
+.results_aggregated .result {
+  display: flex;
+  flex-direction: column;
+  margin-top: 1rem;
+}
+
+.results_aggregated .result h1 a {
+  font-size: 1.5rem;
+  color: var(--2);
+  text-decoration: none;
+  letter-spacing: 0.1rem;
+}
+
+.results_aggregated .result h1 a:hover {
+  color: var(--5);
+}
+
+.results_aggregated .result h1 a:visited {
+  color: var(--bg);
+}
+
+.results_aggregated .result small {
+  color: var(--3);
+  font-size: 1.1rem;
+  word-wrap: break-word;
+  line-break: anywhere;
+}
+
+.results_aggregated .result p {
+  color: var(--fg);
+  font-size: 1.2rem;
+  margin-top: 0.3rem;
+  word-wrap: break-word;
+  line-break: anywhere;
+}
+
+.results_aggregated .result .upstream_engines {
+  text-align: right;
+  font-size: 1.2rem;
+  padding: 1rem;
+  color: var(--5);
+}
+```
+
+### Styles for the 404 page 
+
+``` css
+.error_container {
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  width: 100%;
+  gap: 5rem;
+}
+
+.error_container img {
+  width: 30%;
+}
+
+.error_content {
+  display: flex;
+  flex-direction: column;
+  justify-content: center;
+  gap: 1rem;
+}
+
+.error_content h1,
+.error_content h2 {
+  letter-spacing: 0.1rem;
+}
+
+.error_content h1 {
+  font-size: 3rem;
+}
+
+.error_content h2 {
+  font-size: 2rem;
+}
+
+.error_content p {
+  font-size: 1.2rem;
+}
+
+.error_content p a,
+.error_content p a:visited {
+  color: var(--2);
+  text-decoration: none;
+}
+
+.error_content p a:hover {
+  color: var(--5);
+}
+```
+### Styles for the previous and next button on the search page
+``` css
+.page_navigation {
+  padding: 0 0 2rem 0;
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+}
+
+.page_navigation button {
+  background: var(--bg);
+  color: var(--fg);
+  padding: 1rem;
+  border-radius: 0.5rem;
+  outline: none;
+  border: none;
+}
+
+.page_navigation button:active {
+  filter: brightness(1.2);
+}
+```
+
+### Styles for the about page
+
+This part is only available right now in the **rolling/edge/unstable** version
+
+```css
+.about-container article{
+    font-size: 1.5rem;
+    color:var(--fg);
+    padding-bottom: 10px;
+  }
+
+.about-container article h1{
+    color: var(--2);
+    font-size: 2.8rem;
+  }
+
+.about-container article div{
+    padding-bottom: 15px;
+  }
+
+.about-container a{
+  color:var(--3);
+}
+
+.about-container article h2{
+  color: var(--3);
+  font-size: 1.8rem;
+  padding-bottom: 10px;
+}
+
+.about-container p{
+  color:var(--fg);
+  font-size:  1.6rem;
+  padding-bottom: 10px;
+}
+
+.about-container h3{
+  font-size: 1.5rem;
+}
+```
+
+[⬅️  Go back to Home](./README.md)

public/static/colorschemes/dark-chocolate.css

@@ -0,0 +1,11 @@
+:root {
+    --bg: #000000;
+    --fg: #ffffff;
+    --1: #121212;
+    --2: #808080;
+    --3: #999999;
+    --4: #666666;
+    --5: #bfbfbf;
+    --6: #e0e0e0;
+    --7: #555555;
+  }

public/static/colorschemes/one-dark.css

@@ -0,0 +1,11 @@
+:root {
+  --bg: #282c34;
+  --fg: #abb2bf;
+  --1: #3b4048;
+  --2: #a3be8c;
+  --3: #b48ead;
+  --4: #c8ccd4;
+  --5: #e06c75;
+  --6: #61afef;
+  --7: #be5046;
+}

public/static/colorschemes/tokyo-night.css

@@ -0,0 +1,11 @@
+:root {
+  --bg: #1a1b26;
+  --fg: #c0caf5;
+  --1: #32364a;
+  --2: #a9b1d6;
+  --3: #5a5bb8;
+  --4: #6b7089;
+  --5: #e2afff;
+  --6: #a9a1e1;
+  --7: #988bc7;
+}

public/static/themes/simple.css

@@ -260,3 +260,44 @@ footer {
 .page_navigation button:active {
   filter: brightness(1.2);
 }
+
+/* Styles for the about page */
+
+  .about-container article{
+      font-size: 1.5rem;
+      color:var(--fg);
+      padding-bottom: 10px;
+    }
+
+  .about-container article h1{
+      color: var(--2);
+      font-size: 2.8rem;
+    }
+
+  .about-container article div{
+      padding-bottom: 15px;
+    }
+
+  .about-container a{
+    color:var(--3);
+  }
+
+  .about-container article h2{
+    color: var(--3);
+    font-size: 1.8rem;
+    padding-bottom: 10px;
+  }
+
+  .about-container p{
+    color:var(--fg);
+    font-size:  1.6rem;
+    padding-bottom: 10px;
+  }
+
+  .about-container h3{
+    font-size: 1.5rem;
+  }
+
+  .about-container {
+    width: 80%;
+  }

public/templates/about.html

@@ -1,20 +1,29 @@
 {{>header this}}
 <main class="about-container">
-  <h1>Websurfx</h1>
-  <small
-    >a lightening fast, privacy respecting, secure meta search engine</small
-  >
-  <article>
-    Lorem ipsum dolor sit amet, officia excepteur ex fugiat reprehenderit enim
-    labore culpa sint ad nisi Lorem pariatur mollit ex esse exercitation amet.
-    Nisi anim cupidatat excepteur officia. Reprehenderit nostrud nostrud ipsum
-    Lorem est aliquip amet voluptate voluptate dolor minim nulla est proident.
-    Nostrud officia pariatur ut officia. Sit irure elit esse ea nulla sunt ex
-    occaecat reprehenderit commodo officia dolor Lorem duis laboris cupidatat
-    officia voluptate. Culpa proident adipisicing id nulla nisi laboris ex in
-    Lorem sunt duis officia eiusmod. Aliqua reprehenderit commodo ex non
-    excepteur duis sunt velit enim. Voluptate laboris sint cupidatat ullamco ut
-    ea consectetur et est culpa et culpa duis.
+  <article >
+    <div>
+    <h1 >Websurfx</h1>
+    <hr size="4" width="100%" color="#a6e3a1"> 
+    </div>
+    <p>A modern-looking, lightning-fast, privacy-respecting, secure meta search engine written in Rust. It provides a fast and secure search experience while respecting user privacy.<br> It aggregates results from multiple search engines and presents them in an unbiased manner, filtering out trackers and ads.
+    </p>
+
+    <h2>Some of the Top Features:</h2>
+
+    <ul><strong>Lightning fast </strong>- Results load within milliseconds for an instant search experience.</ul>
+
+    <ul><strong>Secure search</strong> - All searches are performed over an encrypted connection to prevent snooping.</ul>
+
+    <ul><strong>Ad free results</strong> - All search results are ad free and clutter free for a clean search experience.</ul>
+
+    <ul><strong>Privacy focused</strong> - Websurface does not track, store or sell your search data. Your privacy is our priority.</ul>
+
+    <ul><strong>Free and Open source</strong> - The entire project's code is open source and available for free on <a href="https://github.com/neon-mmd/websurfx">GitHub</a> under an GNU Affero General Public License.</ul>
+
+    <ul><strong>Highly customizable</strong> - Websurface comes with 9 built-in color themes and supports creating custom themes effortlessly.</ul>
   </article>
+
+  <h3>Devoloped by: <a href="https://github.com/neon-mmd/websurfx">Websurfx team</a></h3>
 </main>
 {{>footer}}
+

public/templates/index.html

@@ -1,6 +1,6 @@
 {{>header this}}
 <main class="search-container">
-    <img src="images/fps_logo.png" alt="Websurfx meta-search engine logo" />
+    <img src="../images/websurfx_logo.png" alt="Websurfx meta-search engine logo" />
     {{>search_bar}}
 </main>
 <script src="static/index.js"></script>

src/bin/websurfx.rs

@@ -5,7 +5,6 @@
 
 use std::net::TcpListener;
 
-use env_logger::Env;
 use websurfx::{config_parser::parser::Config, run};
 
 /// The function that launches the main server and registers all the routes of the website.
@@ -20,7 +19,10 @@ async fn main() -> std::io::Result<()> {
     let config = Config::parse().unwrap();
 
     // Initializing logging middleware with level set to default or info.
-    env_logger::Builder::from_env(Env::default().default_filter_or("info")).init();
+    if config.logging || config.debug {
+        use env_logger::Env;
+        env_logger::Builder::from_env(Env::default().default_filter_or("info")).init();
+    }
 
     log::info!("started server on port {}", config.port);
 

src/cache/cacher.rs

@@ -10,9 +10,8 @@ use redis::{Client, Commands, Connection};
 /// # Fields
 ///
 /// * `redis_connection_url` - It stores the redis Connection url address.
-#[derive(Clone)]
 pub struct RedisCache {
-    redis_connection_url: String,
+    connection: Connection,
 }
 
 impl RedisCache {
@@ -21,10 +20,11 @@ impl RedisCache {
     /// # Arguments
     ///
     /// * `redis_connection_url` - It stores the redis Connection url address.
-    pub fn new(redis_connection_url: String) -> Self {
-        RedisCache {
-            redis_connection_url,
-        }
+    pub fn new(redis_connection_url: String) -> Result<Self, Box<dyn std::error::Error>> {
+        let client = Client::open(redis_connection_url)?;
+        let connection = client.get_connection()?;
+        let redis_cache = RedisCache { connection };
+        Ok(redis_cache)
     }
 
     /// A helper function which computes the hash of the url and formats and returns it as string.
@@ -32,7 +32,7 @@ impl RedisCache {
     /// # Arguments
     ///
     /// * `url` - It takes an url as string.
-    fn compute_url_hash(self, url: &str) -> String {
+    fn compute_url_hash(url: &str) -> String {
         format!("{:?}", compute(url))
     }
 
@@ -41,11 +41,9 @@ impl RedisCache {
     /// # Arguments
     ///
     /// * `url` - It takes an url as a string.
-    pub fn cached_results_json(self, url: String) -> Result<String, Box<dyn std::error::Error>> {
-        let hashed_url_string = self.clone().compute_url_hash(&url);
-        let mut redis_connection: Connection =
-            Client::open(self.redis_connection_url)?.get_connection()?;
-        Ok(redis_connection.get(hashed_url_string)?)
+    pub fn cached_results_json(&mut self, url: &str) -> Result<String, Box<dyn std::error::Error>> {
+        let hashed_url_string = Self::compute_url_hash(url);
+        Ok(self.connection.get(hashed_url_string)?)
     }
 
     /// A function which caches the results by using the hashed `url` as the key and
@@ -57,20 +55,18 @@ impl RedisCache {
     /// * `json_results` - It takes the json results string as an argument.
     /// * `url` - It takes the url as a String.
     pub fn cache_results(
-        self,
+        &mut self,
         json_results: String,
-        url: String,
+        url: &str,
     ) -> Result<(), Box<dyn std::error::Error>> {
-        let hashed_url_string = self.clone().compute_url_hash(&url);
-        let mut redis_connection: Connection =
-            Client::open(self.redis_connection_url)?.get_connection()?;
+        let hashed_url_string = Self::compute_url_hash(url);
 
         // put results_json into cache
-        redis_connection.set(hashed_url_string.clone(), json_results)?;
+        self.connection.set(&hashed_url_string, json_results)?;
 
         // Set the TTL for the key to 60 seconds
-        redis_connection
-            .expire::<String, u32>(hashed_url_string.clone(), 60)
+        self.connection
+            .expire::<String, u32>(hashed_url_string, 60)
             .unwrap();
 
         Ok(())

src/cache/mod.rs

@@ -1 +1 @@
-pub mod cacher; 
+pub mod cacher;

src/config_parser/parser.rs

@@ -3,7 +3,11 @@
 
 use super::parser_models::Style;
 use rlua::Lua;
-use std::fs;
+use std::{format, fs, path::Path};
+
+// ------- Constants --------
+static COMMON_DIRECTORY_NAME: &str = "websurfx";
+static CONFIG_FILE_NAME: &str = "config.lua";
 
 /// A named struct which stores the parsed config file options.
 ///
@@ -20,6 +24,16 @@ pub struct Config {
     pub binding_ip_addr: String,
     pub style: Style,
     pub redis_connection_url: String,
+    pub aggregator: AggreatorConfig,
+    pub logging: bool,
+    pub debug: bool,
+}
+
+/// Configuration options for the aggregator.
+#[derive(Clone)]
+pub struct AggreatorConfig {
+    /// Whether to introduce a random delay before sending the request to the search engine.
+    pub random_delay: bool,
 }
 
 impl Config {
@@ -32,15 +46,24 @@ impl Config {
     /// or io error if the config.lua file doesn't exists otherwise it returns a newly contructed
     /// Config struct with all the parsed config options from the parsed config file.
     pub fn parse() -> Result<Self, Box<dyn std::error::Error>> {
-        let lua = Lua::new();
-
-        lua.context(|context| {
+        Lua::new().context(|context| -> Result<Self, Box<dyn std::error::Error>> {
             let globals = context.globals();
 
             context
-                .load(&fs::read_to_string("./websurfx/config.lua")?)
+                .load(&fs::read_to_string(
+                    Config::handle_different_config_file_path()?,
+                )?)
                 .exec()?;
 
+            let production_use = globals.get::<_, bool>("production_use")?;
+            let aggregator_config = if production_use {
+                AggreatorConfig { random_delay: true }
+            } else {
+                AggreatorConfig {
+                    random_delay: false,
+                }
+            };
+
             Ok(Config {
                 port: globals.get::<_, u16>("port")?,
                 binding_ip_addr: globals.get::<_, String>("binding_ip_addr")?,
@@ -49,7 +72,53 @@ impl Config {
                     globals.get::<_, String>("colorscheme")?,
                 ),
                 redis_connection_url: globals.get::<_, String>("redis_connection_url")?,
+                aggregator: aggregator_config,
+                logging: globals.get::<_, bool>("logging")?,
+                debug: globals.get::<_, bool>("debug")?,
             })
         })
     }
+    /// A helper function which returns an appropriate config file path checking if the config
+    /// file exists on that path.
+    ///
+    /// # Error
+    ///
+    /// Returns a `config file not found!!` error if the config file is not present under following
+    /// paths which are:
+    /// 1. `~/.config/websurfx/` if it not present here then it fallbacks to the next one (2)
+    /// 2. `/etc/xdg/websurfx/config.lua` if it is not present here then it fallbacks to the next
+    ///    one (3).
+    /// 3. `websurfx/` (under project folder ( or codebase in other words)) if it is not present
+    ///    here then it returns an error as mentioned above.
+    fn handle_different_config_file_path() -> Result<String, Box<dyn std::error::Error>> {
+        if Path::new(
+            format!(
+                "{}/.config/{}/config.lua",
+                std::env::var("HOME").unwrap(),
+                COMMON_DIRECTORY_NAME
+            )
+            .as_str(),
+        )
+        .exists()
+        {
+            Ok(format!(
+                "{}/.config/{}/{}",
+                std::env::var("HOME").unwrap(),
+                COMMON_DIRECTORY_NAME,
+                CONFIG_FILE_NAME
+            ))
+        } else if Path::new(
+            format!("/etc/xdg/{}/{}", COMMON_DIRECTORY_NAME, CONFIG_FILE_NAME).as_str(),
+        )
+        .exists()
+        {
+            Ok("/etc/xdg/websurfx/config.lua".to_string())
+        } else if Path::new(format!("./{}/{}", COMMON_DIRECTORY_NAME, CONFIG_FILE_NAME).as_str())
+            .exists()
+        {
+            Ok("./websurfx/config.lua".to_string())
+        } else {
+            Err(format!("Config file not found!!").into())
+        }
+    }
 }

src/handler/mod.rs

@@ -0,0 +1 @@
+pub mod public_path_handler;

src/handler/public_path_handler.rs

@@ -0,0 +1,31 @@
+//! This module provides the functionality to handle theme folder present on different paths and
+//! provide one appropriate path on which it is present and can be used.
+
+use std::io::Error;
+use std::path::Path;
+
+// ------- Constants --------
+static PUBLIC_DIRECTORY_NAME: &str = "public";
+
+/// A function which returns an appropriate theme directory path checking if the theme
+/// directory exists on that path.
+///
+/// # Error
+///
+/// Returns a `Theme (public) folder not found!!` error if the theme folder is not present under following
+/// paths which are:
+/// 1. `/opt/websurfx` if it not present here then it fallbacks to the next one (2)
+/// 2. Under project folder ( or codebase in other words) if it is not present
+///    here then it returns an error as mentioned above.
+pub fn handle_different_public_path() -> Result<String, Error> {
+    if Path::new(format!("/opt/websurfx/{}/", PUBLIC_DIRECTORY_NAME).as_str()).exists() {
+        Ok(format!("/opt/websurfx/{}", PUBLIC_DIRECTORY_NAME))
+    } else if Path::new(format!("./{}/", PUBLIC_DIRECTORY_NAME).as_str()).exists() {
+        Ok(format!("./{}", PUBLIC_DIRECTORY_NAME))
+    } else {
+        Err(Error::new(
+            std::io::ErrorKind::NotFound,
+            "Themes (public) folder not found!!",
+        ))
+    }
+}

src/lib.rs

@@ -4,6 +4,7 @@
 pub mod cache;
 pub mod config_parser;
 pub mod engines;
+pub mod handler;
 pub mod search_results_handler;
 pub mod server;
 
@@ -15,6 +16,7 @@ use actix_files as fs;
 use actix_web::{dev::Server, middleware::Logger, web, App, HttpServer};
 use config_parser::parser::Config;
 use handlebars::Handlebars;
+use handler::public_path_handler::handle_different_public_path;
 
 /// Runs the web server on the provided TCP listener and returns a `Server` instance.
 ///
@@ -39,8 +41,10 @@ use handlebars::Handlebars;
 pub fn run(listener: TcpListener, config: Config) -> std::io::Result<Server> {
     let mut handlebars: Handlebars = Handlebars::new();
 
+    let public_folder_path: String = handle_different_public_path()?;
+
     handlebars
-        .register_templates_directory(".html", "./public/templates")
+        .register_templates_directory(".html", format!("{}/templates", public_folder_path))
         .unwrap();
 
     let handlebars_ref: web::Data<Handlebars> = web::Data::new(handlebars);
@@ -51,8 +55,14 @@ pub fn run(listener: TcpListener, config: Config) -> std::io::Result<Server> {
             .app_data(web::Data::new(config.clone()))
             .wrap(Logger::default()) // added logging middleware for logging.
             // Serve images and static files (css and js files).
-            .service(fs::Files::new("/static", "./public/static").show_files_listing())
-            .service(fs::Files::new("/images", "./public/images").show_files_listing())
+            .service(
+                fs::Files::new("/static", format!("{}/static", public_folder_path))
+                    .show_files_listing(),
+            )
+            .service(
+                fs::Files::new("/images", format!("{}/images", public_folder_path))
+                    .show_files_listing(),
+            )
             .service(routes::robots_data) // robots.txt
             .service(routes::index) // index page
             .service(routes::search) // search page

src/search_results_handler/aggregation_models.rs

@@ -116,7 +116,7 @@ impl RawSearchResult {
     }
 }
 
-/// A named struct to store, serialize, deserialize the all the search results scraped and 
+/// A named struct to store, serialize, deserialize the all the search results scraped and
 /// aggregated from the upstream search engines.
 ///
 /// # Fields

src/search_results_handler/aggregator.rs

@@ -29,6 +29,7 @@ use crate::engines::{duckduckgo, searx};
 ///
 /// * `query` - Accepts a string to query with the above upstream search engines.
 /// * `page` - Accepts an u32 page number.
+/// * `random_delay` - Accepts a boolean value to add a random delay before making the request.
 ///
 /// # Error
 ///
@@ -38,14 +39,18 @@ use crate::engines::{duckduckgo, searx};
 pub async fn aggregate(
     query: &str,
     page: u32,
+    random_delay: bool,
+    debug: bool,
 ) -> Result<SearchResults, Box<dyn std::error::Error>> {
     let user_agent: String = random_user_agent();
     let mut result_map: HashMap<String, RawSearchResult> = HashMap::new();
 
     // Add a random delay before making the request.
-    let mut rng = rand::thread_rng();
-    let delay_secs = rng.gen_range(1..10);
-    std::thread::sleep(Duration::from_secs(delay_secs));
+    if random_delay || !debug {
+        let mut rng = rand::thread_rng();
+        let delay_secs = rng.gen_range(1..10);
+        std::thread::sleep(Duration::from_secs(delay_secs));
+    }
 
     // fetch results from upstream search engines simultaneously/concurrently.
     let (ddg_map_results, searx_map_results) = join!(

src/search_results_handler/user_agent.rs

@@ -1,13 +1,8 @@
 //! This module provides the functionality to generate random user agent string.
 
-use fake_useragent::{Browsers, UserAgentsBuilder};
+use fake_useragent::{Browsers, UserAgents, UserAgentsBuilder};
 
-/// A function to generate random user agent to improve privacy of the user.
-///
-/// # Returns
-///
-/// A randomly generated user agent string.
-pub fn random_user_agent() -> String {
+static USER_AGENTS: once_cell::sync::Lazy<UserAgents> = once_cell::sync::Lazy::new(|| {
     UserAgentsBuilder::new()
         .cache(false)
         .dir("/tmp")
@@ -21,6 +16,13 @@ pub fn random_user_agent() -> String {
                 .set_mozilla(),
         )
         .build()
-        .random()
-        .to_string()
+});
+
+/// A function to generate random user agent to improve privacy of the user.
+///
+/// # Returns
+///
+/// A randomly generated user agent string.
+pub fn random_user_agent() -> String {
+    USER_AGENTS.random().to_string()
 }

src/server/routes.rs

@@ -7,6 +7,7 @@ use std::fs::read_to_string;
 use crate::{
     cache::cacher::RedisCache,
     config_parser::parser::Config,
+    handler::public_path_handler::handle_different_public_path,
     search_results_handler::{aggregation_models::SearchResults, aggregator::aggregate},
 };
 use actix_web::{get, web, HttpRequest, HttpResponse};
@@ -73,7 +74,7 @@ pub async fn search(
     let params = web::Query::<SearchParams>::from_query(req.query_string())?;
 
     //Initialize redis cache connection struct
-    let redis_cache = RedisCache::new(config.redis_connection_url.clone());
+    let mut redis_cache = RedisCache::new(config.redis_connection_url.clone())?;
     match &params.q {
         Some(query) => {
             if query.trim().is_empty() {
@@ -81,11 +82,10 @@ pub async fn search(
                     .insert_header(("location", "/"))
                     .finish())
             } else {
-                // Initialize the page url as an empty string
-                let mut page_url = String::new();
+                let page_url: String; // Declare the page_url variable without initializing it
+
+                // ...
 
-                // Find whether the page is valid page number if not then return
-                // the first page number and also construct the page_url accordingly
                 let page = match params.page {
                     Some(page_number) => {
                         if page_number <= 1 {
@@ -117,7 +117,7 @@ pub async fn search(
                 };
 
                 // fetch the cached results json.
-                let cached_results_json = redis_cache.clone().cached_results_json(page_url.clone());
+                let cached_results_json = redis_cache.cached_results_json(&page_url);
                 // check if fetched results was indeed fetched or it was an error and if so
                 // handle the data accordingly.
                 match cached_results_json {
@@ -128,12 +128,10 @@ pub async fn search(
                     }
                     Err(_) => {
                         let mut results_json: crate::search_results_handler::aggregation_models::SearchResults =
-                    aggregate(query, page).await?;
+                            aggregate(query, page, config.aggregator.random_delay, config.debug).await?;
                         results_json.add_style(config.style.clone());
-                        redis_cache.clone().cache_results(
-                            serde_json::to_string(&results_json)?,
-                            page_url.clone(),
-                        )?;
+                        redis_cache
+                            .cache_results(serde_json::to_string(&results_json)?, &page_url)?;
                         let page_content: String = hbs.render("search", &results_json)?;
                         Ok(HttpResponse::Ok().body(page_content))
                     }
@@ -149,7 +147,8 @@ pub async fn search(
 /// Handles the route of robots.txt page of the `websurfx` meta search engine website.
 #[get("/robots.txt")]
 pub async fn robots_data(_req: HttpRequest) -> Result<HttpResponse, Box<dyn std::error::Error>> {
-    let page_content: String = read_to_string("./public/robots.txt")?;
+    let page_content: String =
+        read_to_string(format!("{}/robots.txt", handle_different_public_path()?))?;
     Ok(HttpResponse::Ok()
         .content_type("text/plain; charset=ascii")
         .body(page_content))

websurfx/config.lua

@@ -1,8 +1,15 @@
--- Server
+-- ### General ###
+logging = true -- an option to enable or disable logs.
+debug = false -- an option to enable or disable debug mode.
+
+-- ### Server ###
 port = "8080" -- port on which server should be launched
 binding_ip_addr = "127.0.0.1" --ip address on the which server should be launched.
+production_use = false -- whether to use production mode or not (in other words this option should be used if it is to be used to host it on the server to provide a service to a large number of users)
+-- if production_use is set to true
+-- There will be a random delay before sending the request to the search engines, this is to prevent DDoSing the upstream search engines from a large number of simultaneous requests.
 
--- Website
+-- ### Website ###
 -- The different colorschemes provided are:
 -- {{
 -- catppuccin-mocha
@@ -17,5 +24,5 @@ binding_ip_addr = "127.0.0.1" --ip address on the which server should be launche
 colorscheme = "catppuccin-mocha" -- the colorscheme name which should be used for the website theme
 theme = "simple" -- the theme name which should be used for the website
 
--- Caching
+-- ### Caching ###
 redis_connection_url = "redis://127.0.0.1:8082" -- redis connection url address on which the client should connect on.