Setting Up a Private Search Proxy Service: OpenWebUI + Backend Simulating SearXNG Local Search

Implement a local search service using Python and Flask that is compatible with the SearXNG API and supports deep web crawling

Background

When building a personal AI assistant, one often encounters a problem: the knowledge in large language models has a cutoff date, so they cannot access the latest information. Although it’s possible to integrate search engine APIs, commercial services are quite expensive, and the free versions have limits on the number of calls.

So I wrote a lightweight local search proxy service, which essentially takesBaidu SearchPackaged as SearXNG-Compatible APIs, and at the same time addIn-Depth Web ScrapingCapabilities—Not only does it return search results, but it also automatically extracts the body text of top-ranked pages, allowing the AI to read the original content directly.

Feature Overview

Workflow

User Request → Flask Service → Baidu Search API → Retrieve Results List
 ↓
 (Optional) Deep Crawl Top N
 ↓
 Concurrently Extract Body Content
 ↓
 Quality Score + Content Attachments
 ↓
 Return SearXNG-Compatible JSON

Getting Started Quickly

1. Install dependencies

pip install flask baidu-serp-api trafilatura readability-lxml aiohttp fake-useragent

Optional installation (to improve extraction success rate):

pip install goose3

2. Start the service

python SearxBaiduService.py

Or just double-click RunSearcher.bat(Windows).

By default, the service runs on http://localhost:8888.

3. Example of Use

# Regular Search
curl "http://localhost:8888/search?q=人工智能最新进展"

# Limit the Number of Results Returned
curl "http://localhost:8888/search?q=深度学习&count=4"

# Disable Deep Crawling
curl "http://localhost:8888/search?q=Transformer&deep_crawl=false"

# Specify the number of items to fetch at a given depth
curl "http://localhost:8888/search?q=大模型&deep_count=2"

4. Return Format (SearXNG-compatible)

{
  "query": "Artificial Intelligence",
  "number_of_results": 6,
  "results": [
    {
 "title": "Article Title",
 "url": "https://..." ,
 "content": "Summary\n\n[In-depth content]\nMain text...",
 "source": "baidu",
 "engine": "baidu"
    }
  ]
}

Optimizing Key Parameters

In the "Adjustable Parameters" section at the beginning of the code, you can adjust the following parameters:

Key Points for Implementing Deep Crawling

The following was used to extract the body text:Three-Tier Demotion Strategy::

1. drawing: First choice—offers the highest extraction quality and excellent support for Chinese
2. readability-lxml: An alternative option from Mozilla with good stability
3. goose3: A fallback option, suitable for news pages

After each crawl, the following will also be performed:Quality Assessment(text length, percentage of Chinese content, headline quality, etc.); pages with scores that are too low will not have additional content added to them, to avoid introducing noise.

Open WebUI Network Search Configuration

Once the service is launched, it can integrate seamlessly Open WebUI online search feature. The configuration steps are as follows:

procedure

1. Log in to Open WebUI → Click the administrator avatar in the lower-left corner → Administrator Panel(Admin Panel)
2. Navigate to Settings → Online Search(Web Search) tab
3. Fill in the following parameters:

4. Click Save

Parameter Description

• Set the number of search results to 1–3: When used with the deep crawl feature, setting this value too high may increase response latency. We recommend setting it to 2 to ensure both sufficient information and optimal speed.
• Set the number of concurrent requests to 0: Concurrency is controlled internally by this service (DEEP_CRAWL_CONCURRENT), to avoid duplicate rate limiting on the Open WebUI side.
• Bypassing Embedding and Retrieval + Bypass the page loader: This service has appended deep-crawled content to content Fields—no need for Open WebUI to load an additional page, thereby avoiding duplicate scraping and IP-based access restrictions.
• SSL Validation Disabled: This service prevents errors when using self-signed or non-standard certificates (for security in local environments).

Verify whether it is in effect

Enter a question requiring real-time information (such as “What’s in the news today?”) in the chat window and see if it triggers a search request. You can also check the Open WebUI logs to see if there are any http://localhost:8888/search Call logs.

Important Information and Disclaimer

1. For educational and research purposes only, Please do not use this for commercial purposes or in scenarios involving high-frequency requests
2. The Baidu Search API essentially simulates browser behavior, so there is a risk of being restricted. Please manage your request frequency appropriately.
3. We recommend deploying this locally or on an internal network; do not expose it to the public internet.
4. In-depth crawling may access third-party websites; please respect the target websites' robots.txt files and terms of service.
5. This project does not store any user data; all caching is used solely to improve response times.

The above configuration is based on Open WebUI version 0.9.x. The interface may vary slightly across different versions, but the core parameters remain the same.