{"id":1548,"date":"2026-06-17T17:57:33","date_gmt":"2026-06-17T09:57:33","guid":{"rendered":"https:\/\/www.livetops.com\/?p=1548"},"modified":"2026-06-17T17:57:34","modified_gmt":"2026-06-17T09:57:34","slug":"%e6%90%ad%e5%bb%ba%e7%a7%81%e6%9c%89%e6%90%9c%e7%b4%a2%e4%bb%a3%e7%90%86%e6%9c%8d%e5%8a%a1%ef%bc%9aopenwebui-%e5%90%8e%e7%ab%af%e6%a8%a1%e6%8b%9f-searxng-%e6%9c%ac%e5%9c%b0%e6%90%9c%e7%b4%a2","status":"publish","type":"post","link":"https:\/\/www.livetops.com\/en\/archives\/1548","title":{"rendered":"Setting Up a Private Search Proxy Service: OpenWebUI + Backend Simulating SearXNG Local Search"},"content":{"rendered":"
\n

Implement a local search service using Python and Flask that is compatible with the SearXNG API and supports deep web crawling<\/p>\n<\/blockquote>\n\n\n\n

Background<\/h2>\n\n\n\n

When building a personal AI assistant, one often encounters a problem: the knowledge in large language models has a cutoff date, so they cannot access the latest information. Although it\u2019s possible to integrate search engine APIs, commercial services are quite expensive, and the free versions have limits on the number of calls.<\/p>\n\n\n\n

So I wrote a lightweight local search proxy service, which essentially takesBaidu Search<\/strong>Packaged as SearXNG-Compatible APIs<\/strong>, and at the same time addIn-Depth Web Scraping<\/strong>Capabilities\u2014Not only does it return search results, but it also automatically extracts the body text of top-ranked pages, allowing the AI to read the original content directly.<\/p>\n\n\n\n

\n\n\n\n

Feature Overview<\/h2>\n\n\n\n
Features<\/th>instructions<\/th><\/tr><\/thead>
Baidu Search<\/td>Based on baidu_serp_api<\/code> Simulate a Baidu search to retrieve the title, link, and summary<\/td><\/tr>
Deep Crawling<\/td>Asynchronously and concurrently fetch the top N results pages, extract the body text, and append it to the summary<\/td><\/tr>
Multi-Extractor<\/td>Automatic downgrade using trafilatura \/ readability \/ goose3<\/td><\/tr>
Caching Mechanism<\/td>Search results are cached for 1 hour to reduce duplicate requests<\/td><\/tr>
Over-frequency Protection<\/td>Random delay + cooldown mechanism, reducing the probability of risk control<\/td><\/tr>
API Compatibility<\/td>The output format is consistent with SearXNG, allowing for direct integration with various AI applications.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n

Workflow<\/h2>\n\n\n\n
User Request \u2192 Flask Service \u2192 Baidu Search API \u2192 Retrieve Results List\n \u2193\n (Optional) Deep Crawl Top N\n \u2193\n Concurrently Extract Body Content\n \u2193\n Quality Score + Content Attachments\n \u2193\n Return SearXNG-Compatible JSON<\/code><\/pre>\n\n\n\n
\n\n\n\n
Getting Started Quickly<\/h2>\n\n\n\n
1. Install dependencies<\/h3>\n\n\n\n
pip install flask baidu-serp-api trafilatura readability-lxml aiohttp fake-useragent<\/code><\/pre>\n\n\n\n
Optional installation (to improve extraction success rate):<\/p>\n\n\n\n
pip install goose3<\/code><\/pre>\n\n\n\n
2. Start the service<\/h3>\n\n\n\n
python SearxBaiduService.py<\/code><\/pre>\n\n\n\n
Or just double-click RunSearcher.bat<\/code>(Windows).<\/p>\n\n\n\n
By default, the service runs on http:\/\/localhost:8888<\/code>.<\/p>\n\n\n\n
3. Example of Use<\/h3>\n\n\n\n
# Regular Search\ncurl \"http:\/\/localhost:8888\/search?q=\u4eba\u5de5\u667a\u80fd\u6700\u65b0\u8fdb\u5c55\"\n\n# Limit the Number of Results Returned\ncurl \"http:\/\/localhost:8888\/search?q=\u6df1\u5ea6\u5b66\u4e60&count=4\"\n\n# Disable Deep Crawling\ncurl \"http:\/\/localhost:8888\/search?q=Transformer&deep_crawl=false\"\n\n# Specify the number of items to fetch at a given depth\ncurl \"http:\/\/localhost:8888\/search?q=\u5927\u6a21\u578b&deep_count=2\"<\/code><\/pre>\n\n\n\n
4. Return Format (SearXNG-compatible)<\/h3>\n\n\n\n
{\n  \"query\": \"Artificial Intelligence\",\n  \"number_of_results\": 6,\n  \"results\": [\n    {\n \"title\": \"Article Title\",\n \"url\": \"https:\/\/...\" ,\n \"content\": \"Summary\\n\\n[In-depth content]\\nMain text...\",\n \"source\": \"baidu\",\n \"engine\": \"baidu\"\n    }\n  ]\n}<\/code><\/pre>\n\n\n\n
\n\n\n\n
Optimizing Key Parameters<\/h2>\n\n\n\n
In the \"Adjustable Parameters\" section at the beginning of the code, you can adjust the following parameters:<\/p>\n\n\n\n
Parameters<\/th>default value<\/th>instructions<\/th><\/tr><\/thead>
DEFAULT_RESULT_COUNT<\/code><\/td>6<\/td>Default number of results returned<\/td><\/tr>
DEEP_CRAWL_MAX_RESULTS<\/code><\/td>6<\/td>Maximum number of items to crawl at maximum depth<\/td><\/tr>
DEEP_CRAWL_CONCURRENT<\/code><\/td>2<\/td>Number of concurrent scrapes (recommended: \u22643)<\/td><\/tr>
DEEP_CRAWL_TIMEOUT<\/code><\/td>6<\/td>Single-page timeout (seconds)<\/td><\/tr>
CACHE_TTL<\/code><\/td>3600<\/td>Cache Expiration (seconds)<\/td><\/tr>
RANDOM_DELAY_MIN\/MAX<\/code><\/td>5~10<\/td>Random delay for search requests (seconds)<\/td><\/tr>
ERROR_COOLDOWN<\/code><\/td>60<\/td>Cooldown after failure (seconds)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Key Points for Implementing Deep Crawling<\/h2>\n\n\n\n
The following was used to extract the body text:Three-Tier Demotion Strategy<\/strong>::<\/p>\n\n\n\n
    \n
  1. drawing<\/strong>: First choice\u2014offers the highest extraction quality and excellent support for Chinese<\/li>\n\n\n\n
  2. readability-lxml<\/strong>: An alternative option from Mozilla with good stability<\/li>\n\n\n\n
  3. goose3<\/strong>: A fallback option, suitable for news pages<\/li>\n<\/ol>\n\n\n\n
    After each crawl, the following will also be performed:Quality Assessment<\/strong>(text length, percentage of Chinese content, headline quality, etc.); pages with scores that are too low will not have additional content added to them, to avoid introducing noise.<\/p>\n\n\n\n
    \n\n\n\n
    Open WebUI Network Search Configuration<\/h2>\n\n\n\n
    Once the service is launched, it can integrate seamlessly Open WebUI<\/strong> online search feature. The configuration steps are as follows:<\/p>\n\n\n\n
    procedure<\/h3>\n\n\n\n
      \n
    1. Log in to Open WebUI \u2192 Click the administrator avatar in the lower-left corner \u2192 Administrator Panel<\/strong>(Admin Panel)<\/li>\n\n\n\n
    2. Navigate to Settings<\/strong> \u2192 Online Search<\/strong>(Web Search) tab<\/li>\n\n\n\n
    3. Fill in the following parameters:<\/li>\n<\/ol>\n\n\n\n
      Configuration Options<\/th>Recommended Value<\/th><\/tr><\/thead>
      Search Engine<\/td>searxng<\/code><\/td><\/tr>
      Searxng Query Interface URL<\/td>http:\/\/localhost:8888\/search<\/code><\/td><\/tr>
      Number of search results<\/td>1-3<\/code>(Recommendation 2)<\/td><\/tr>
      Number of concurrent requests<\/td>0<\/code><\/td><\/tr>
      \u2705 Bypass embedding and retrieval<\/td>Open<\/strong><\/td><\/tr>
      \u2705 Bypass the page loader<\/td>Open<\/strong><\/td><\/tr>
      Web Page Loading Engine<\/td>Default<\/code><\/td><\/tr>
      Timeout<\/td>5<\/code> second<\/td><\/tr>
      SSL Validation<\/td>Close<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        \n
      1. Click Save<\/strong><\/li>\n<\/ol>\n\n\n\n
        Parameter Description<\/h3>\n\n\n\n