I have open-sourced a website directory system on GitHub that can automatically generate website information — AigoTools. By simply inputting the address of the websites to be included, the system can automatically take screenshots of the websites, crawl the website information, and process this information through a llm model. With this system, you can easily deploy a navigation site with 10,000+ URLs.

GitHub: https://github.com/someu/aigotools

I have open-sourced a navigation site that can automatically collect content. - YouTube

AigoTools is an open-source template for a navigation website. In addition to basic site management functions, it can automatically collect site information,...

youtube.com

Core Features

1. Site Management
2. Automatic Site Information Collection
3. User Management (Clerk)
4. Internationalization
5. Dark/Light Theme Switching
6. SEO Optimization
7. Multiple Image Storage Solutions
8. Open Source UI Design Drafts

Ideas

1. The project is developed based on Next.js and NestJS. The navigation site main body and the crawling service are separated. If you don't need the crawling service, you can directly deploy the main body of the navigation site on Vercel, making it very convenient and fast.
2. Website information processing based on large models. This project uses Jina to read website information and OpenAI to summarize and automatically categorize website information. There are prompts for information summarization and categorization in the project, using the GPT-4 model to summarize website information.
3. The crawling service is based on Bull.js for queue management, easily handling crawling tasks for tens of thousands of navigation sites.
4. The project's UI is simple, and the author has also open-sourced the design drafts. We can make adjustments and develop our own site based on these design drafts.

Project Links

GitHub: https://github.com/someu/aigotools

Demo Site: https://www.aigotools.com/