聲明

請您遵守網(wǎng)站的robots文件規(guī)定，本文目的只是做學(xué)習(xí)交流使用，包括多個模塊，例如數(shù)據(jù)存儲、日志記錄、錯誤處理、多線程或異步請求

安裝必要的庫

pip install requests beautifulsoup4 sqlite3

項目結(jié)構(gòu)

創(chuàng)建以下文件和目錄結(jié)構(gòu)
my_crawler/
├── config.py
├── crawler.py
├── db.py
├── logger.py
└── main.py

技術(shù)細節(jié)

配置文件 config.py
配置文件用于存儲常量和配置項：

BASE_URL = 'https://example.com'
ARTICLES_URL = f'{BASE_URL}/articles'
DATABASE_NAME = 'articles.db'
LOG_FILE = 'crawler.log'

1.數(shù)據(jù)庫操作 db.py
用于創(chuàng)建數(shù)據(jù)庫表和插入數(shù)據(jù)：

#db.pyimport sqlite3
from config import DATABASE_NAMEdef init_db():conn = sqlite3.connect(DATABASE_NAME)cursor = conn.cursor()cursor.execute('''CREATE TABLE IF NOT EXISTS articles (id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT NOT NULL,url TEXT NOT NULL)''')conn.commit()conn.close()def insert_article(title, url):conn = sqlite3.connect(DATABASE_NAME)cursor = conn.cursor()cursor.execute('INSERT INTO articles (title, url) VALUES (?, ?)', (title, url))conn.commit()conn.close()

2.日志記錄 logger.py
用于配置日志記錄：

logger.pyimport logging
from config import LOG_FILEdef setup_logger():logging.basicConfig(filename=LOG_FILE,level=logging.INFO,format='%(asctime)s - %(levelname)s - %(message)s')logger = setup_logger()

3.爬蟲邏輯 crawler.py
包含爬蟲的主要邏輯：

 crawler.pyimport requests
from bs4 import BeautifulSoup
from config import ARTICLES_URL
from db import insert_article
from logger import loggerdef fetch_page(url):"""獲取網(wǎng)頁內(nèi)容"""try:response = requests.get(url)response.raise_for_status()  # 檢查請求是否成功return response.textexcept requests.RequestException as e:logger.error(f"請求錯誤: {e}")return Nonedef parse_html(html):"""解析HTML，提取文章標題和URL"""soup = BeautifulSoup(html, 'html.parser')articles = []for article in soup.find_all('div', class_='article'):title = article.find('h1', class_='article-title').get_text(strip=True)url = article.find('a')['href']articles.append((title, url))return articlesdef crawl_articles():"""抓取并保存文章"""html = fetch_page(ARTICLES_URL)if html:articles = parse_html(html)for title, url in articles:insert_article(title, url)logger.info(f"已保存文章: {title} - {url}")

4. 主程序 main.py
   啟動爬蟲的主程序：

 main.pyfrom crawler import crawl_articles
from db import init_db
from logger import loggerdef main():logger.info("開始爬蟲任務(wù)")init_db()crawl_articles()logger.info("爬蟲任務(wù)完成")if __name__ == '__main__':main()

小結(jié)

請您遵守網(wǎng)站的robots文件規(guī)定，本文目的只是做學(xué)習(xí)交流使用，感謝csdn平臺