最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

【Python实战】---- 爬取 CSDN 专栏文章列表

运维笔记admin36浏览0评论

1. 场景

需求就是专栏中文章随着时间写的越多,如果后边需要去查找的时候比较麻烦,比如一些不常用的 git 命令,或者有些开发场景的细节,在之前已经开发完了,现在忘记部分细节,需要在之前的输出文章中去查找,当几十几百篇文章时,查找就比较麻烦,但是如果没发布一篇文章,自己去更新专栏的文章目录又是一个比较繁琐的事情,因此写了一个小的爬取程序,在每次发布新的文章时,运行此程序,就可以更新文章目录,方便后期在需要的时候能够快速查找。

2. 引入使用模块

  1. requests 获取网页的内容;
  2. re 使用正则匹配文章的发布日期;
  3. time 用于每次获取网页后的等待,防止被 CSDN 识别为爬虫;
  4. datetime 用于文章发布日期的排序格式化;
  5. BeautifulSoup HTML 解析。
import requests
import re
import time
from datetime import datetime
from bs4 import BeautifulSoup

3. 获取专栏文章

发布评论

评论列表(0)

  1. 暂无评论
ok 不同模板 switch ($forum['model']) { /*case '0': include _include(APP_PATH . 'view/htm/read.htm'); break;*/ default: include _include(theme_load('read', $fid)); break; } } break; case '10': // 主题外链 / thread external link http_location(htmlspecialchars_decode(trim($thread['description']))); break; case '11': // 单页 / single page $attachlist = array(); $imagelist = array(); $thread['filelist'] = array(); $threadlist = NULL; $thread['files'] > 0 and list($attachlist, $imagelist, $thread['filelist']) = well_attach_find_by_tid($tid); $data = data_read_cache($tid); empty($data) and message(-1, lang('data_malformation')); $tidlist = $forum['threads'] ? page_find_by_fid($fid, $page, $pagesize) : NULL; if ($tidlist) { $tidarr = arrlist_values($tidlist, 'tid'); $threadlist = well_thread_find($tidarr, $pagesize); // 按之前tidlist排序 $threadlist = array2_sort_key($threadlist, $tidlist, 'tid'); } $allowpost = forum_access_user($fid, $gid, 'allowpost'); $allowupdate = forum_access_mod($fid, $gid, 'allowupdate'); $allowdelete = forum_access_mod($fid, $gid, 'allowdelete'); $access = array('allowpost' => $allowpost, 'allowupdate' => $allowupdate, 'allowdelete' => $allowdelete); $header['title'] = $thread['subject']; $header['mobile_link'] = $thread['url']; $header['keywords'] = $thread['keyword'] ? $thread['keyword'] : $thread['subject']; $header['description'] = $thread['description'] ? $thread['description'] : $thread['brief']; $_SESSION['fid'] = $fid; if ($ajax) { empty($conf['api_on']) and message(0, lang('closed')); $apilist['header'] = $header; $apilist['extra'] = $extra; $apilist['access'] = $access; $apilist['thread'] = well_thread_safe_info($thread); $apilist['thread_data'] = $data; $apilist['forum'] = $forum; $apilist['imagelist'] = $imagelist; $apilist['filelist'] = $thread['filelist']; $apilist['threadlist'] = $threadlist; message(0, $apilist); } else { include _include(theme_load('single_page', $fid)); } break; default: message(-1, lang('data_malformation')); break; } ?>