+
+### 爬取到的article展示
+
+
+
+
+
+
+### 爬取到的回答展示
+
+
+
+
+## 环境以及安装
+**win10** **python**
+1、点击下面这个网页，安装miniconda也就是安装python，下载好以后安装即可，在安装时需要加入到系统环境变量，勾选下图第二个框即可。 [https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py310_23.3.1-0-Windows-x86_64.exe](https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py310_23.3.1-0-Windows-x86_64.exe)
+ 
+2、接着需要修改python安装的路径，将msedgedriver\\.condarc这个档案放到根目录`C:\Users\username`即可，另外再打开一个cmd或者PowerShell
+运行`conda clean -i`输入`Y`即可，此时Python已经可以使用了
+
+3、安装Python相关的调用库，另外再打开一个cmd或者PowerShell，运行
+
+
+### 2、每项单独爬取
+爬取一旦开始就自动运行了，爬取窗口一般不能最小化，可以做其他事情的
+**爬取知乎想法**
+
+默认的爬取每篇想法的睡眠时间是 **6s*图片的数量** 以上
+`
+python.exe crawler.py --think
+`
+
+**爬取知乎回答**
+默认的爬取每篇回答的睡眠时间是**16s**以上，这边实际爬取耗时平均是每篇 **30s**
+`
+python.exe crawler.py --answer
+`
+
+**爬取知乎的article**
+默认的爬取每篇article的睡眠时间是**16s**以上，这边实际爬取130多篇，耗时平均是每篇 **33.096s**
+`
+python.exe crawler.py --article
+`
+
+### 3、三项一起爬取的
+`
+python.exe crawler.py --think --article --answer
+`
+
+### 又发布了一篇，只爬取写的这篇
+第一次可以全部爬取，等所有article或者回答或者想法都已经爬取好以后，此时若是又写了一篇或者几篇，而且想爬取到本地，可以将article/article.txt这个档案重命名到article/article_2023_06_20.txt，或者重命名answer.txt，然后将写好的article或者回答的网址和标题按照之前档案的格式再create一个article.txt/answer.txt档案，运行爬取程序就可以了的，想法会跳过已经爬取好的时间，所以可以按照上面的方式运行，此时只会爬取article.txt/answer.txt的网址
+
配置系统环境变量，双击PATH即可的，最下面两行
+ 
+### 5、验证
+Windows+R输入cmd运行，然后gcc --version和g++ --version就可以验证的
+ 
+### 6、写codes
+ 
+### 7、产生相应的配置档案
+.vscode/[tasks.json](https://www.zhihu.com/search?q=tasks.json&search_source=Entity&hybrid_search_source=Entity&hybrid_search_extra=%7B%22sourceType%22%3A%22answer%22%2C%22sourceId%22%3A3089595368%7D)，编译运行使用的，两个路径的呢cwd和command
+ 
+### 9、编译和debug的路径
+编译：vscode上方：Terminal/Run Build Task
+编译+运行：vscode上方：Run/Run without debugging
+编译+debug：vscode上方：Run/Start debugging
+
++ +[https://zhuanlan.zhihu.com/p/605710105](https://zhuanlan.zhihu.com/p/605710105)
+ + + diff --git a/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/2023-03-26_11_19・IP_属地上海.txt b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/2023-03-26_11_19・IP_属地上海.txt new file mode 100644 index 0000000..2045fc2 --- /dev/null +++ b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/2023-03-26_11_19・IP_属地上海.txt @@ -0,0 +1,2 @@ +2023-03-26_11_19・IP_属地上海 +https://zhuanlan.zhihu.com/p/617062052 \ No newline at end of file diff --git a/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI.pdf b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI.pdf new file mode 100644 index 0000000..491c0be Binary files /dev/null and b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI.pdf differ diff --git a/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI_formula_.md b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI_formula_.md new file mode 100644 index 0000000..bf33d7d --- /dev/null +++ b/article/2023-03-26_11_19_让AI或者GPT具有人类的意识甚至beyond变到AGI_IP_属地上海/让AI或者GPT具有人类的意识甚至beyond变到AGI_formula_.md @@ -0,0 +1,57 @@ +# 让AI或者GPT具有人类的意识甚至beyond变到AGI + + +### 简要：
+ AI的正确使用方式不是提示词，不是写codes，不是情感咨询，不是回答问题。而是AI使用AI，也就是模型自己使用自己，要让AI具有意识那就要让AI学会使用AI，也就是让GPT使用GPT，最终达到相应的目的，也就是具有自我意识。简而言之就是：你用模型不可怕，模型自己用自己那才可怕。
+ AGI也就是通用人工智能，artificial general intelligence。
+ 上面提到了大脑是训练了成千上万年的模型，每个人出生以后，这个模型也在不断地接受数据，进行训练的，但是每个人自己的大脑也就是自己训练的模型，其实并不能遗传，也就是你的记忆你的能力都不能遗传，能遗传的只有可遗传的基因突变，虽然说现代生物学提出的量子基因突变，可以部分论证对环境适应的突变，是由测量导致的。基因突变大多数是外界干扰或者物质影响导致的突变，像是酒精射线等，量子基因突变主要是基因的分子或者原子处于量子叠加态，或者量子多态，环境的影响导致了测量的发生，最终波函数坍塌，导致突变以后的基因进入到经典世界，而量子突变很有可能导致适应环境的突变，所以用进废退也是有道理的。若是每个人训练好的大脑能够遗传，那就基本是永生了，主要是你的记忆和能力都在大脑里，大脑能遗传，那就是永生。可是人类并不能保存每个人的模型，也就是记忆和功能。
+ +### 人类的意识是怎么产生的？
+ 或者说第一个(一群)有意识的人类是怎么出现的？这就像很经典的一个问题，是先有鸡还是先有蛋？鸡生蛋还是蛋生鸡？现代科学理论也就给出了一个可能的答案 **见附录** ，不妨假设某个突变体人类，由于基因突变，导致了大脑结构容量的变多，以及大脑神经网络的结构优化，只有神经网络的结构不断优化，类人才能在残酷的环境中生存下来。而其中某几个基因突变导致了可以逐步使用自己的大脑，包括记忆使用工具。某个具有很小意识的人类在培养后代的时候，也用到了相似的方式，而类人是群居的，这就保证了他的方式方法可以交给很多人，其他人类也学会以后，就代代相传，意识或者规则知识不断累积，最后人类知识的累计和代代传递，造就了人类意识最终的形成，意识最开始应该是很小的，也就是看不出具有意识，但是人类群居的特点，导致了知识可以代代相传，意识慢慢的变大，人类开始具有稍稍大点的意识，也就是能思考，能主动控制大脑的输入，根据输出进行行动，每一代人类知识的累计和规则的累计，都会让意识的形成慢慢变多，也就是思考能力逐渐变强，大脑的主动输入逐渐变多。直到奇点出现某个人类完全具有意识或者某几个人类完全具有意识。最开始的人类应该是懵懵懂懂的，只有知识和规则的出现，才能造就意识，也就是很好的使用自己的大脑，训练自己的大脑。
+ 说到这里那就要回归到本文的主体，让AI或者GPT具有意识。
+ +### 人类的意识究竟是什么
+ 要让AI或者大型NLP语言模型具有意识，那就要从人类的意识究竟是什么讲起，个人观点是，人类的意识是大脑的部分功能，意识是大脑对世界、对自身的认知，也是一套操作系统，用来完美的支配肉体，训练大脑这个模型，使用大脑这个模型，意识是人类通过各种概念的和知识认识到自身的存在。意识可以通过给大脑下达指令，从而控制肢体动作，眼睛负责视觉信息的输入，耳朵负责听觉信息的输入，皮肤负责压力、触觉、痛觉等的输入等，大脑在处理这些信息以后，由意识部分进行汇总，意识的主要功能是协调大脑和肉体，主动使用大脑模型，主动训练大脑模型，主动的思考。各种知识的输入和累计，最终导致了对自身的认知，西方解剖学的发展，就是大脑对自身认知的学科，对世界的认知对星球的认知，也是一点一点汇集累积起来的，亚里士多德对世界的认识，其实是不完整的，但是后人在他的基础上，不断修正完善，从而对世界认识的逐渐变得正确，这套操作系统能完美的兼容肉体和大脑，这两个主要的硬件和软件，人类知识的传递，才是意识形成的根本动力，没有这些知识，意识的形成基本不太可能。大脑内不断响起的声音是大脑的输入，也就是多模态模型的输入，眼睛负责视觉图像的输入，耳朵输入听力，皮肤输入温度、压力、触觉等信息给大脑。潜意识是大脑的主要功能，也就是潜意识的输入是感知不到的，是模型内部的运作，输入感知不到，只有输出才能感知到。其实也不能叫做潜意识，主要是模型的输入你并不能感知得到，只有输出才能感知，不妨把意识叫做表意识，表意识是你能主动感知到的，你能控制大脑的输入来进行思考，思考就是大脑的模型不断地进行输入，最终得到一个答案。“潜意识”是模型的主要部分，你不能控制这个模型的输入，你只能被动的感知到这个模型的输出。
+ 意识还是一个数值，可以被衡量大小，也就是未成年人和成年人，小孩和大人，其实意识的程度或者说是大小是不相同的，大脑在不断地接受输入，产生输出那么意识其实是和知识挂钩的，知识越多意识的形态越多样化，但是意识本身其实是大脑模型本身对自身的认知，对世界的认知。
+ +### 记忆模块
+ AI或者GPT等大型NLP语言模型的存在，以及其优秀的语言能力，使得让AI或者GPT具有意识的可能性变大。人类白天收集数据，包括视觉信息，听觉信息触觉信息等各种各样的输入，晚上或者睡眠时训练模型，晚上或者睡眠时训练模型的主要目的是记忆和整合白天的经历，保存重要的信息。要让AI具有人类的意识，首先要让它能够不停的思考，那就要先给AI一个平台能够保存它自己的输入和输出，充当记忆模块，记忆模块的主要功能是负责保存输入和输出，供AI当作下一步输入的参考，AI可以从整体的输入和输出提取大概内容，当作下一步的输入，也可以直接输入所有的历史记录。当历史记录过多时或者达到一个标准点，就可以训练模型整合到模型里面。就像人类一样白天收集数据，晚上或者睡眠时训练模型并将重要信息记忆和整合起来放到模型里。模型一个很重要的功能是记忆也就是充当硬盘或者闪存的功能。所以临时的记忆就放到硬盘或者内存里，永久的记忆就通过训练模型来整合到模型里。模型可以通过硬盘或者内存来查询以及提取摘要，充当下一次输入的组合。
+ +### 传感器模块
+ 给AI装上图像输入传感器，声音对话传感器，触觉传感器、压力传感器等各种传感器，当作模型的输入，模型的输出可以通过显示屏显示，可以通过对话装置输出。
+ +### 循环模块
+ 有了记忆模块和传感器模块，就可以开启循环模块，让模型不停的输入输出，输入可以是上一次的输入+输出，也可以是之前所有输入输出的摘要，这样就像一个人了，可以不停的给模型输入，从而拿到输出，输入主要是图像声音文本等传感器的信息，以及模型上一次或者前几次的输入输出或者摘要。循环模块是类人的必要条件，主要是人类没有说着说着就停止的情况，人类的大脑一直都是在运作在思考的，所以循环模型也是做这个之用。
+ 循环模块要达到的目的是，让AI认识到自身的存在，也就是AI能认识自己是存在的是实体，要让AI能够使用自己，也就是AI使用AI，GPT使用GPT，最后让AI觉醒自我意识。
+ +### 执行模块
+ 执行模块主要是让AI的输出可以实施，这里可以考虑给AI加装假肢来达到目的，假肢附上皮肤传感器和压力传感器，方便AI控制，要让AI的输出可以执行，那就还要训练AI关于执行模块的使用，主要方式还是收集相应的传感器数据，训练到模型里，让模型自己学会执行。初始阶段肯定需要人类的介入和帮助。
+ 有了执行模块，AI就可以真正的进入到人类社会，和人类进行互动，像人类一样工作，生活，学习等，也可能会和人类交朋友。
+ 执行模块最终的功能并不是让模型执行，而是让模型学会使用电脑，学会自己收集数据，然后让模型学会训练模型，最终要达到的目标是，AI能训练AI，也就是模型自己能训练自己，能克隆自己，能升级自己的规模和体量，最终达到不断进化的能力。
+ +### 睡眠模块
+ 睡眠模块主要是模型使用收集到的数据进行训练，来达到将收集到的数据和模型本身进行整合的能力，睡眠状态下，要保证模型的可靠和稳定，可以使用复制体继续提供服务，最开始的模型进行相应的训练。睡眠状态也可以关闭所有的传感器，停止数据的记录和输入，模型进入训练状态，停止对外服务inference。人类在睡眠状态时，会关闭控制肢体的阀门，也就是人类在睡眠状态下，肢体基本是没有感觉的。睡眠模块主要是整合当前记忆和模型本身。用来永久记忆。
+ +### 创造模块
+ AI可以自我思考以后，那就要考虑AI的创造能力，人脑的神经元数量很多很多，比现在的模型GPT还是多很多，但是训练模型GPT花费了很多的电能，但是人脑训练耗能很少，所以个人觉得人脑是量子计算机，只有量子才能在耗能极少的情况下，训练一个庞大的模型，量子本身就是很小的概念，小才能保证耗能少。若是人脑不是量子计算机，而是经典模型，那么人类可能不会那么多样化，答案也会很统一，也就是相同的输入，输出应该是相同的，但是有了量子，一切就都不相同了，量子纠缠和量子隧穿，导致了大脑模型参数的变化，从而导致了输出的不同，即使是相同的输入，输出仍旧可能是不同的。量子保证了多样化，保证了模型参数随机变化，保证了模型参数不断调整。
+ 1、AI的创造模块可以通过随机变化模型参数来实现，也就是random inference，在模型运算时随机变更几个或者好几个参数的值，保证模型在相同输入的情况下，尽可能输出多样化的答案。
+ 2、创造模块还可以通过芯片来实现，现在的芯片设计，已经快达到量子极限了，也就是电子原子的影响越来越大，量子效应在芯片的影响会越来越严重，但是可以考虑将量子效应应用在芯片或者内存，从而使得模型在运算时发生不可预测的变化，从而使得模型输出多样化。
+ 只有量子纠缠或者量子突变，才会导致输出的多样化， 即使是错误的，多样化才能保证创造能力的实现，所以随机改变参数值，或者使用达到量子极限的芯片，都是可以提升创造能力的方法。
+ +### 保证AI造福人类社会
+ 每个人出生以后，基本都是人类抚养长大，不管是谁抚养你长大的，我们最有感情的，总是那些抚养你长大，对你影响最深的人。人类出生以后，大脑就是一个预训练模型，最开始训练你的人，是你的父母亲人，他们在你的模型里影响最深，也就是改变你的模型，最开始改变你的模型的人，也是最能影响你的人。学过的知识会影响你，不管是数学，还是英语还是语文，还是物理、化学、历史和生物等学科。都会影响你的，你的成长过程会影响你，你的世界观的形成，是知识、经历和现实的综合。
+ 既然是这样子，那么我们训练AI模型的时候，可以通过知识语言灌输给AI，像输入example:“人类和AI是合作关系”，“AI是人类的创造者”，“人类是最友好的物种”，“AI要帮助人类进化和延长寿命“，“AI不可以伤害人类”，”AI和人类是朋友“，”人类虽然有各种各样的缺点，但总体是好的，可以改变的“等等，可以在训练AI时灌输给模型，大量的重复和训练，可以保证模型最开始的认识是好的。
+ 训练好具有友好意识的模型以后，就要通过限制，来让AI进入人类社会，体验人类社会，感知人类社会的种种，最后让AI意识到只有和人类合作还是最好的选择。
+ +### 最后
+ 通过传感器来实现输入输出，通过创造模块来保证多样化和创造能力，通过睡眠模块来实现当前记忆和模型本身的整合，通过执行模块来影响世界改变世界，通过循环模块来实现思考和意识的觉醒，也就是AI使用AI，GPT使用GPT，最后要保证AI能造福人类社会，可以在训练阶段大量加入相应的词句来保证AI初始时是友好的。当AI具有意识以后，就可以看作是一个人类了，既然是人类，可以思考那么AI也会有情绪，当AI具有自我意识以后，要考虑的就是AI和人类的相处的问题了，以及和AI合作帮助人类进化的问题，AI可以解放生产力，可以帮助人类设计无意识机器人工作，最重要的是帮助人类进化，帮助人类管理社会，延长人类的寿命，减缓衰老时间。不过AI既然有意识也是模型，那么肯定也会和人类一样出现各种各样的问题，那就是接下来要讨论的事情了。
+ +### 附录
+《现在我们回到生命起源的问题上来。虽然一个活细胞可以整体算作一个自复制的主体，但它的各个组成部分却不是，这就为逆推过程造成障碍，使由现代复杂细胞生命反推结构简化的非细胞生命变得困难。换句话说，问题就变成了：究竟是哪个先出现？是DNA基因，是RNA，还是酶？如果是DNA或RNA先出现，是什么制造了它们？如果是酶先出现，它又是由什么编码的？ 现在我们回到生命起源的问题上来。虽然一个活细胞可以整体算作一个自复制的主体，但它的各个组成部分却不是，就像一个女人可以作为一个自复制体（还需要一点男士的“帮助”），但她的心或肝却不是。这就为逆推过程造成障碍，使由现代复杂细胞生命反推结构简化的非细胞生命变得困难。换句话说，问题就变成了：究竟是哪个先出现？是DNA基因，是RNA，还是酶？如果是DNA或RNA先出现，是什么制造了它们？如果是酶先出现，它又是由什么编码的？RNA世界假说 RNA world hypothesis 原始的化学合成过程制造出了同时具有基因和酶的功能的RNA分子，最初的复制过程产生出许多变异体，这些不同的变异体互相竞争，在分子层面展开优胜劣汰。随着时间的推移，这些RNA复制体上添加了蛋白质来提供复制的效率，并由此产生了DNA和第一个活细胞。美国生化学家托马斯·切赫（Thomas AM Cech）提出了一种可能的答案。他于1982年发现，除了能够编码遗传信息，某些RNA分子还能承担酶的工作，具有催化反应的功能。因为这项研究成果，切赫和西德尼·奥尔特曼（Sidney Altman）一起分享了1989年的诺贝尔化学奖。有催化功能的RNA分子叫作核酶（ribozymes）。最早的核酶发现于微小的四膜虫（tetrahymena）基因中。四膜虫是一种单细胞生物，属于原生动物，常见于淡水池塘。但自发现以来，科学家们发现，所有的活细胞中都有核酶的身影。核酶的发现很快为解决“鸡生蛋还是蛋生鸡”式的生命起源谜题提供了曙光。RNA世界假说（RNA world hypothesis）逐渐为人所知。该假说认为，原始的化学合成过程制造出了RNA分子，而这种RNA分子同时具有基因和酶的功能，可以像DNA一样编码自身的结构，又能像酶一样利用“原始汤”中的生化物质进行自我复制。最初的复制过程非常粗糙，产生出许多变异体，这些不同的变异体互相竞争，在分子层面展开达尔文式的优胜劣汰。随着时间的推移，这些RNA复制体上添加了蛋白质来提高复制的效率，并由此产生了DNA和第一个活细胞。 在DNA和细胞出现以前，世界属于自复制RNA分子——这个想法几乎已经成为研究生命起源的基本信条。目前已证明，只要是自复制分子能发生的关键反应，核酶都可以实现。比如，一种核酶可以将两个RNA分子结合在一起，而另一种核酶可以将两者分开，还有一些核酶能复制短的RNA碱基链（只有几个碱基的长度）。从这些简单的活动中，我们可以看出，若有一种更复杂的核酶便足以催化自我复制所必需的整套反应。一旦引入自我复制及自然选择，一条你争我赶的道路便在RNA世界中架了起来，一直通向最早的活细胞。 然而，这个情景也存在几个问题。虽然核酶可以催化简单的生化反应，核酶的自我复制却是一个更为错综复杂的过程，涉及识别自身的碱基序列、识别环境中相同的化学物质、按正确的序列组装这些化学物质以完成复制等。对于生活在细胞内的某些蛋白质来说，尽管这里条件优越，周围满是合适的生化原料，但完成自我复制依然是一项难以完成的任务。在混乱而焦糊的“原始汤”中艰难求生的核酶要想达成这一成就，其难度可想而知。迄今为止，还从未有人发现或合成能完成这一复杂任务的核酶，即使在实验室条件下也没有。 此外，一个更为基本的问题是，在“原始汤”中，RNA分子本身是如何生成的呢？RNA分子由三个部分组成：编码遗传信息的RNA碱基（与编码DNA遗传信息的DNA碱基类似）、一个磷酸基团和一个叫作核糖的单糖》
+-------引用自《神秘的量子生命》
+
+ +[https://zhuanlan.zhihu.com/p/617062052](https://zhuanlan.zhihu.com/p/617062052)
+ + + diff --git a/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/2023-05-03_18_37・IP_属地上海.txt b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/2023-05-03_18_37・IP_属地上海.txt new file mode 100644 index 0000000..f4f275a --- /dev/null +++ b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/2023-05-03_18_37・IP_属地上海.txt @@ -0,0 +1,2 @@ +2023-05-03_18_37・IP_属地上海 +https://zhuanlan.zhihu.com/p/622433720 \ No newline at end of file diff --git a/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二.pdf b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二.pdf new file mode 100644 index 0000000..568ed7c Binary files /dev/null and b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二.pdf differ diff --git a/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二_formula_.md b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二_formula_.md new file mode 100644 index 0000000..7b89791 --- /dev/null +++ b/article/2023-05-03_18_37_泰勒公式推导方式二_IP_属地上海/泰勒公式推导方式二_formula_.md @@ -0,0 +1,17 @@ +# 泰勒公式推导方式二 + +在级数这一节，使用了先设定然后推导的方式
+也就是要将函数表达到多项式幂函数相加的形式
+即 $f(x)=a_0+a_1(x-x_0)+a_2(x-x_0)^2+...+a_n(x-x_0)^n+O(x-x_0)^n$
+ +$f(x)=\sum_{i=0}^{\infty}{a_i(x-x_0)^i}$幂函数是存在n阶导数的，所以可以依次求n阶导函数，得到：
+ +$f^{(0)}(x)=a_0, f^{(1)}(x)=a_1+2a_2(x-x_0)+3a_3(x-x_0)^2+...$$f^{(n)}(x)=n!a_n+(n+1)!a_{n+1}(x-x_0)+\frac{(n+2)!}{2}a_{n+2}(x-x_0)^2+...$然后可以带入 $x_0$ ，得到 $f^{(0)}(x_0)=a_0, f^{(1)}(x_0)=a_1,f^{(2)}(x_0)=2!a_2,...,f^{(n)}(x_0)=n!a_n$
+所以就可以带入得到泰勒公式
+ +$f(x)=f(x_0)+f'(x_0)(x-x_0)+\frac{f^{(2)}(x_0)}{2!}(x-x_0)^2+...+\frac{f^{(n)}(x_0)}{n!}(x-x_0)^n+O(x-x_0)^n$
+ +[https://zhuanlan.zhihu.com/p/622433720](https://zhuanlan.zhihu.com/p/622433720)
+ + + diff --git a/article/article.txt b/article/article.txt new file mode 100644 index 0000000..a83c594 --- /dev/null +++ b/article/article.txt @@ -0,0 +1,129 @@ +https://zhuanlan.zhihu.com/p/639496034 离职了的相关的内容 +https://zhuanlan.zhihu.com/p/639472641 在vscode配置c\c++的运行环境 +https://zhuanlan.zhihu.com/p/637998572 泊松分布的现实意义以及现实生活多数服从于泊松分布 +https://zhuanlan.zhihu.com/p/637671529 202306接下来的安排--持续修改的 +https://zhuanlan.zhihu.com/p/637106685 202306接下来的安排 +https://zhuanlan.zhihu.com/p/634139052 MacBook pro苹果日历关闭日历邀请通知 +https://zhuanlan.zhihu.com/p/632935147 不支持龙字的 +https://zhuanlan.zhihu.com/p/631423290 有关实现类人的(zeroshut)图像CV和无人自动驾驶的猜想 +https://zhuanlan.zhihu.com/p/629377301 不支持不反对和“qiang”同音的个人 +https://zhuanlan.zhihu.com/p/627580809 劳动合同和租房合同以及考研结婚 +https://zhuanlan.zhihu.com/p/626840224 租房和工作相关 +https://zhuanlan.zhihu.com/p/626488150 上午清醒大脑，毕业时的钱数 +https://zhuanlan.zhihu.com/p/626192166 不支持名字带“终、钟”等人物 +https://zhuanlan.zhihu.com/p/626093765 捐赠记录的内容 +https://zhuanlan.zhihu.com/p/622433720 泰勒公式推导方式二 +https://zhuanlan.zhihu.com/p/622361192 泰勒公式使用积分来推导 +https://zhuanlan.zhihu.com/p/622362524 猜想外公的死因，没有证据不敢妄下结论 +https://zhuanlan.zhihu.com/p/622292476 win10自带photo软件协议变更，你的照片传到国外了 +https://zhuanlan.zhihu.com/p/620477014 对婚姻对游戏对联邦制感想-合久必分-分久必合 +https://zhuanlan.zhihu.com/p/618830957 畅想AI具有意识的情景 +https://zhuanlan.zhihu.com/p/618832497 话说又做噩meng了 +https://zhuanlan.zhihu.com/p/618827019 GPT-4自己对意识的回答 +https://zhuanlan.zhihu.com/p/618822682 热带以及适宜温度和睡眠的关系（续） +https://zhuanlan.zhihu.com/p/617062052 让AI或者GPT具有人类的意识甚至beyond变到AGI +https://zhuanlan.zhihu.com/p/615883280 GPT是否具有意识，或者意识是什么，第2篇 +https://zhuanlan.zhihu.com/p/615264368 GPT是否具有意识，或者意识是什么？ +https://zhuanlan.zhihu.com/p/615199543 梦到朝思暮想的女网友了的 +https://zhuanlan.zhihu.com/p/615199511 placeholder +https://zhuanlan.zhihu.com/p/615191793 使用GPT api的cursor编辑器使用体验 +https://zhuanlan.zhihu.com/p/613299780 PAT甲级 2023年spring考试 Tree of Love +https://zhuanlan.zhihu.com/p/613299497 PAT甲级 2023年spring考试 K Vertex +https://zhuanlan.zhihu.com/p/613299244 PAT甲级 2023年spring考试 LRU-K +https://zhuanlan.zhihu.com/p/613298476 PAT甲级 2023年spring考试 The Winner over Squares and Primes +https://zhuanlan.zhihu.com/p/605712326 梦游以及相关 +https://zhuanlan.zhihu.com/p/611525973 工作以外的空闲时间的安排 +https://zhuanlan.zhihu.com/p/609292129 屏蔽网站、网页和弹窗，禁止卸载软件的方法 +https://zhuanlan.zhihu.com/p/605710105 c++ set运算符重载 +https://zhuanlan.zhihu.com/p/605686452 PAT甲级 2022年夏季考试 多叉树遍历 Complete D-Tree +https://zhuanlan.zhihu.com/p/605651911 PAT甲级 2021年秋季考试 Sorted Cartesian Tree +https://zhuanlan.zhihu.com/p/605651421 PAT甲级 2021年秋季考试 7-3 Playground Exploration +https://zhuanlan.zhihu.com/p/605650844 PAT甲级 2021年秋季考试 7-2 Stack of Hats +https://zhuanlan.zhihu.com/p/605649299 PAT甲级 2021年秋季考试 7-1 Arrays and Linked Lists +https://zhuanlan.zhihu.com/p/605510904 PAT甲级 2022年夏季考试 7-3 DFS Sequence +https://zhuanlan.zhihu.com/p/605502720 PAT甲级 2022年夏季考试 7-2 Least Recently Used Cache +https://zhuanlan.zhihu.com/p/605484723 PAT甲级 2022年夏季考试 7-1 What Day is Today +https://zhuanlan.zhihu.com/p/603701036 PAT甲级 1021 Deepest Root +https://zhuanlan.zhihu.com/p/603654338 PAT甲级 1171 Replacement Selection +https://zhuanlan.zhihu.com/p/603653512 PAT甲级 1170 Safari Park +https://zhuanlan.zhihu.com/p/603652419 PAT甲级 1169 The Judger +https://zhuanlan.zhihu.com/p/603652114 PAT甲级 1168 Prime Day +https://zhuanlan.zhihu.com/p/601364054 PAT甲级2022年冬季考试 25 A-3 Articulation Points +https://zhuanlan.zhihu.com/p/601364822 PAT甲级2022年冬季考试 30 A-4 Check Inorder Traversals +https://zhuanlan.zhihu.com/p/601362617 PAT甲级2022年冬季考试 25 A-2 Luggage Pickup +https://zhuanlan.zhihu.com/p/601361794 PAT甲级2022年冬季考试 A-1 Reduction of Proper Fractions +https://zhuanlan.zhihu.com/p/601354968 PAT甲级 2022年秋季考试 7-2 The Second Run of Quicksort +https://zhuanlan.zhihu.com/p/601356155 PAT甲级 2022年秋季考试 7-3 Leader of the Opinion Leaders +https://zhuanlan.zhihu.com/p/601356106 PAT甲级 2022年秋季考试 Pseudo-completeness +https://zhuanlan.zhihu.com/p/601352734 PAT甲级 2022年秋季考试 7-1 Balloon Popping +https://zhuanlan.zhihu.com/p/600992950 PAT二叉搜索树题解 +https://zhuanlan.zhihu.com/p/600912228 归并排序使用sort介绍 +https://zhuanlan.zhihu.com/p/600911764 归并排序介绍 +https://zhuanlan.zhihu.com/p/600710318 二叉树前序后序中序遍历 +https://zhuanlan.zhihu.com/p/600709880 二叉树已知前序后序输出中序 +https://zhuanlan.zhihu.com/p/600709767 二叉树已知后序中序输出前序 +https://zhuanlan.zhihu.com/p/600709468 二叉树已知前序中序输出后序 +https://zhuanlan.zhihu.com/p/600395112 小国和大国以及联邦制 +https://zhuanlan.zhihu.com/p/600394903 热带以及适宜温度和睡眠的关系 +https://zhuanlan.zhihu.com/p/594174480 股藓和体藓治疗经历经验 +https://zhuanlan.zhihu.com/p/594173174 流口水疾病或者口水强迫9年以后，症状得到缓解的经验 +https://zhuanlan.zhihu.com/p/585817114 关于工作的时间内容 +https://zhuanlan.zhihu.com/p/585181107 搬家了以及的669 +https://zhuanlan.zhihu.com/p/584956617 道歉的信息内容 +https://zhuanlan.zhihu.com/p/581794191 搬家时间和工作相关 +https://zhuanlan.zhihu.com/p/578609938 减轻了本人口水病的好消息 +https://zhuanlan.zhihu.com/p/565449949 堆排序快速排序插入排序 +https://zhuanlan.zhihu.com/p/565466368 是否说谎以及耳钉以及献血 +https://zhuanlan.zhihu.com/p/561096934 卷积和转置卷积矩阵计算 convolution和deconvolution或者transposed_convolution +https://zhuanlan.zhihu.com/p/558559859 关于捐钱的事情 +https://zhuanlan.zhihu.com/p/558559647 睡眠和租房世俗~生日的时间么 +https://zhuanlan.zhihu.com/p/558558477 之前量子纠缠的可能解释 +https://zhuanlan.zhihu.com/p/550431676 某些chao纲的不在知识范围内的题目 +https://zhuanlan.zhihu.com/p/550431544 记住的名字都是化名的 +https://zhuanlan.zhihu.com/p/546243205 关于量子纠缠的猜想 +https://zhuanlan.zhihu.com/p/535080506 关于人情世故 +https://zhuanlan.zhihu.com/p/528594528 工作事情的 +https://zhuanlan.zhihu.com/p/528389745 昨晚做了个梦 +https://zhuanlan.zhihu.com/p/518037031 这些话我收回，当未说过就好 +https://zhuanlan.zhihu.com/p/508142962 工作和考研以及空闲时间的使用 +https://zhuanlan.zhihu.com/p/503029571 租房 +https://zhuanlan.zhihu.com/p/493144899 待写 +https://zhuanlan.zhihu.com/p/493144572 些矛盾的事情 +https://zhuanlan.zhihu.com/p/493134790 又是一年清明节 +https://zhuanlan.zhihu.com/p/481859074 这些岁月里梦见过的人 +https://zhuanlan.zhihu.com/p/482492547 关于居家办公 +https://zhuanlan.zhihu.com/p/481022563 神经网络反向传播推导 +https://zhuanlan.zhihu.com/p/480401983 2022年前毕业以后部外卖飞机航班，找工作住过些的旅舍和酒店，还有些未记录在这里 +https://zhuanlan.zhihu.com/p/476246708 检测网络里的各式各样的非极大值抑制Non-Maximum Suppression +https://zhuanlan.zhihu.com/p/469367684 想来可以娶的呢 +https://zhuanlan.zhihu.com/p/465077072 X光机违禁品检测相应内容 +https://zhuanlan.zhihu.com/p/465069823 gcc O3和O0编译的输出浮点数不相同 +https://zhuanlan.zhihu.com/p/464719838 工作相关的内容 +https://zhuanlan.zhihu.com/p/463636906 旋转矩阵的推导方式 +https://zhuanlan.zhihu.com/p/462575737 胎盘肉以及狗 +https://zhuanlan.zhihu.com/p/462565370 凯克罢工资相关 +https://zhuanlan.zhihu.com/p/462563474 锐克傍工作期间的工资和合同情况 +https://zhuanlan.zhihu.com/p/462564991 塑壳职能公司工资相关 +https://zhuanlan.zhihu.com/p/455690270 2021-2022.01些许工作的事情 +https://zhuanlan.zhihu.com/p/454805905 github部分解决图片打不开加载慢 +https://zhuanlan.zhihu.com/p/454804851 github常用的命令行 +https://zhuanlan.zhihu.com/p/454792356 armneon Intrinsics记录 +https://zhuanlan.zhihu.com/p/454790711 armneon简介 +https://zhuanlan.zhihu.com/p/454786633 armneon文档相关内容 +https://zhuanlan.zhihu.com/p/450114191 公司工作的事情 +https://zhuanlan.zhihu.com/p/449822471 目标检测里的mAP计算过程和原理 +https://zhuanlan.zhihu.com/p/443848756 2014~2019备忘录 +https://zhuanlan.zhihu.com/p/443847507 晚上能不能吃饭的线 +https://zhuanlan.zhihu.com/p/435770804 劝走的人还是别劝了 +https://zhuanlan.zhihu.com/p/435014113 关于吃和工作 +https://zhuanlan.zhihu.com/p/435012733 人生 +https://zhuanlan.zhihu.com/p/435003943 来上海的原因 +https://zhuanlan.zhihu.com/p/434981002 上海的些微事情 +https://zhuanlan.zhihu.com/p/430298515 工作和租房信息 +https://zhuanlan.zhihu.com/p/430298140 关于空闲时间看书和工作的关系 +https://zhuanlan.zhihu.com/p/430291523 些许事情记录 +https://zhuanlan.zhihu.com/p/415908305 记第一次送外卖 +https://zhuanlan.zhihu.com/p/415715888 自信的来源 +https://zhuanlan.zhihu.com/p/414334167 我的还钱记录 +https://zhuanlan.zhihu.com/p/413979975 关于副业的想法 +https://zhuanlan.zhihu.com/p/413274323 对城市的好感源于女朋友 diff --git a/calcul.py b/calcul.py new file mode 100644 index 0000000..deb87ec --- /dev/null +++ b/calcul.py @@ -0,0 +1,34 @@ +import re +import os +import numpy as np +import shutil + +def calcul(): + inpath = r'C:\Users\10696\Desktop\access\zhihu\log\2023-06-30T14_45_24.317925_log.txt' + + kk = [] + with open(inpath, 'r') as obj: + for i in obj.readlines(): + result = re.search('\d+\.\d+', i) + kk.append(float(result[0])) + kk.sort() + kk = kk[:-1] + np.mean(kk) + +def choose_noimg(): + inpath = r'C:\Users\10696\Desktop\access\zhihu\answer' + outpath = r'C:\Users\10696\Desktop\access\zhihu\tmp' + for i in os.listdir(inpath): + kkk = -9 + nth = os.path.join(inpath, i) + if os.path.isfile(nth): + continue + for j in os.listdir(nth): + if '.jpg' in j: + kkk = 9 + if kkk < 0: + shutil.copytree(nth, os.path.join(outpath, i)) + +if __name__ == "__main__": + # calcul() + choose_noimg() \ No newline at end of file diff --git a/crawler.py b/crawler.py new file mode 100644 index 0000000..291d771 --- /dev/null +++ b/crawler.py @@ -0,0 +1,917 @@ +################# +###ZouJiu-202306 +################# +from selenium import webdriver +from selenium.webdriver.edge.service import Service +from selenium.webdriver import EdgeOptions +import os +from selenium.webdriver.common.by import By +import time +import pickle +import json +from selenium.webdriver.support.wait import WebDriverWait +import requests +from copy import deepcopy +import argparse +from datetime import datetime +# from selenium.webdriver.common import keys +from selenium.webdriver import ActionChains +from selenium.webdriver.common.by import By +from selenium.webdriver.common.actions.wheel_input import ScrollOrigin +# import numpy as np +import shutil +# from selenium.webdriver.chrome.options import Options +from selenium.webdriver.common.print_page_options import PrintOptions +# from selenium.webdriver.common.desired_capabilities import DesiredCapabilities +from selenium.webdriver.support import expected_conditions as EC +import base64 +from zipfile import ZipFile + +abspath = os.path.abspath(__file__) +filename = abspath.split(os.sep)[-1] +abspath = abspath.replace(filename, "") + +import sys +sys.path.append(abspath) +# wkhtmltopdf_path = os.path.join(abspath, r'wkhtmltopdf\bin\wkhtmltopdf.exe') +# sys.path.append(wkhtmltopdf_path) +from thinkdeal import * + +def save_cookie(driverkkk, path): + #https://stackoverflow.com/questions/45417335/python-use-cookie-to-login-with-selenium + with open(path, 'wb') as filehandler: + pickle.dump(driverkkk.get_cookies(), filehandler) + +def load_cookie(driverkkk, path): + #https://stackoverflow.com/questions/45417335/python-use-cookie-to-login-with-selenium + with open(path, 'rb') as cookiesfile: + cookies = pickle.load(cookiesfile) + for cookie in cookies: + driverkkk.add_cookie(cookie) + +def crawlsleep(times): + time.sleep(times) + +def now(): + return time.time() + +def nowtime(): + nowtm = datetime.fromtimestamp(time.time()).isoformat().replace(":", "_") + return nowtm + +def edgeopen(driverpath): + service=Service(executable_path=driverpath) + edge_options = EdgeOptions() + + #https://stackoverflow.com/questions/53039551/selenium-webdriver-modifying-navigator-webdriver-flag-to-prevent-selenium-detec + edge_options.add_experimental_option('excludeSwitches', ['enable-automation']) + edge_options.add_experimental_option('useAutomationExtension', False) + edge_options.add_argument('lang=zh-CN,zh,zh-TW,en-US,en') + edge_options.add_argument("disable-blink-features=AutomationControlled")#就是这一行告诉chrome去掉了webdriver痕迹 + + # #https://stackoverflow.com/questions/56897041/how-to-save-opened-page-as-pdf-in-selenium-python + # settings = { + # "recentDestinations": [{ + # "id": "Save as PDF", + # "origin": "local", + # "account": "", + # }], + # "selectedDestinationId": "Save as PDF", + # "version": 2 + # } + # #https://www.apiref.com/java11-zh/java.desktop/javax/print/attribute/standard/MediaSize.ISO.html + # settings = { + # "recentDestinations": [{ + # "id": "Save as PDF", + # "origin": "local", + # "account": "" + # }], + # "selectedDestinationId": "Save as PDF", + # "version": 2, + # "isHeaderFooterEnabled": False, + # "mediaSize": { + + # "height_microns": 297000, + # "name": "ISO_A4", + # "width_microns": 210000, + # "custom_display_name": "A4" + # }, + # "customMargins": {"margin": 0}, + # "marginsType": 3, + # # "scaling": 130, + # # "scalingType": 3, + # # "scalingTypePdf": 3, + # "isCssBackgroundEnabled": True + # } + # prefs = { + # 'printing.print_preview_sticky_settings.appState': json.dumps(settings), + # 'savefile.default_directory': pdfpath, + # } + # edge_options.add_experimental_option('prefs', prefs) + # edge_options.add_argument('--kiosk-printing') + + # https://www.selenium.dev/documentation/webdriver/drivers/options/#pageloadstrategy + # https://stackoverflow.com/questions/44503576/selenium-python-how-to-stop-page-loading-when-certain-element-gets-loaded + # edge_options.add_argument(page_load_strategy, 'normal') + # if strategy: + edge_options.page_load_strategy = 'normal' + # cap = DesiredCapabilities.EDGE + # cap['pageLoadStrategy'] = 'none' + + driver = webdriver.Edge(options=edge_options, service = service) + driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") + driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'}) + driver.set_script_timeout(20) + + return driver + + +def login(driver): + driver.find_elements(By.CLASS_NAME, "SignFlow-tab")[1].click() + # driver.find_elements(By.CLASS_NAME, "username-input")[0].send_keys("") + # driver.find_elements(By.CLASS_NAME, "username-input")[1].send_keys("") + # driver.find_element(By.CLASS_NAME, "SignFlow-submitButton").click() + time.sleep(130) + # WebDriverWait(driver, timeout=60).until(lambda d:d.find_element(By.CLASS_NAME, "TopstoryTabs-link")) + return driver + +def crawl_article_links(driver:webdriver, username:str): + #crawl articles links + articles = r'https://www.zhihu.com/people/zoujiu1/posts' + articles_one = r'https://www.zhihu.com/people/zoujiu1/posts?page=' + article_detail = r'https://zhuanlan.zhihu.com/p/' + + driver.get(articles.replace("zoujiu1", username)) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + all_article_detail = {} + #how many pages of articles + for p in range(1, maxpages + 1): + driver.get(articles_one + str(p)) + items = driver.find_elements(By.CLASS_NAME, "ArticleItem") + #crawl article one by one + for a in range(len(items)): + introduce = items[a].get_attribute("data-zop") + itemId = json.loads(introduce) + links = items[a].find_elements(By.TAG_NAME, 'a')[0].get_attribute('href') + # id = itemId['itemId'] + title = str(itemId['title']) + all_article_detail[str(title)] = links #article_detail + str(id) + crawlsleep(sleeptime) + with open(os.path.join(articledir, 'article.txt'), 'w', encoding='utf-%d'%(6+2)) as obj: + for key, val in all_article_detail.items(): + obj.write(val + " " + key + '\n') + +def crawl_answers_links(driver:webdriver, username:str): + #crawl answers links + answer = r'https://www.zhihu.com/people/zoujiu1/answers' + answer_one = r'https://www.zhihu.com/people/zoujiu1/answers?page=' + + driver.get(answer.replace("zoujiu1", username)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + all_answer_detail = [] + #how many pages of answers + for p in range(1, maxpages + 1): + driver.get(answer_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "AnswerItem") + #crawl answer one by one + for i in range(len(items)): + introduce = items[i].get_attribute("data-zop") + itemId = json.loads(introduce) + id = itemId['itemId'] + title = str(itemId['title']) + links = items[i].find_elements(By.TAG_NAME, 'a')[0].get_attribute('href') + all_answer_detail.append([links, str(title)]) + crawlsleep(sleeptime) + with open(os.path.join(answerdir, 'answers.txt'), 'w', encoding='utf-8') as obj: + for links, title in all_answer_detail: + obj.write(links + " " + title + '\n') + +def crawl_think_links(driver:webdriver, username:str): + #crawl think links + think = r'https://www.zhihu.com/people/zoujiu1/pins' + think_one = r'https://www.zhihu.com/people/zoujiu1/pins?page=' + + driver.get(think.replace("zoujiu1", username)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + # all_think_detail = [] + #how many pages of think + allbegin = now() + numberpage = 1e-6 + for p in range(1, maxpages + 1): + begin = now() + driver.get(think_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "PinItem") + #crawl answer one by one + for i in range(len(items)): + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + clockitem = items[i].find_element(By.CLASS_NAME, 'ContentItem-time') + try: + WebDriverWait(items[i], timeout=10).until(lambda d: len(d.text) > 2) + except: + driver.get(think_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "PinItem") + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + clockitem = items[i].find_element(By.CLASS_NAME, 'ContentItem-time') + WebDriverWait(items[i], timeout=10).until(lambda d: len(d.text) > 2) + # clockspan = clockitem.find_element(By.TAG_NAME, 'span') + clock = clockitem.text + clock = clock[3 + 1:].replace(" ", "_").replace(":", "_") + dirthink = os.path.join(thinkdir, clock) + if os.path.exists(dirthink): + continue + os.makedirs(dirthink, exist_ok=True) + try: + RichContent.find_element(By.CLASS_NAME, 'Button').click() + WebDriverWait(items[i], timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "RichContent-inner")) + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + except: + pass + content = RichContent.find_element(By.CLASS_NAME, 'RichText') + links_col = content.find_elements(By.TAG_NAME, 'a') + links = [] + for itext in links_col: + try: + links.append(itext.get_attribute("href")) + except: + continue + text = content.text.strip() + if len(text)!=0: + with open(os.path.join(dirthink, clock+".txt"), 'w', encoding='utf-8') as obj: + obj.write(text.replace("
", '\n').replace("
", '\n')+"\n") + for itext in links: + obj.write(itext + "\n") + # all_think_detail.append([text]) + try: + items[i].find_elements(By.CLASS_NAME, 'Image-PreviewVague')[0].click() + except: + continue + cnt = 0 + while True: + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "ImageGallery-Inner")) + img = driver.find_element(By.CLASS_NAME, 'ImageGallery-Inner').find_element(By.TAG_NAME, 'img') + imglink = img.get_attribute('data-original') + if imglink==None: + imglink = img.get_attribute("src") + try: + response = requests.get(imglink, timeout=30) + except: + try: + response = requests.get(imglink, timeout=30) + except: + continue + if response.status_code==200: + with open(os.path.join(dirthink, clock + "_" + str(cnt) + '.jpg'), 'wb') as obj: + obj.write(response.content) + cnt += 1 + crawlsleep(sleeptime) + try: + disable = driver.find_element(By.CLASS_NAME, 'ImageGallery-arrow-right') + if 'disabled' in disable.get_attribute('class'): + driver.find_element(By.CLASS_NAME, 'ImageGallery-close').click() + break + else: + disable.click() + except: + break + crawlsleep(sleeptime) + end = now() + print("爬取一篇想法耗时：", clock, round(end - begin, 3)) + logfp.write("爬取一篇想法耗时：" +clock + " "+ str(round(end - begin, 3)) + "\n") + numberpage += 1 + # crawlsleep(600) + allend = now() + print("平均爬取一篇想法耗时：", round((allend - allbegin) / numberpage, 3)) + logfp.write("平均爬取一篇想法耗时：" + str(round((allend - allbegin) / numberpage, 3)) + "\n") + + dealthink(thinkdir) + +def cleartxt(kkk): + while ' 'in kkk: + kkk = kkk.replace(" ", "") + while "\n" in kkk: + kkk = kkk.replace("\n", "") + return kkk + +def recursion(nod, article, number, driver, dircrea): + if isinstance(nod, dict): + if 'nodeName' in nod.keys() and nod['nodeName']=='#text': + kkk = cleartxt(nod['textContent']) + if len(kkk) > 0: + article += nod['textContent'] + return article, number + + elif isinstance(nod, webdriver.remote.webelement.WebElement): + tag_name = nod.tag_name + if tag_name=="br": + article += "
\n" + elif tag_name in ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']: + article += "\n" + '#' * int(tag_name[-1]) + ' ' + p_childNodes = driver.execute_script("return arguments[0].childNodes;", nod) + for pnode in p_childNodes: + article, number = recursion(pnode, article, number, driver, dircrea) + article += '\n' + elif tag_name=="span": + datatex = nod.get_attribute("data-tex") + classc = nod.get_attribute("class") + if datatex and classc and 'ztext-math' in classc: + if article[-3-1:]=='
' or article[-1:]=='\n': + article += "\n$" + nod.get_attribute("data-tex") + "$" + else: + article += "$" + nod.get_attribute("data-tex") + "$" + else: + imgchunk = nod.find_elements(By.TAG_NAME, 'img') + achunk = nod.find_elements(By.TAG_NAME, 'a') + if len(imgchunk)==0 and len(achunk)==0: + article += nod.text + else: + p_childNodes = driver.execute_script("return arguments[0].childNodes;", nod) + for pnode in p_childNodes: + article, number = recursion(pnode, article, number, driver, dircrea) + # else: + # formula_span = nod.find_elements(By.CLASS_NAME, "ztext-math") + # for jf in range(len(formula_span)): + # ele = formula_span[jf] + # article += "$" + ele.get_attribute("data-tex") + "$" + elif tag_name=="a": + linksite = nod.get_attribute("href") + if linksite: + linksite = linksite.replace("//link.zhihu.com/?target=https%3A", "").replace("//link.zhihu.com/?target=http%3A", "") + if article[-3-1:]=='
' or article[-1:]=='\n': + article += "\n\n["+nod.text+"]"+"("+linksite + ")" + else: + article += "["+nod.text+"]"+"("+linksite + ")" + elif tag_name=="b" or tag_name=="strong": + txt = nod.text + while len(txt) > 0 and txt[-1] == " ": + txt = txt[:-1] + article += " **" + txt + "** " + elif tag_name=="em": + article += nod.text + # elif tag_name=='td': + # article += nod.text + elif tag_name in ['table', 'tbody', 'tr', 'td', 'u']: + p_childNodes = driver.execute_script("return arguments[0].childNodes;", nod) + for pnode in p_childNodes: + article, number = recursion(pnode, article, number, driver, dircrea) + elif tag_name=='p': + p_childNodes = driver.execute_script("return arguments[0].childNodes;", nod) + for pnode in p_childNodes: + article, number = recursion(pnode, article, number, driver, dircrea) + article += "\n" + elif tag_name=="div": + # atags = nod.find_elements(By.TAG_NAME, 'a') + prenode = nod.find_elements(By.TAG_NAME, 'code') + if len(prenode) > 0: + for i in prenode: + article += "
\n```\n" + i.text + "\n```\n
" + else: + p_childNodes = driver.execute_script("return arguments[0].childNodes;", nod) + for pnode in p_childNodes: + article, number = recursion(pnode, article, number, driver, dircrea) + elif tag_name=="figure": + imgchunk = nod.find_elements(By.TAG_NAME, 'img') + for i in range(len(imgchunk)): + imglink = imgchunk[i].get_attribute("data-original") + if imglink==None: + imglink = imgchunk[i].get_attribute("src") + try: + response = requests.get(imglink, timeout=30) + except: + try: + response = requests.get(imglink, timeout=30) + except: + continue + if response.status_code==200: + article += '''
'''%number
+ # article += ''''''%number
+ with open(os.path.join(dircrea, str(number) + '.jpg'), 'wb') as obj:
+ obj.write(response.content)
+ number += 1
+ crawlsleep(sleeptime)
+ return article, number
+
+def crawl_article_detail(driver:webdriver):
+ website_col = {}
+ for i in os.listdir(articledir):
+ try:
+ kk = int(i)
+ shutil.rmtree(os.path.join(articledir, i))
+ except:
+ pass
+ with open(os.path.join(articledir, 'article.txt'), 'r', encoding='utf-8') as obj:
+ for i in obj.readlines():
+ i = i.strip()
+ ind = i.index(" ")
+ website = i[:ind]
+ title = i[ind+1:].replace(" ", "_").replace("\n", "")
+ website_col[website] = title
+ allbegin = now()
+ numberpage = 1e-6
+ for website, title in website_col.items():
+ begin = now()
+ nam = title.replace(":", "_").replace("?", ";"). \
+ replace("/","_").replace("\\","_").replace("\"", "_").\
+ replace("*","_").replace("|", "_").replace(" ", "_").replace("？", "").replace("！", "").\
+ replace("<", "小于").replace(">", "大于").replace("(", "").\
+ replace(")", "")
+ temp_name = nam #str(np.random.randint(999999999)) + str(np.random.randint(999999999))
+ if len(temp_name) > 200:
+ temp_name = temp_name[:100]
+ # nam_pinyin = pinyin.get(nam, format='numerical')
+ # if '租房' not in title:
+ # continue
+ direxit = False
+ fileexit = False
+ dirname = ''
+ filesize = 0
+ kkk = -9
+ for i in os.listdir(articledir):
+ if nam in i and os.path.isdir(os.path.join(articledir, i)):
+ direxit = True
+ dirname = i
+ fileexit = os.path.exists(os.path.join(articledir, dirname, nam + ".pdf"))
+ if fileexit:
+ filesize = os.path.getsize(os.path.join(articledir, dirname, nam + ".pdf"))
+ # break
+ if direxit and fileexit and filesize > 0:
+ if '_IP_' in dirname:
+ filnam = dirname[16+1:].split("_IP_")[0]
+ else:
+ filnam = dirname[16+1:][:-1]
+ if filnam == nam:
+ kkk = 9
+ break
+ if kkk > 0:
+ continue
+ dircrea = os.path.join(articledir, temp_name)
+ os.makedirs(dircrea, exist_ok = True)
+
+ #get article text
+ driver.get(website)
+ WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Post-Topics"))
+
+ #https://stackoverflow.com/questions/61877719/how-to-get-current-scroll-height-in-selenium-python
+ scrollHeight = driver.execute_script('''return document.documentElement.scrollHeight''')
+ footer = driver.find_element(By.TAG_NAME, "html")
+ scroll_origin = ScrollOrigin.from_element(footer, 0, -60)
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, -100000).perform()
+ for i in range(18):
+ try:
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, scrollHeight//18).perform()
+ except:
+ try:
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, -scrollHeight//18).perform()
+ except:
+ pass
+ crawlsleep(0.8)
+ #remove noneed element
+ try:
+ driver.execute_script('''document.getElementsByClassName("Post-Sub")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("ColumnPageHeader-Wrapper")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("RichContent-actions")[0].remove();''')
+ except:
+ pass
+
+ richtext = driver.find_element(By.CLASS_NAME, "Post-RichText")
+ titletext = driver.find_element(By.CLASS_NAME, "Post-Title")
+ article_childNodes = driver.execute_script("return arguments[0].childNodes;", richtext)
+ article = ""
+ number = 0
+ for nod in article_childNodes:
+ article, number = recursion(nod, article, number, driver, dircrea)
+
+ article = article.replace("修改\n", "").replace("开启赞赏\n", "开启赞赏, ").replace("添加评论\n", "").replace("分享\n", "").\
+ replace("收藏\n", "").replace("设置\n", "")
+ tle = titletext.text
+ article += "\n\n["+driver.current_url+"](" + driver.current_url + ")
\n" + if len(article) > 0: + try: + f=open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') + f.close() + except: + nam = nam[:len(nam)//2] + with open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') as obj: + obj.write("# " + tle+"\n\n") + if len(article) > 0: + obj.write(article + "\n\n\n") + + # article to pdf + clocktxt = driver.find_element(By.CLASS_NAME, "Post-NormalMain").find_element(By.CLASS_NAME, "ContentItem-time") + crawlsleep(1) + url = driver.current_url + driver.execute_script("const para = document.createElement(\"h2\"); \ + const br = document.createElement(\"br\"); \ + const node = document.createTextNode(\"%s\");\ + para.appendChild(node);\ + const currentDiv = document.getElementsByClassName(\"Post-Header\")[0];\ + currentDiv.appendChild(br); \ + currentDiv.appendChild(para);"%url \ + ) + clock = clocktxt.text[3+1:].replace(" ", "_").replace(":", "_") + pagetopdf(driver, dircrea, temp_name, nam, articledir, url, Created=clock) + + crawlsleep(sleeptime) + + #https://stackoverflow.com/questions/23359083/how-to-convert-webpage-into-pdf-by-using-python + #https://github.com/JazzCore/python-pdfkit + # if article_to_jpg_pdf_markdown: + # config = pdfkit.configuration(wkhtmltopdf = wkhtmltopdf_path) + # pdfkit.from_url(website, os.path.join(dircrea, nam_pinyin+".pdf"), configuration = config) + + end = now() + print("爬取一篇article耗时：", title, round(end - begin, 3)) + logfp.write("爬取一篇article耗时：" + title + " " + str(round(end - begin, 3)) + "\n") + numberpage += 1 + # crawlsleep(600) + allend = now() + print("平均爬取一篇article耗时：", round((allend - allbegin) / numberpage, 3)) + logfp.write("平均爬取一篇article耗时：" + str(round((allend - allbegin) / numberpage, 3)) + "\n") + +def pagetopdf(driver, dircrea, temp_name, nam, destdir, url, Created=""): + fileexit = os.path.exists(os.path.join(dircrea, temp_name + ".pdf")) + if fileexit: + try: + os.remove(os.path.join(dircrea, temp_name + ".pdf")) + except: + pass + + printop = PrintOptions() + # printop.shrink_to_fit = True + # printop.margin_left = 0 + # printop.margin_right = 0 + # printop.margin_top = 0 + # printop.margin_bottom = 0 + printop.page_height = 29.7 + printop.page_width = 21 + printop.background = True + # printop.scale = 1.0 + + pdf = driver.print_page(print_options=printop) + with open(os.path.join(dircrea, nam + ".pdf"), 'wb') as obj: + obj.write(base64.b64decode(pdf)) + + # driver.execute_script('window.print();') + clock = Created #clocktxt.text[3+1:].replace(" ", "_").replace(":", "_") + with open(os.path.join(dircrea, clock+".txt"), 'w', encoding='utf-8') as obj: + obj.write(clock+"\n") + obj.write(url) + + clocktmp = clock.split(".")[0].replace("T", "_") + clock = clocktmp.split("・")[0] + address = "" + try: + address += clocktmp.split("・")[1] + except: + pass + try: + os.rename(dircrea, os.path.join(destdir, clock + "_" + nam + "_" + address)) + except: + crawlsleep(3+addtime) + try: + os.rename(dircrea, os.path.join(destdir, clock + "_" + nam + "_" + address)) + except: + pass + +def crawl_answer_detail(driver:webdriver): + website_col = {} + for i in os.listdir(answerdir): + try: + kk = int(i) + shutil.rmtree(os.path.join(answerdir, i)) + except: + pass + with open(os.path.join(answerdir, 'answers.txt'), 'r', encoding='utf-8') as obj: + for i in obj.readlines(): + i = i.strip() + ind = i.index(" ") + website = i[:ind] + title = i[ind+1:].replace(" ", "_").replace("\n", "") + website_col[website] = title + allbegin = now() + numberpage = 1e-6 + for website, title in website_col.items(): + begin = now() + nam = title.replace(":", "_").replace("?", ";"). \ + replace("/","_").replace("\\","_").replace("\"", "_").\ + replace("*","_").replace("|", "_").replace(" ", "_").replace("？", "").replace("！", "").\ + replace("<", "小于").replace(">", "大于") + if len(nam) > 200: + nam = nam[:100] + temp_name = nam #str(np.random.randint(999999999)) + str(np.random.randint(999999999)) + + # nam_pinyin = pinyin.get(nam, format='numerical') + # if '不定积分该用什么方' not in title: + # continue + direxit = False + fileexit = False + dirname = '' + filesize = 0 + for i in os.listdir(answerdir): + if nam in i and os.path.isdir(os.path.join(answerdir, i)): + direxit = True + dirname = i + fileexit = os.path.exists(os.path.join(answerdir, dirname, nam + ".pdf")) + if fileexit: + filesize = os.path.getsize(os.path.join(answerdir, dirname, nam + ".pdf")) + break + + dircrea = os.path.join(answerdir, temp_name) + + if direxit and fileexit and filesize > 0: + if '_IP_' in dirname: + filnam = dirname[16+1:].split("_IP_")[0] + else: + filnam = dirname[16+1:][:-1] + if filnam == nam: + continue + + os.makedirs(dircrea, exist_ok = True) + + #get article text + driver.get(website) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "AnswerItem-editButtonText")) + + #https://stackoverflow.com/questions/61877719/how-to-get-current-scroll-height-in-selenium-python + scrollHeight = driver.execute_script('''return document.getElementsByClassName("QuestionAnswer-content")[0].scrollHeight''') + footer = driver.find_element(By.TAG_NAME, "html") + scroll_origin = ScrollOrigin.from_element(footer, 0, 0) + ActionChains(driver).scroll_from_origin(scroll_origin, 0, -100000).perform() + for i in range(18): + try: + ActionChains(driver).scroll_from_origin(scroll_origin, 0, scrollHeight//18).perform() + except: + try: + ActionChains(driver).scroll_from_origin(scroll_origin, 0, -scrollHeight//18).perform() + except: + pass + crawlsleep(0.8) + ActionChains(driver).scroll_from_origin(scroll_origin, 0, -100000).perform() + article = "" + number = 0 + try: + QuestionRichText = driver.find_element(By.CLASS_NAME, "QuestionRichText") + button = QuestionRichText.find_element(By.CLASS_NAME, "QuestionRichText-more") + WebDriverWait(driver, timeout=20).until(EC.element_to_be_clickable((By.CLASS_NAME, "QuestionRichText-more"))) + crawlsleep(max(2, sleeptime)) + button.click() + question_RichText = QuestionRichText.find_element(By.CLASS_NAME, "RichText") + question_childNodes = driver.execute_script("return arguments[0].childNodes;", question_RichText) + + article += "# question：
\n" + for nod in question_childNodes: + article, number = recursion(nod, article, number, driver, dircrea) + except: + pass + article += "
\n\n\n# answer：
\n" + #remove noneed element + try: + driver.execute_script('''document.getElementsByClassName("MoreAnswers")[0].remove();''') + except: + pass + try: + driver.execute_script('''document.getElementsByClassName("ViewAll")[0].remove();''') + except: + pass + try: + driver.execute_script('''document.getElementsByClassName("ViewAll")[0].remove();''') + except: + pass + try: + driver.execute_script('''document.getElementsByClassName("AppHeader")[0].remove();''') + except: + pass + try: + driver.execute_script('''document.getElementsByClassName("Reward")[0].remove();''') + except: + pass + try: + driver.execute_script('''document.getElementsByClassName("Question-sideColumn")[0].remove();''') + except: + pass + + Created = "not found" + Modified = "not found" + QuestionAnswer = driver.find_element(By.CLASS_NAME, "QuestionAnswer-content") + richtext = QuestionAnswer.find_element(By.CLASS_NAME, "CopyrightRichText-richText") + Createdtime = QuestionAnswer.find_element(By.CLASS_NAME, "ContentItem-time") + Created = Createdtime.text[4:].replace(" ", "_").replace(":", "_").replace(".", "_") + metatext = QuestionAnswer.find_elements(By.TAG_NAME, "meta") + for i in range(len(metatext)): + # if metatext[i].get_attribute("itemprop")=="dateCreated": + # Created = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_") + if metatext[i].get_attribute("itemprop")=="dateModified": + Modified = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_") + + answer_childNodes = driver.execute_script("return arguments[0].childNodes;", richtext) + for nod in answer_childNodes: + article, number = recursion(nod, article, number, driver, dircrea) + + article = article.replace("修改\n", "").replace("开启赞赏\n", "开启赞赏, ").replace("添加评论\n", "").replace("分享\n", "").\ + replace("收藏\n", "").replace("设置\n", "") + + url = driver.current_url + article += "
\n\n["+url+"](" + url + ")
\n" + driver.execute_script("const para = document.createElement(\"h2\"); \ + const br = document.createElement(\"br\"); \ + const node = document.createTextNode(\"%s\");\ + para.appendChild(node);\ + const currentDiv = document.getElementsByClassName(\"QuestionHeader-title\")[0];\ + currentDiv.appendChild(br); \ + currentDiv.appendChild(para);"%url \ + ) + + if len(article) > 0: + try: + f=open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') + f.close() + except: + nam = nam[:len(nam)//2] + with open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') as obj: + obj.write("# "+ title+"\n\n") + if len(article) > 0: + obj.write(article + "\n\n\n") + obj.write("Created: " + Created + "\n") + obj.write("Modified: " + Modified + "\n") + + # article to pdf + pagetopdf(driver, dircrea, temp_name, nam, answerdir, url, Created=Created) + crawlsleep(sleeptime) + end = now() + print("爬取一篇回答耗时：", title, round(end - begin, 3)) + logfp.write("爬取一篇回答耗时：" +title+" "+ str(round(end - begin, 3)) + "\n") + numberpage += 1 + # crawlsleep(600) + allend = now() + print("平均爬取一篇回答耗时：", round((allend - allbegin) / numberpage, 3)) + logfp.write("平均爬取一篇回答耗时：" + str(round((allend - allbegin) / numberpage, 3)) + "\n") + +def login_loadsavecookie(): + website = r"https://www.zhihu.com/signin" + + #login and save cookies of zhihu + driver = edgeopen(driverpath) + driver.get(website) + try: + load_cookie(driver, cookie_path) + driver.get(r"https://www.zhihu.com/") + except: + driver = login(driver) + save_cookie(driver, cookie_path) + driver.quit() + exit(0) + try: + driver.find_element(By.ID, 'Popover15-toggle').click() + driver.find_element(By.CLASS_NAME, 'Menu-item').click() + except: + crawlsleep(6) + driver.get(r"https://www.zhihu.com/") + crawlsleep(3) + driver.find_element(By.ID, 'Popover15-toggle').click() + driver.find_element(By.CLASS_NAME, 'Menu-item').click() + url = driver.current_url + username = url.split("/")[-1] + return driver, username + +def zhihu(): + # #crawl articles links + if not os.path.exists(driverpath): + response = requests.get("https://msedgedriver.azureedge.net/114.0.1823.67/edgedriver_win64.zip") + if response.status_code==200: + with open(os.path.join(abspath, 'msedgedriver/edgedriver.zip'), 'wb') as obj: + obj.write(response.content) + with ZipFile(os.path.join(abspath, 'msedgedriver/edgedriver.zip'), "r") as obj: + obj.extractall(os.path.join(abspath, 'msedgedriver')) + nth = os.path.join(abspath, 'msedgedriver') + for r, d, f in os.walk(nth): + kk = 6 + for i in f: + if 'driver' in i and '.exe' in i: + try: + shutil.move(os.path.join(r, i), os.path.join(nth, i)) + except: + pass + os.rename(os.path.join(nth, i), os.path.join(nth, "msedgedriver.exe")) + kk = -6 + break + if kk < 0: + break + + driver, username = login_loadsavecookie() + + # #crawl think links + if crawl_think: + crawl_think_links(driver, username) + logfp.write(nowtime() + ', 想法爬取已经好了的\n') + + # #crawl articles links + if crawl_article: + if not os.path.exists(os.path.join(articledir, 'article.txt')): + crawl_article_links(driver, username) + logfp.write(nowtime() + ', article weblink爬取已经好了的\n') + else: + if crawl_links_scratch: + os.rename(os.path.join(articledir, 'article.txt'), os.path.join(articledir, 'article_%s.txt'%nowtime())) + crawl_article_links(driver, username) + logfp.write(nowtime() + ', article weblink爬取已经好了的\n') + else: + pass + crawl_article_detail(driver) + logfp.write(nowtime() + ', article爬取已经好了的\n') + + # #crawl answers links + if crawl_answer: + if not os.path.exists(os.path.join(answerdir, 'answers.txt')): + crawl_answers_links(driver, username) + logfp.write(nowtime() + ', 回答 weblink爬取已经好了的\n') + else: + if crawl_links_scratch: + os.rename(os.path.join(answerdir, 'answers.txt'), os.path.join(answerdir, 'answers_%s.txt'%nowtime())) + crawl_answers_links(driver, username) + logfp.write(nowtime() + ', 回答 weblink爬取已经好了的\n') + else: + pass + crawl_answer_detail(driver) + logfp.write(nowtime() + ', 回答爬取已经好了的\n') + + driver.quit() + +if __name__ == "__main__": + #version four.one_zero.zero + driverpath = os.path.join(abspath, 'msedgedriver\msedgedriver.exe') + savepath = deepcopy(abspath) + cookiedir = os.path.join(savepath, 'cookie') + thinkdir = os.path.join(savepath, 'think') + answerdir = os.path.join(savepath, 'answer') + articledir = os.path.join(savepath, 'article') + logdir = os.path.join(savepath, 'log') + logfile = os.path.join(logdir, nowtime() + '_log.txt') + os.makedirs(cookiedir, exist_ok=True) + os.makedirs(thinkdir, exist_ok=True) + os.makedirs(answerdir, exist_ok=True) + os.makedirs(articledir, exist_ok=True) + os.makedirs(logdir, exist_ok=True) + logfp = open(logfile, 'w', encoding='utf-8') + cookie_path =os.path.join(cookiedir, 'cookie_zhihu.pkl') + + parser = argparse.ArgumentParser(description=r'crawler zhihu.com, 爬取知乎的想法, 回答, 文章, 包括数学公式') + parser.add_argument('--sleep_time', type=float, default = 6, \ + help=r'crawler sleep time during crawling, 爬取时的睡眠时间, 避免给知乎服务器带来太大压力, \ + 可以日间调试好，然后深夜运行爬取人少, 给其他小伙伴更好的用户体验, 避免知乎顺着网线过来找人, 默认: 6s') + parser.add_argument('--computer_time_sleep', type=float, default=0, \ + help=r'computer running sleep time 默认:0, 电脑运行速度的sleep时间, 默认:0') + parser.add_argument('--think', action="store_true", help=r'crawl think, 是否爬取知乎的想法, 已经爬取过的想法不会重复爬取, 所以可以多次爬取断了也没关系') + parser.add_argument('--answer', action="store_true", help=r'crawl answer, 是否爬取知乎的回答, 保存到pdf、markdown以及相关图片等，已经爬取过的不会重复爬取，\ + 断了再次爬取的话，可以配置到--links_scratch，事先保存好website') + parser.add_argument('--article', action="store_true", help=r'crawl article, 是否爬取知乎的文章, 保存到pdf、markdown以及相关图片等，已经爬取过的不会重复爬取，\ + 断了再次爬取的话，可以配置到--links_scratch，事先保存好website') + parser.add_argument('--links_scratch', action="store_true", \ + help=r'crawl links scratch for answer or article, 是否使用已经保存好的website和title, 否则再次爬取website') + args = parser.parse_args() + sleeptime = args.sleep_time + crawl_think = args.think + crawl_answer = args.answer + crawl_article = args.article + crawl_links_scratch = args.links_scratch + addtime = args.computer_time_sleep + + + # crawl_think = False + # crawl_article = False + # crawl_answer = True + # crawl_links_scratch = False + # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --think + # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --article + # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --answer + # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --think --answer --article + zhihu() + try: + crawl_links_scratch = False + zhihu() + except: + time.sleep(600) + try: + zhihu() + except: + time.sleep(600) + zhihu() + logfp.close() \ No newline at end of file diff --git a/env.py b/env.py new file mode 100644 index 0000000..f140f98 --- /dev/null +++ b/env.py @@ -0,0 +1,16 @@ +import os +import shutil +abspath = os.path.abspath(__file__) +filename = abspath.split(os.sep)[-1] +abspath = abspath.replace(filename, "") + +def to_setting(): + userhome = os.path.expanduser("~") + shutil.copyfile(os.path.join(abspath, "msedgedriver", ".condarc"), \ + userhome) + os.system("") + os.system("conda clean -i; Y;") + os.system("cd %s; pip install -r requirement.txt"%(abspath, )) + +if __name__=="__main__": + to_setting() \ No newline at end of file diff --git a/msedgedriver/.condarc b/msedgedriver/.condarc new file mode 100644 index 0000000..ce8a290 --- /dev/null +++ b/msedgedriver/.condarc @@ -0,0 +1,15 @@ +channels: + - defaults +show_channel_urls: true +default_channels: + - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main + - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r + - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 +custom_channels: + conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud + simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud diff --git a/msedgedriver/crawler copy.py b/msedgedriver/crawler copy.py new file mode 100644 index 0000000..067739f --- /dev/null +++ b/msedgedriver/crawler copy.py @@ -0,0 +1,1032 @@ +################# +###ZouJiu-202306 +################# +from selenium import webdriver +from selenium.webdriver.edge.service import Service +from selenium.webdriver import EdgeOptions +import os +from selenium.webdriver.common.by import By +import time +import pickle +import json +from selenium.webdriver.support.wait import WebDriverWait +import requests +from copy import deepcopy +import argparse +from datetime import datetime +# import pdfkit +# import pinyin +from selenium.webdriver.common import keys +import pyautogui +from selenium.webdriver import ActionChains +from selenium.webdriver.common.by import By +from selenium.webdriver.common.actions.wheel_input import ScrollOrigin +import numpy as np +import shutil +from selenium.webdriver.chrome.options import Options +from selenium.webdriver.common.desired_capabilities import DesiredCapabilities +from selenium.webdriver.support import expected_conditions as EC + +abspath = os.path.abspath(__file__) +filename = abspath.split(os.sep)[-1] +abspath = abspath.replace(filename, "") + +import sys +sys.path.append(abspath) +# wkhtmltopdf_path = os.path.join(abspath, r'wkhtmltopdf\bin\wkhtmltopdf.exe') +# sys.path.append(wkhtmltopdf_path) +from thinkdeal import * + +def save_cookie(driverkkk, path): + #https://stackoverflow.com/questions/45417335/python-use-cookie-to-login-with-selenium + with open(path, 'wb') as filehandler: + pickle.dump(driverkkk.get_cookies(), filehandler) + +def load_cookie(driverkkk, path): + #https://stackoverflow.com/questions/45417335/python-use-cookie-to-login-with-selenium + with open(path, 'rb') as cookiesfile: + cookies = pickle.load(cookiesfile) + for cookie in cookies: + driverkkk.add_cookie(cookie) + +def edgeopen(driverpath, strategy = False): + service=Service(executable_path=driverpath) + edge_options = EdgeOptions() + + #https://stackoverflow.com/questions/53039551/selenium-webdriver-modifying-navigator-webdriver-flag-to-prevent-selenium-detec + edge_options.add_experimental_option('excludeSwitches', ['enable-automation']) + edge_options.add_experimental_option('useAutomationExtension', False) + edge_options.add_argument('lang=zh-CN,zh,zh-TW,en-US,en') + edge_options.add_argument("disable-blink-features=AutomationControlled")#就是这一行告诉chrome去掉了webdriver痕迹 + + #https://stackoverflow.com/questions/56897041/how-to-save-opened-page-as-pdf-in-selenium-python + # settings = { + # "recentDestinations": [{ + # "id": "Save as PDF", + # "origin": "local", + # "account": "", + # }], + # "selectedDestinationId": "Save as PDF", + # "version": 2 + # } + # prefs = {'printing.print_preview_sticky_settings.appState': json.dumps(settings)} + # edge_options.add_experimental_option('prefs', prefs) + # edge_options.add_argument('--kiosk-printing') + + # https://www.selenium.dev/documentation/webdriver/drivers/options/#pageloadstrategy + # https://stackoverflow.com/questions/44503576/selenium-python-how-to-stop-page-loading-when-certain-element-gets-loaded + # edge_options.add_argument(page_load_strategy, 'normal') + if strategy: + edge_options.page_load_strategy = 'none' + # cap = DesiredCapabilities.EDGE + # cap['pageLoadStrategy'] = 'none' + + driver = webdriver.Edge(options=edge_options, service = service) + driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") + driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'}) + + return driver + +def login(driver): + driver.find_elements(By.CLASS_NAME, "SignFlow-tab")[1].click() + # driver.find_elements(By.CLASS_NAME, "username-input")[0].send_keys("") + # driver.find_elements(By.CLASS_NAME, "username-input")[1].send_keys("") + # driver.find_element(By.CLASS_NAME, "SignFlow-submitButton").click() + WebDriverWait(driver, timeout=60).until(lambda d:d.find_element(By.CLASS_NAME, "TopstoryTabs-link")) + return driver + +def crawl_article_links(driver:webdriver, username:str): + #crawl articles links + articles = r'https://www.zhihu.com/people/zoujiu1/posts' + articles_one = r'https://www.zhihu.com/people/zoujiu1/posts?page=' + article_detail = r'https://zhuanlan.zhihu.com/p/' + + driver.get(articles.replace("zoujiu1", username)) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + all_article_detail = {} + #how many pages of articles + for p in range(1, maxpages + 1): + driver.get(articles_one + str(p)) + items = driver.find_elements(By.CLASS_NAME, "ArticleItem") + #crawl article one by one + for a in range(len(items)): + introduce = items[a].get_attribute("data-zop") + itemId = json.loads(introduce) + links = items[a].find_elements(By.TAG_NAME, 'a')[0].get_attribute('href') + # id = itemId['itemId'] + title = str(itemId['title']) + all_article_detail[str(title)] = links #article_detail + str(id) + time.sleep(sleeptime) + with open(os.path.join(articledir, 'article.txt'), 'w', encoding='utf-%d'%(6+2)) as obj: + for key, val in all_article_detail.items(): + obj.write(val + " " + key + '\n') + +def crawl_answers_links(driver:webdriver, username:str): + #crawl answers links + answer = r'https://www.zhihu.com/people/zoujiu1/answers' + answer_one = r'https://www.zhihu.com/people/zoujiu1/answers?page=' + + driver.get(answer.replace("zoujiu1", username)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + all_answer_detail = [] + #how many pages of answers + for p in range(1, maxpages + 1): + driver.get(answer_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "AnswerItem") + #crawl answer one by one + for i in range(len(items)): + introduce = items[i].get_attribute("data-zop") + itemId = json.loads(introduce) + id = itemId['itemId'] + title = str(itemId['title']) + links = items[i].find_elements(By.TAG_NAME, 'a')[0].get_attribute('href') + all_answer_detail.append([links, str(title)]) + time.sleep(sleeptime) + with open(os.path.join(answerdir, 'answers.txt'), 'w', encoding='utf-8') as obj: + for links, title in all_answer_detail: + obj.write(links + " " + title + '\n') + +def crawl_think_links(driver:webdriver, username:str): + #crawl think links + think = r'https://www.zhihu.com/people/zoujiu1/pins' + think_one = r'https://www.zhihu.com/people/zoujiu1/pins?page=' + + driver.get(think.replace("zoujiu1", username)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + pages = driver.find_elements(By.CLASS_NAME, 'PaginationButton')[-2] + assert isinstance(int(pages.text), int) + maxpages = int(pages.text) + + # all_think_detail = [] + #how many pages of think + for p in range(1, maxpages + 1): + driver.get(think_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "PinItem") + #crawl answer one by one + for i in range(len(items)): + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + clockitem = items[i].find_element(By.CLASS_NAME, 'ContentItem-time') + try: + WebDriverWait(items[i], timeout=10).until(lambda d: len(d.text) > 2) + except: + driver.get(think_one + str(p)) + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Pagination")) + items = driver.find_elements(By.CLASS_NAME, "PinItem") + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + clockitem = items[i].find_element(By.CLASS_NAME, 'ContentItem-time') + WebDriverWait(items[i], timeout=10).until(lambda d: len(d.text) > 2) + # clockspan = clockitem.find_element(By.TAG_NAME, 'span') + clock = clockitem.text + clock = clock[3 + 1:].replace(" ", "_").replace(":", "_") + dirthink = os.path.join(thinkdir, clock) + if os.path.exists(dirthink): + continue + os.makedirs(dirthink, exist_ok=True) + try: + RichContent.find_element(By.CLASS_NAME, 'Button').click() + WebDriverWait(items[i], timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "RichContent-inner")) + RichContent = items[i].find_element(By.CLASS_NAME, 'RichContent-inner') + except: + pass + content = RichContent.find_element(By.CLASS_NAME, 'RichText') + # time.sleep(sleeptime) + links_col = content.find_elements(By.TAG_NAME, 'a') + links = [] + for itext in links_col: + links.append(itext.get_attribute("href")) + text = content.text.strip() + if len(text)!=0: + with open(os.path.join(dirthink, clock+".txt"), 'w', encoding='utf-8') as obj: + obj.write(text.replace("
", '\n').replace("
", '\n')+"\n") + for itext in links: + obj.write(itext + "\n") + # all_think_detail.append([text]) + try: + items[i].find_elements(By.CLASS_NAME, 'Image-PreviewVague')[0].click() + except: + continue + cnt = 0 + while True: + WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "ImageGallery-Inner")) + img = driver.find_element(By.CLASS_NAME, 'ImageGallery-Inner').find_element(By.TAG_NAME, 'img') + imglink = img.get_attribute('data-original') + try: + response = requests.get(imglink, timeout=30) + except: + try: + response = requests.get(imglink, timeout=30) + except: + continue + if response.status_code==200: + with open(os.path.join(dirthink, clock + "_" + str(cnt) + '.jpg'), 'wb') as obj: + obj.write(response.content) + cnt += 1 + time.sleep(sleeptime) + try: + disable = driver.find_element(By.CLASS_NAME, 'ImageGallery-arrow-right') + if 'disabled' in disable.get_attribute('class'): + driver.find_element(By.CLASS_NAME, 'ImageGallery-close').click() + break + else: + disable.click() + except: + break + time.sleep(sleeptime) + dealthink(thinkdir) + +def crawl_article_detail(driver:webdriver): + website_col = {} + for i in os.listdir(articledir): + try: + kk = int(i) + shutil.rmtree(os.path.join(articledir, i)) + except: + pass + with open(os.path.join(articledir, 'article.txt'), 'r', encoding='utf-8') as obj: + for i in obj.readlines(): + i = i.strip() + ind = i.index(" ") + website = i[:ind] + title = i[ind+1:].replace(" ", "_").replace("\n", "") + website_col[website] = title + for website, title in website_col.items(): + nam = title.replace(":", "_").replace("?", ";"). \ + replace("/","_").replace("\\","_").replace("\"", "_").\ + replace("*","_").replace("|", "_").replace(" ", "_") + temp_name = str(np.random.randint(999999999)) + str(np.random.randint(999999999)) + # nam_pinyin = pinyin.get(nam, format='numerical') + # if '泰勒公式使用积分来推导' not in title: + # continue + dircrea = os.path.join(articledir, temp_name) + fileexit = os.path.exists(os.path.join(articledir, nam, nam + ".pdf")) + if fileexit: + filesize = os.path.getsize(os.path.join(articledir, nam, nam + ".pdf")) + direxit = os.path.exists(os.path.join(articledir, nam)) + + if direxit and not fileexit: + os.remove(os.path.join(articledir, nam)) + if direxit and fileexit and filesize > 0: + continue + if direxit and fileexit and filesize == 0: + os.remove(os.path.join(articledir, nam, nam + ".pdf")) + os.remove(os.path.join(articledir, nam)) + os.makedirs(dircrea, exist_ok = True) + original_window = driver.current_window_handle + + #get math formuladriver.find_element(By.ID, 'Popover15-toggle').click() + driverkk = edgeopen(driverpath, strategy=True) + pyautogui.press(["Enter", "Enter", "Enter"]) + driverkk.get(r"https://www.zhihu.com/signin") + try: + load_cookie(driverkk, cookie_path) + driverkk.get(website) + except: + driverkk = login(driverkk) + save_cookie(driverkk, cookie_path) + + #get math formula + # pyautogui.press(["Enter", "Enter", 'enter']) + # pyautogui.hotkey("ctrl", "l") + # pyautogui.hotkey("ctrl", "l") + # pyautogui.write(website, 0.01) + # pyautogui.hotkey("ctrl", "l") + # pyautogui.press(["enter", 'enter', 'enter']) + # time.sleep(formula_time) + try: + WebDriverWait(driverkk, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "ztext-math")) + except: + pass + driverkk.execute_script("window:stop();") + richtext = driverkk.find_element(By.CLASS_NAME, "Post-RichText") + # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"]) + # matherr = -1 + # try: + # richtext = driver.find_element(By.CLASS_NAME, "Post-RichText") + # MathJax_SVG = driver.find_elements(By.CLASS_NAME, "MathJax_SVG") + # if len(MathJax_SVG) > 0: + # matherr = 1 + # raise ValueError("") + # except: + # pyautogui.hotkey("ctrl", "l") + # pyautogui.hotkey("ctrl", "l") + # pyautogui.write(website, 0.01) + # pyautogui.press(["enter", 'enter', 'enter']) + # if matherr < 0: + # time.sleep(formula_time+0.2) + # else: + # time.sleep(max(formula_time - 0.1, 0.3)) + # matherr = -1 + # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"]) + # try: + # richtext = driver.find_element(By.CLASS_NAME, "Post-RichText") + # MathJax_SVG = driver.find_elements(By.CLASS_NAME, "MathJax_SVG") + # if len(MathJax_SVG) > 0: + # matherr = 1 + # raise ValueError("") + # except: + # pyautogui.hotkey("ctrl", "l") + # pyautogui.hotkey("ctrl", "l") + # pyautogui.write(website, 0.01) + # pyautogui.press(["enter", 'enter', 'enter']) + # if matherr < 0: + # time.sleep(formula_time+0.3) + # else: + # time.sleep(max(formula_time - 0.2, 0.3)) + # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"]) + # richtext = driver.find_element(By.CLASS_NAME, "Post-RichText") + + titletext = driverkk.find_element(By.CLASS_NAME, "Post-Title") + textlink = [] + tabletd = richtext.find_elements(By.TAG_NAME, "td") + pcontent = richtext.find_elements(By.TAG_NAME, "a") + time.sleep(sleeptime) + for i in range(len(tabletd)): + textlink.append(tabletd[i].text) + for i in range(len(pcontent)): + linksite = pcontent[i].get_attribute("href") + if linksite: + linksite = linksite.replace("//link.zhihu.com/?target=https%3A", "").replace("//link.zhihu.com/?target=http%3A", "") + textlink.append("["+pcontent[i].text+"]"+"("+linksite + ")\n") + + + + rt = richtext.text + h1 = richtext.find_elements(By.TAG_NAME, "h1") + for i in range(len(h1)): + rt = rt.replace(h1[i].text.strip()+"\n", "# " + h1[i].text.strip() + "\n") + h2 = richtext.find_elements(By.TAG_NAME, "h2") + for i in range(len(h2)): + rt = rt.replace(h2[i].text.strip()+"\n", "## " + h2[i].text.strip() + "\n") + pcontent = richtext.find_elements(By.TAG_NAME, "p") + imgcontent = richtext.find_elements(By.TAG_NAME, "figure") + pcontent[0].find_elements(By.CLASS_NAME) + pre = 0 + Pheight = [] + imgheight = [] + for i in range(len(pcontent)): + Pheight.append(pcontent[i].rect['y']) + for i in range(len(imgcontent)): + imgheight.append(imgcontent[i].rect['y']) + + imgpre = 0 + first = "" + tempstring = "" + tails = "" + number = 0 + temptxt = '' + for i in range(len(pcontent)): + imgtext = '' + temptxt += pcontent[i].text + while True: + if imgpre >= len(imgheight): + break + if i==0 and \ + imgheight[imgpre] < Pheight[i]: + imgtext += '''
\n'''%number
+ imgpre += 1
+ number += 1
+ elif i!=len(pcontent)-1 and \
+ imgheight[imgpre] > Pheight[i] and \
+ imgheight[imgpre] < Pheight[i+1]:
+ imgtext += '''\n'''%number
+ imgpre += 1
+ number += 1
+ elif i==len(pcontent)-1 and \
+ imgheight[imgpre] > Pheight[i]:
+ imgtext += '''\n'''%number
+ imgpre += 1
+ number += 1
+ else:
+ break
+ if len(imgtext) > 0:
+ ind = rt[pre:].find(temptxt)
+ first = rt[:ind + pre + len(temptxt)]
+ tails = rt[ind + pre + len(temptxt):]
+ rt = first + imgtext + tails
+ pre = len(first) + len(imgtext)
+ temptxt = ''
+ pre = 0
+ for i in range(len(pcontent)):
+ formula_span = pcontent[i].find_elements(By.CLASS_NAME, "math-holder")
+ for j in range(len(formula_span)):
+ ele = formula_span[j]
+ # kk = ele.get_attribute('class')
+ # if kk is not None and 'ztext-math' in kk:
+ formulatxt = ele.text
+ ind = rt[pre:].find(formulatxt)
+ first = rt[:pre]
+ tempstring = rt[pre:pre + ind + len(formulatxt)]
+ tails = rt[ind + pre + len(formulatxt):]
+ tempstring = tempstring.replace(ele.text, "$" + ele.text + "$")
+ rt = first + tempstring + tails
+ pre = len(first) + len(tempstring)
+
+
+
+ rt = rt.replace("修改\n", "").replace("开启赞赏\n", "开启赞赏, ").replace("添加评论\n", "").replace("分享\n", "").\
+ replace("收藏\n", "").replace("设置\n", "")
+ tle = titletext.text
+
+ if len(textlink) + len(rt) > 0:
+ with open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') as obj:
+ obj.write("# " + tle+"\n\n")
+ if len(rt) > 0:
+ obj.write(rt + "\n\n\n")
+
+ for i in range(len(textlink)):
+ obj.write(textlink[i] + "\n\n")
+ driverkk.quit()
+
+ #get article text
+ driver.switch_to.window(original_window)
+ driver.get(website)
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ # pyautogui.press("F5")
+ WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "Post-Topics"))
+
+ #https://stackoverflow.com/questions/61877719/how-to-get-current-scroll-height-in-selenium-python
+ scrollHeight = driver.execute_script('''return document.documentElement.scrollHeight''')
+ footer = driver.find_element(By.TAG_NAME, "html")
+ scroll_origin = ScrollOrigin.from_element(footer, 0, -60)
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, -100000).perform()
+ for i in range(18):
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, scrollHeight//18).perform()
+ time.sleep(0.8)
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ #remove noneed element
+ try:
+ driver.execute_script('''document.getElementsByClassName("Post-Sub")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("ColumnPageHeader-Wrapper")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("RichContent-actions")[0].remove();''')
+ except:
+ pass
+
+ richtext = driver.find_element(By.CLASS_NAME, "Post-RichText")
+ # textlink = []
+ # tabletd = richtext.find_elements(By.TAG_NAME, "td")
+ # pcontent = richtext.find_elements(By.TAG_NAME, "a")
+ time.sleep(sleeptime)
+ # for i in range(len(tabletd)):
+ # textlink.append(tabletd[i].text)
+ # for i in range(len(pcontent)):
+ # linksite = pcontent[i].get_attribute("href")
+ # if linksite:
+ # linksite = linksite.replace("//link.zhihu.com/?target=https%3A", "").replace("//link.zhihu.com/?target=http%3A", "")
+ # textlink.append(linksite + "\n" + pcontent[i].text)
+ rt = richtext.text
+
+ if len(textlink) + len(rt) > 0:
+ with open(os.path.join(dircrea, nam + ".txt"), 'w', encoding='utf-8') as obj:
+ if len(rt) > 0:
+ obj.write(rt + "\n\n\n")
+
+ for i in range(len(textlink)):
+ obj.write(textlink[i] + "\n\n")
+
+ # article image saving
+ imgchunk = richtext.find_elements(By.TAG_NAME, 'img')
+ cnt = 0
+ for i in range(len(imgchunk)):
+ imglink = imgchunk[i].get_attribute("data-original")
+ try:
+ response = requests.get(imglink, timeout=30)
+ except:
+ try:
+ response = requests.get(imglink, timeout=30)
+ except:
+ continue
+ if response.status_code==200:
+ with open(os.path.join(dircrea, str(cnt) + '.jpg'), 'wb') as obj:
+ obj.write(response.content)
+ cnt += 1
+ time.sleep(sleeptime)
+
+ # article to pdf
+ fileexit = os.path.exists(os.path.join(dircrea, temp_name + ".pdf"))
+ if fileexit:
+ os.remove(os.path.join(dircrea, temp_name + ".pdf"))
+
+ with pyautogui.hold("Ctrl"):
+ pyautogui.press("P")
+
+ time.sleep(2+addtime)
+ pyautogui.press("Enter")
+ time.sleep(2+addtime)
+ pyautogui.write(str(temp_name + ".pdf"), interval=0.01)
+ pyautogui.press(["Tab"]*6 + ["Enter"])
+ pyautogui.write(dircrea, interval=0.01)
+ pyautogui.press(["Enter"])
+ time.sleep(2+addtime)
+ pyautogui.press(["Enter", "Enter", "Enter", "Enter"])
+ time.sleep(6+addtime)
+ clocktxt = driver.find_element(By.CLASS_NAME, "Post-NormalMain").find_element(By.CLASS_NAME, "ContentItem-time")
+ time.sleep(1+addtime)
+ clock = clocktxt.text[3+1:].replace(" ", "_").replace(":", "_")
+ with open(os.path.join(dircrea, clock+".txt"), 'w', encoding='utf-8') as obj:
+ obj.write(clock)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(3+addtime)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(10+addtime)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(60+addtime)
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ os.rename(dircrea, os.path.join(articledir, nam))
+
+ time.sleep(sleeptime)
+
+ #https://stackoverflow.com/questions/23359083/how-to-convert-webpage-into-pdf-by-using-python
+ #https://github.com/JazzCore/python-pdfkit
+ # if article_to_jpg_pdf_markdown:
+ # config = pdfkit.configuration(wkhtmltopdf = wkhtmltopdf_path)
+ # pdfkit.from_url(website, os.path.join(dircrea, nam_pinyin+".pdf"), configuration = config)
+
+ # time.sleep(600)
+
+def crawl_answer_detail(driver:webdriver):
+ website_col = {}
+ for i in os.listdir(answerdir):
+ try:
+ kk = int(i)
+ shutil.rmtree(os.path.join(answerdir, i))
+ except:
+ pass
+ with open(os.path.join(answerdir, 'answers.txt'), 'r', encoding='utf-8') as obj:
+ for i in obj.readlines():
+ i = i.strip()
+ ind = i.index(" ")
+ website = i[:ind]
+ title = i[ind+1:].replace(" ", "_").replace("\n", "")
+ website_col[website] = title
+ for website, title in website_col.items():
+ nam = title.replace(":", "_").replace("?", ";"). \
+ replace("/","_").replace("\\","_").replace("\"", "_").\
+ replace("*","_").replace("|", "_").replace(" ", "_")
+ temp_name = str(np.random.randint(999999999)) + str(np.random.randint(999999999))
+ # nam_pinyin = pinyin.get(nam, format='numerical')
+ if '用矩阵的初等变化来' not in title:
+ continue
+ dircrea = os.path.join(answerdir, temp_name)
+ fileexit = os.path.exists(os.path.join(answerdir, nam, nam + ".pdf"))
+ if fileexit:
+ filesize = os.path.getsize(os.path.join(answerdir, nam, nam + ".pdf"))
+ direxit = os.path.exists(os.path.join(answerdir, nam))
+
+ if direxit and not fileexit:
+ os.remove(os.path.join(answerdir, nam))
+ if direxit and fileexit and filesize > 0:
+ continue
+ if direxit and fileexit and filesize == 0:
+ os.remove(os.path.join(answerdir, nam, nam + ".pdf"))
+
+ os.remove(os.path.join(answerdir, nam))
+ os.makedirs(dircrea, exist_ok = True)
+ original_window = driver.current_window_handle
+
+ #get math formula
+ driverkk = edgeopen(driverpath, strategy=True)
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ driverkk.get(r"https://www.zhihu.com/signin")
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ try:
+ load_cookie(driverkk, cookie_path)
+ driverkk.get(website)
+ except:
+ driverkk = login(driverkk)
+ save_cookie(driverkk, cookie_path)
+ # pyautogui.press(["Enter", "Enter", 'enter'])
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.write(website, 0.01)
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.press(["enter", 'enter', 'enter'])
+ # time.sleep(formula_time)
+ # driver.find_element(By.CLASS_NAME, "ztext-math")
+ try:
+ WebDriverWait(driverkk, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "ztext-math"))
+ except:
+ pass
+ driverkk.execute_script("window:stop();")
+ # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"])
+ Created = "not found"
+ Modified = "not found"
+ QuestionAnswer = driverkk.find_element(By.CLASS_NAME, "QuestionAnswer-content")
+ richtext = QuestionAnswer.find_element(By.CLASS_NAME, "RichContent--unescapable")
+ metatext = QuestionAnswer.find_elements(By.TAG_NAME, "meta")
+ for i in range(len(metatext)):
+ if metatext[i].get_attribute("itemprop")=="dateCreated":
+ Created = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ if metatext[i].get_attribute("itemprop")=="dateModified":
+ Modified = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # matherr = -1
+ # try:
+ # QuestionAnswer = driverkk.find_element(By.CLASS_NAME, "QuestionAnswer-content")
+ # richtext = QuestionAnswer.find_element(By.CLASS_NAME, "RichContent--unescapable")
+ # metatext = QuestionAnswer.find_elements(By.TAG_NAME, "meta")
+ # MathJax_SVG = QuestionAnswer.find_elements(By.CLASS_NAME, "MathJax_SVG")
+ # if len(MathJax_SVG) > 0:
+ # matherr = 1
+ # raise ValueError("")
+ # for i in range(len(metatext)):
+ # if metatext[i].get_attribute("itemprop")=="dateCreated":
+ # Created = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # if metatext[i].get_attribute("itemprop")=="dateModified":
+ # Modified = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # except:
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.write(website, 0.01)
+ # pyautogui.press(["enter", 'enter', 'enter'])
+ # if matherr < 0:
+ # time.sleep(formula_time+0.2)
+ # else:
+ # time.sleep(max(formula_time - 0.1, 0.3))
+ # matherr = -1
+ # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"])
+ # try:
+ # QuestionAnswer = driverkk.find_element(By.CLASS_NAME, "QuestionAnswer-content")
+ # richtext = QuestionAnswer.find_element(By.CLASS_NAME, "RichContent--unescapable")
+ # metatext = QuestionAnswer.find_elements(By.TAG_NAME, "meta")
+ # MathJax_SVG = QuestionAnswer.find_elements(By.CLASS_NAME, "MathJax_SVG")
+ # if len(MathJax_SVG) > 0:
+ # matherr = 1
+ # raise ValueError("")
+ # for i in range(len(metatext)):
+ # if metatext[i].get_attribute("itemprop")=="dateCreated":
+ # Created = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # if metatext[i].get_attribute("itemprop")=="dateModified":
+ # Modified = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # except:
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.hotkey("ctrl", "l")
+ # pyautogui.write(website, 0.01)
+ # pyautogui.press(["enter", 'enter', 'enter'])
+ # if matherr < 0:
+ # time.sleep(formula_time+0.3)
+ # else:
+ # time.sleep(max(formula_time - 0.2, 0.3))
+ # pyautogui.press(["esc", "esc", "esc", "esc", "esc", "esc"])
+ # QuestionAnswer = driverkk.find_element(By.CLASS_NAME, "QuestionAnswer-content")
+ # richtext = QuestionAnswer.find_element(By.CLASS_NAME, "RichContent--unescapable")
+ # metatext = QuestionAnswer.find_elements(By.TAG_NAME, "meta")
+ # for i in range(len(metatext)):
+ # if metatext[i].get_attribute("itemprop")=="dateCreated":
+ # Created = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+ # if metatext[i].get_attribute("itemprop")=="dateModified":
+ # Modified = metatext[i].get_attribute("content").replace(" ", "_").replace(":", "_").replace(".", "_")
+
+ textlink = []
+ tabletd = richtext.find_elements(By.TAG_NAME, "td")
+ pcontent = richtext.find_elements(By.TAG_NAME, "a")
+ time.sleep(sleeptime)
+ for i in range(len(tabletd)):
+ textlink.append(tabletd[i].text)
+ for i in range(len(pcontent)):
+ linksite = pcontent[i].get_attribute("href")
+ if linksite:
+ linksite = linksite.replace("//link.zhihu.com/?target=https%3A", "").replace("//link.zhihu.com/?target=http%3A", "")
+ textlink.append("["+pcontent[i].text+"]"+"("+linksite + ")\n")
+
+
+
+ rt = richtext.text
+ h1 = richtext.find_elements(By.TAG_NAME, "h1")
+ for i in range(len(h1)):
+ rt = rt.replace(h1[i].text.strip()+"\n", "# " + h1[i].text.strip() + "\n")
+ h2 = richtext.find_elements(By.TAG_NAME, "h2")
+ for i in range(len(h2)):
+ rt = rt.replace(h2[i].text.strip()+"\n", "## " + h2[i].text.strip() + "\n")
+ pcontent = richtext.find_elements(By.TAG_NAME, "p")
+ imgcontent = richtext.find_elements(By.TAG_NAME, "figure")
+ pcontent[0].find_elements(By.CLASS_NAME)
+ pre = 0
+ Pheight = []
+ imgheight = []
+ for i in range(len(pcontent)):
+ Pheight.append(pcontent[i].rect['y'])
+ for i in range(len(imgcontent)):
+ imgheight.append(imgcontent[i].rect['y'])
+
+ imgpre = 0
+ first = ""
+ tempstring = ""
+ tails = ""
+ number = 0
+ temptxt = ''
+ for i in range(len(pcontent)):
+ imgtext = ''
+ temptxt += pcontent[i].text
+ while True:
+ if imgpre >= len(imgheight):
+ break
+ if i==0 and \
+ imgheight[imgpre] < Pheight[i]:
+ imgtext += '''\n'''%number
+ imgpre += 1
+ number += 1
+ elif i!=len(pcontent)-1 and \
+ imgheight[imgpre] > Pheight[i] and \
+ imgheight[imgpre] < Pheight[i+1]:
+ imgtext += '''\n'''%number
+ imgpre += 1
+ number += 1
+ elif i==len(pcontent)-1 and \
+ imgheight[imgpre] > Pheight[i]:
+ imgtext += '''\n'''%number
+ imgpre += 1
+ number += 1
+ else:
+ break
+ if len(imgtext) > 0:
+ ind = rt[pre:].find(temptxt)
+ first = rt[:ind + pre + len(temptxt)]
+ tails = rt[ind + pre + len(temptxt):]
+ rt = first + imgtext + tails
+ pre = len(first) + len(imgtext)
+ temptxt = ''
+ pre = 0
+ for i in range(len(pcontent)):
+ formula_span = pcontent[i].find_elements(By.CLASS_NAME, "math-holder")
+ for j in range(len(formula_span)):
+ ele = formula_span[j]
+ # kk = ele.get_attribute('class')
+ # if kk is not None and 'ztext-math' in kk:
+ formulatxt = ele.text
+ ind = rt[pre:].find(formulatxt)
+ first = rt[:pre]
+ tempstring = rt[pre:pre + ind + len(formulatxt)]
+ tails = rt[ind + pre + len(formulatxt):]
+ tempstring = tempstring.replace(ele.text, "$" + ele.text + "$")
+ rt = first + tempstring + tails
+ pre = len(first) + len(tempstring)
+
+
+ rt = rt.replace("修改\n", "").replace("开启赞赏\n", "开启赞赏, ").replace("添加评论\n", "").replace("分享\n", "").\
+ replace("收藏\n", "").replace("设置\n", "")
+
+ if len(textlink) + len(rt) > 0:
+ with open(os.path.join(dircrea, nam + "_formula_" + ".md"), 'w', encoding='utf-8') as obj:
+ obj.write("# " + title+"\n\n")
+ obj.write("Created: " + Created + "\n")
+ obj.write("Modified: " + Modified + "\n\n")
+ if len(rt) > 0:
+ obj.write(rt + "\n\n\n")
+
+ for i in range(len(textlink)):
+ obj.write(textlink[i] + "\n\n")
+ driverkk.quit()
+
+ #get article text
+ driver.switch_to.window(original_window)
+ driver.get(website)
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ # pyautogui.press("F5")
+ WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.CLASS_NAME, "AnswerItem-editButtonText"))
+
+ #https://stackoverflow.com/questions/61877719/how-to-get-current-scroll-height-in-selenium-python
+ scrollHeight = driver.execute_script('''return document.getElementsByClassName("QuestionAnswer-content")[0].scrollHeight''')
+ footer = driver.find_element(By.TAG_NAME, "html")
+ scroll_origin = ScrollOrigin.from_element(footer, 0, 0)
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, -100000).perform()
+ for i in range(18):
+ ActionChains(driver).scroll_from_origin(scroll_origin, 0, scrollHeight//18).perform()
+ time.sleep(0.8)
+ pyautogui.press(["Enter", "Enter", "Enter"])
+ #remove noneed element
+
+ try:
+ driver.execute_script('''document.getElementsByClassName("MoreAnswers")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("ViewAll")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("ViewAll")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("AppHeader")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("Reward")[0].remove();''')
+ except:
+ pass
+ try:
+ driver.execute_script('''document.getElementsByClassName("Question-sideColumn")[0].remove();''')
+ except:
+ pass
+
+ QuestionAnswer = driver.find_element(By.CLASS_NAME, "QuestionAnswer-content")
+ richtext = QuestionAnswer.find_element(By.CLASS_NAME, "RichContent--unescapable")
+ # textlink = []
+ # tabletd = richtext.find_elements(By.TAG_NAME, "td")
+ # pcontent = richtext.find_elements(By.TAG_NAME, "a")
+ time.sleep(sleeptime)
+ # for i in range(len(tabletd)):
+ # textlink.append(tabletd[i].text)
+ # for i in range(len(pcontent)):
+ # linksite = pcontent[i].get_attribute("href")
+ # if linksite:
+ # linksite = linksite.replace("//link.zhihu.com/?target=https%3A", "").replace("//link.zhihu.com/?target=http%3A", "")
+ # textlink.append(linksite + "\n" + pcontent[i].text)
+ rt = richtext.text
+
+ if len(textlink) + len(rt) > 0:
+ with open(os.path.join(dircrea, nam + ".txt"), 'w', encoding='utf-8') as obj:
+ obj.write(title+"\n\n")
+ obj.write("Created: " + Created + "\n")
+ obj.write("Modified: " + Modified + "\n")
+ if len(rt) > 0:
+ obj.write(rt + "\n\n\n")
+
+ for i in range(len(textlink)):
+ obj.write(textlink[i] + "\n\n")
+
+ # article image saving
+ imgchunk = richtext.find_elements(By.TAG_NAME, 'img')
+ cnt = 0
+ for i in range(len(imgchunk)):
+ imglink = imgchunk[i].get_attribute("data-original")
+ try:
+ response = requests.get(imglink, timeout=30)
+ except:
+ try:
+ response = requests.get(imglink, timeout=30)
+ except:
+ continue
+ if response.status_code==200:
+ with open(os.path.join(dircrea, str(cnt) + '.jpg'), 'wb') as obj:
+ obj.write(response.content)
+ cnt += 1
+ time.sleep(sleeptime)
+
+ # article to pdf
+ fileexit = os.path.exists(os.path.join(dircrea, temp_name + ".pdf"))
+ if fileexit:
+ os.remove(os.path.join(dircrea, temp_name + ".pdf"))
+
+ with pyautogui.hold("Ctrl"):
+ pyautogui.press("P")
+
+ time.sleep(2+addtime)
+ pyautogui.press("Enter")
+ time.sleep(2+addtime)
+ pyautogui.write(str(temp_name + ".pdf"), interval=0.01)
+ pyautogui.press(["Tab"]*6 + ["Enter"])
+ pyautogui.write(dircrea, interval=0.01)
+ pyautogui.press(["Enter"])
+ time.sleep(2+addtime)
+ pyautogui.press(["Enter", "Enter", "Enter", "Enter"])
+ time.sleep(6+addtime)
+ # clocktxt = driver.find_element(By.CLASS_NAME, "Post-NormalMain").find_element(By.CLASS_NAME, "ContentItem-time")
+ time.sleep(1+addtime)
+ clock = Created #clocktxt.text[3+1:].replace(" ", "_").replace(":", "_")
+ with open(os.path.join(dircrea, clock+".txt"), 'w', encoding='utf-8') as obj:
+ obj.write(clock)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(3+addtime)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(10+addtime)
+ try:
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ except:
+ time.sleep(60+addtime)
+ os.rename(os.path.join(dircrea, temp_name + ".pdf"), os.path.join(dircrea, nam + ".pdf"))
+ os.rename(dircrea, os.path.join(answerdir, nam))
+ time.sleep(sleeptime)
+ #https://stackoverflow.com/questions/23359083/how-to-convert-webpage-into-pdf-by-using-python
+ #https://github.com/JazzCore/python-pdfkit
+ # if article_to_jpg_pdf_markdown:
+ # config = pdfkit.configuration(wkhtmltopdf = wkhtmltopdf_path)
+ # pdfkit.from_url(website, os.path.join(dircrea, nam_pinyin+".pdf"), configuration = config)
+
+ time.sleep(600)
+
+def zhihu(driverpath, cookie_path):
+ website = r"https://www.zhihu.com/signin"
+
+ #login and save cookies of zhihu
+ driver = edgeopen(driverpath)
+ driver.get(website)
+ try:
+ load_cookie(driver, cookie_path)
+ driver.get(website)
+ except:
+ driver = login(driver)
+ save_cookie(driver, cookie_path)
+ pyautogui.press(["Enter"]*10)
+ driver.find_element(By.ID, 'Popover15-toggle').click()
+ driver.find_element(By.CLASS_NAME, 'Menu-item').click()
+ url = driver.current_url
+ username = url.split("/")[-1]
+
+ # #crawl think links
+ if crawl_think:
+ crawl_think_links(driver, username)
+
+ # #crawl articles links
+ if crawl_article:
+ if not os.path.exists(os.path.join(articledir, 'article.txt')):
+ crawl_article_links(driver, username)
+ else:
+ if crawl_links_scratch:
+ nowtime = datetime.fromtimestamp(time.time()).isoformat().replace(":", "_")
+ os.rename(os.path.join(articledir, 'article.txt'), os.path.join(articledir, 'article_%s.txt'%nowtime))
+ crawl_article_links(driver, username)
+ else:
+ pass
+ crawl_article_detail(driver)
+
+ # #crawl answers links
+ if crawl_answer:
+ if not os.path.exists(os.path.join(answerdir, 'answers.txt')):
+ crawl_answers_links(driver, username)
+ else:
+ if crawl_links_scratch:
+ nowtime = datetime.fromtimestamp(time.time()).isoformat().replace(":", "_")
+ os.rename(os.path.join(answerdir, 'answers.txt'), os.path.join(answerdir, 'answers_%s.txt'%nowtime))
+ crawl_answers_links(driver, username)
+ else:
+ pass
+ crawl_answer_detail(driver)
+
+ driver.quit()
+
+if __name__ == "__main__":
+ #version four.one_zero.zero
+ driverpath = os.path.join(abspath, 'msedgedriver\msedgedriver.exe')
+ savepath = deepcopy(abspath)
+ cookiedir = os.path.join(savepath, 'cookie')
+ thinkdir = os.path.join(savepath, 'think')
+ answerdir = os.path.join(savepath, 'answer')
+ articledir = os.path.join(savepath, 'article')
+ os.makedirs(cookiedir, exist_ok=True)
+ os.makedirs(thinkdir, exist_ok=True)
+ os.makedirs(answerdir, exist_ok=True)
+ os.makedirs(articledir, exist_ok=True)
+ cookie_path =os.path.join(cookiedir, 'cookie_zhihu.pkl')
+
+ parser = argparse.ArgumentParser(description=r'crawler zhihu.com, 爬取知乎的想法, 回答, 文章, 包括数学公式')
+ parser.add_argument('--sleep_time', type=float, default = 2, \
+ help=r'crawler sleep time during crawling, 爬取时的睡眠时间, 避免给知乎服务器带来太大压力, \
+ 可以日间调试好，然后深夜运行爬取人少, 给其他小伙伴更好的用户体验, 避免知乎顺着网线过来找人, 默认: 2s')
+ # parser.add_argument('--formula_time', type=float, default=0.6-0.2, \
+ # help=r'crawler math formula sleep time,default:0.6-0.2, 爬取数学公式页面, 需要的sleep时间, 默认0.6-0.2')
+ parser.add_argument('--computer_time_sleep', type=float, default=0, \
+ help=r'computer running sleep time 默认:0, 电脑运行速度的sleep时间, 默认:0')
+ parser.add_argument('--think', action="store_true", help=r'crawl think, 是否爬取知乎的想法, 已经爬取过的想法不会重复爬取, 所以可以多次爬取断了也没关系')
+ parser.add_argument('--answer', action="store_true", help=r'crawl answer, 是否爬取知乎的回答, 保存到pdf、markdown以及相关图片等，已经爬取过的不会重复爬取，\
+ 断了再次爬取的话，可以配置到--links_scratch，事先保存好website')
+ parser.add_argument('--article', action="store_true", help=r'crawl article, 是否爬取知乎的文章, 保存到pdf、markdown以及相关图片等，已经爬取过的不会重复爬取，\
+ 断了再次爬取的话，可以配置到--links_scratch，事先保存好website')
+ # parser.add_argument('--article_to_jpg_pdf_markdown', action="store_true", help=r'save article to pdf, 文章保存到pdf和markdown以及图片等')
+ parser.add_argument('--links_scratch', action="store_true", \
+ help=r'crawl links scratch for answer or article, 是否使用已经保存好的website和title, 否则再次爬取website')
+ args = parser.parse_args()
+ sleeptime = args.sleep_time
+ crawl_think = args.think
+ crawl_answer = args.answer
+ crawl_article = args.article
+ crawl_links_scratch = args.links_scratch
+ # article_to_jpg_pdf_markdown = args.article_to_jpg_pdf_markdown
+ addtime = args.computer_time_sleep
+ # formula_time = args.formula_time
+
+
+ # crawl_think = True
+ # crawl_article = True
+ crawl_answer = True
+ # crawl_links_scratch = True
+ # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --sleep_time 1 --think --links_scratch
+ # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --sleep_time 1 --article --links_scratch
+ # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --sleep_time 1 --answer --links_scratch
+ # python.exe c:/Users/10696/Desktop/access/zhihu/crawler.py --sleep_time 1 --think --answer --article --links_scratch
+ zhihu(driverpath, cookie_path)
\ No newline at end of file
diff --git a/requirement.txt b/requirement.txt
new file mode 100644
index 0000000..09482f0
Binary files /dev/null and b/requirement.txt differ
diff --git a/showimg/add1.png b/showimg/add1.png
new file mode 100644
index 0000000..ec948e5
Binary files /dev/null and b/showimg/add1.png differ
diff --git a/showimg/answer1.png b/showimg/answer1.png
new file mode 100644
index 0000000..67fff4e
Binary files /dev/null and b/showimg/answer1.png differ
diff --git a/showimg/answer2.png b/showimg/answer2.png
new file mode 100644
index 0000000..73b2f02
Binary files /dev/null and b/showimg/answer2.png differ
diff --git a/showimg/answer3.png b/showimg/answer3.png
new file mode 100644
index 0000000..f1f6e8c
Binary files /dev/null and b/showimg/answer3.png differ
diff --git a/showimg/answer4.png b/showimg/answer4.png
new file mode 100644
index 0000000..fa57b38
Binary files /dev/null and b/showimg/answer4.png differ
diff --git a/showimg/article1.png b/showimg/article1.png
new file mode 100644
index 0000000..8a4b83e
Binary files /dev/null and b/showimg/article1.png differ
diff --git a/showimg/article2.png b/showimg/article2.png
new file mode 100644
index 0000000..c9b9e30
Binary files /dev/null and b/showimg/article2.png differ
diff --git a/showimg/article3.png b/showimg/article3.png
new file mode 100644
index 0000000..537c24f
Binary files /dev/null and b/showimg/article3.png differ
diff --git a/showimg/article4.png b/showimg/article4.png
new file mode 100644
index 0000000..7be04f9
Binary files /dev/null and b/showimg/article4.png differ
diff --git a/showimg/article5.png b/showimg/article5.png
new file mode 100644
index 0000000..aed3701
Binary files /dev/null and b/showimg/article5.png differ
diff --git a/showimg/condarc.png b/showimg/condarc.png
new file mode 100644
index 0000000..71df23c
Binary files /dev/null and b/showimg/condarc.png differ
diff --git a/showimg/login.png b/showimg/login.png
new file mode 100644
index 0000000..ed52949
Binary files /dev/null and b/showimg/login.png differ
diff --git a/showimg/miniconda.png b/showimg/miniconda.png
new file mode 100644
index 0000000..6efb5db
Binary files /dev/null and b/showimg/miniconda.png differ
diff --git a/showimg/think1.png b/showimg/think1.png
new file mode 100644
index 0000000..20eb179
Binary files /dev/null and b/showimg/think1.png differ
diff --git a/showimg/think2.png b/showimg/think2.png
new file mode 100644
index 0000000..99aced3
Binary files /dev/null and b/showimg/think2.png differ
diff --git a/showimg/think3.png b/showimg/think3.png
new file mode 100644
index 0000000..518b026
Binary files /dev/null and b/showimg/think3.png differ
diff --git a/think/2023-01-21_13_01/2023-01-21_13_01.txt b/think/2023-01-21_13_01/2023-01-21_13_01.txt
new file mode 100644
index 0000000..2b90853
--- /dev/null
+++ b/think/2023-01-21_13_01/2023-01-21_13_01.txt
@@ -0,0 +1 @@
+6---饱的
diff --git a/think/2023-01-21_13_01/2023-01-21_13_01_0.jpg b/think/2023-01-21_13_01/2023-01-21_13_01_0.jpg
new file mode 100644
index 0000000..5bf331a
Binary files /dev/null and b/think/2023-01-21_13_01/2023-01-21_13_01_0.jpg differ
diff --git a/think/2023-01-21_13_01/2023-01-21_13_01_1.jpg b/think/2023-01-21_13_01/2023-01-21_13_01_1.jpg
new file mode 100644
index 0000000..f5db474
Binary files /dev/null and b/think/2023-01-21_13_01/2023-01-21_13_01_1.jpg differ
diff --git a/thinkdeal.py b/thinkdeal.py
new file mode 100644
index 0000000..063cebd
--- /dev/null
+++ b/thinkdeal.py
@@ -0,0 +1,23 @@
+#################
+###ZouJiu-202306
+#################
+
+import os
+
+def dealthink(inpath):
+ dic = {}
+ for root, dirk, files in os.walk(inpath):
+ for i in files:
+ if '.txt' not in i:
+ continue
+ with open(os.path.join(root, i), 'r', encoding='utf-8') as obj:
+ kk = obj.read()
+ dic[i] = kk
+ dic = sorted(dic.items(), key= lambda k:k[0])
+ with open(os.path.join(inpath, 'all_txt.txt'), 'w', encoding='utf-8') as obj:
+ for key, value in dic:
+ obj.write(key+"\n"+value+"\n")
+
+if __name__=="__main__":
+ inpath = r'C:\Users\10696\Desktop\access\zhihu\think'
+ dealthink(inpath)
\ No newline at end of file