Compile frequency of keywords on all articles of certain websites

We are looking for a web scraping script that searches through all articles on a number of websites for certain keywords and outputs the article’s entire contents, the frequency of the keywords, and other metadata. We are also looking for a script that compiles the 50 most frequent words in those articles by month. More specific details are provided below. Output • CSV with one row for each article and columns for the following features (see “article summary” tab in attachment for template): o Date o Website o Article title o URL o Location of website headquarters o Article contents o Frequency of keyword 1 in article body o Presence of keyword 1 in article title (true/false or 1/0) o Repeat frequency and presence measures for other keywords • CSV with one row for each of the top 50 most frequent words and columns for the following features (see “top 50 monthly” tab in attachment for template): o Date (month-year) o Keyword o Frequency • Web scraping script(s) in Python or R that produced the output Scope • Articles containing at least one of the keywords below • Articles from 2014 to 2015 inclusive • For the “top 50” exercise, please exclude all common words, as listed on the website [login to view URL] Keywords o Ad/advertising/advertiser o Ad tech/advertising tech/ad technology/advertising technology o Ad exchange o Ad network o Web publisher/online publisher/digital publisher o Real time bidding/real-time bidding/RTB/real time auction/real-time auction o Demand side platform/demand-side platform/DSP o Supply side platform/supply-side platform/SSP o Data management platform/data-management platform/DMP o Programmatic o Programmatic advertising o Programmatic direct o Programmatic guaranteed o Programmatic reserved o Programmatic premium o Programmatic non-reserved o Programmatic buying o Programmatic selling o Programmatic real-time bidding/programmatic real time bidding/programmatic RTB o Preferred deal o Targeting o Geotargeting o Behavioral targeting o Retargeting/re-targeting o Cross-device tracking/cross device tracking/cross-device targeting/cross device targeting o Revenue optimization o Monetize/monetizing/monetization o Native advertising/native ad/native advertisement o Mobile advertising/mobile ad/mobile advertisement o Sponsored content/sponsored post o Branded content o Content recommendation o Advertorial o Viewability o In-feed/in feed o In-stream/in stream o Direct sale o Banner ad/banner advertising/banner advertisement o Display ad/display advertising/display advertisement o Malvertising/malicious advertising/malicious ad/malware o Ad fraud/advertising fraud/impression fraud/click fraud o Ad-blocking/ad blocking/ad-blocker/ad blocker o Adblock o Bot traffic/non-human traffic/non human traffic o Ad stacking o Whitelist/white list/whitelisting o Blacklist/black list/blacklisting o Rich media o Waterfall/waterfalling o Tag management o In-app advertising/in app advertising o Audiences o Measurement o Digital advertising/digital ad/digital advertisement o Big data o Third-party data/third party data o Click-thru rate/click thru rate/click-through rate/click through rate/click rate/CTR o Cost-per-action/cost per action/CPA o Cost-per-click/cost per click/CPC o Cost-per-install/cost per install/CPI o Cost-per-thousand/cost per thousand/cost-per-mille/cost per mille/CPM o Effective cost-per-thousand/effective cost per thousand/effective cost-per-mille/effective cost per mille/eCPM o Demand fill o Private marketplace/private exchange o Privacy Websites • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL] • [login to view URL]