51 lines
2.1 KiB
Plaintext
51 lines
2.1 KiB
Plaintext
This program opens a range of datastores between two specified dates and extracts all the data
|
|
for a set of user-defined publishers.
|
|
|
|
Add the gitea repo (this has to be done just once):
|
|
dotnet nuget add source --name Gitea https://www.gitea.econlabs.org/api/packages/markusmobiuspublic/nuget/index.json
|
|
|
|
Clone repo:
|
|
git clone https://www.gitea.econlabs.org/markusmobiuspublic/RetroIndex
|
|
|
|
The navigate to RetroIndex folder:
|
|
cd RetroSampleApp
|
|
dotnet publish
|
|
cd bin\Debug\net7.0\publish
|
|
dotnet RetroSampleApp.dll
|
|
|
|
|
|
Expected Output
|
|
-------------
|
|
|
|
7 stores are available between 2/15/2023 12:00:00 AM and 2/21/2023 12:00:00 AM:
|
|
first available time stub: 2023_02_15_00_00_00
|
|
last available time stub: 2023_02_21_00_00_00
|
|
Opening datastore 2023_02_15_00_00_00 from disk ...
|
|
Opening datastore 2023_02_16_00_00_00 from disk ...
|
|
Opening datastore 2023_02_17_00_00_00 from disk ...
|
|
Opening datastore 2023_02_18_00_00_00 from disk ...
|
|
Opening datastore 2023_02_19_00_00_00 from disk ...
|
|
Opening datastore 2023_02_20_00_00_00 from disk ...
|
|
Opening datastore 2023_02_21_00_00_00 from disk ...
|
|
publishers: 9020
|
|
nytimes.com: exists in data
|
|
washingtonpost.com: exists in data
|
|
Reading filtered publishers ..
|
|
-----------------------
|
|
Publisher: nytimes.com
|
|
7/7 days: ["2023_02_15_00_00_00","2023_02_16_00_00_00","2023_02_17_00_00_00","2023_02_18_00_00_00","2023_02_19_00_00_00","2023_02_20_00_00_00","2023_02_21_00_00_00"]
|
|
total pages: 105223
|
|
pages with publish data between 2/15/2023 12:00:00 AM and 2/21/2023 12:00:00 AM: 17030
|
|
-----------------------
|
|
Publisher: washingtonpost.com
|
|
7/7 days: ["2023_02_15_00_00_00","2023_02_16_00_00_00","2023_02_17_00_00_00","2023_02_18_00_00_00","2023_02_19_00_00_00","2023_02_20_00_00_00","2023_02_21_00_00_00"]
|
|
total pages: 28779
|
|
pages with publish data between 2/15/2023 12:00:00 AM and 2/21/2023 12:00:00 AM: 4143
|
|
-----------------------
|
|
|
|
|
|
In order to incorporate in your own project make sure you reference the BingScrapeReadLib (1.5.0+) nuget package:
|
|
https://www.gitea.econlabs.org/markusmobiuspublic/-/packages/nuget/bingscrapereadlib/1.5.0
|
|
|
|
The RetroSampleApp demonstrates how.
|