2025-06-28 17:16:27 -04:00
2025-06-28 17:16:27 -04:00
2025-06-28 17:16:27 -04:00
2023-07-25 01:57:00 -04:00
2023-07-25 01:57:00 -04:00
2025-06-28 17:16:27 -04:00
2025-06-28 17:16:27 -04:00
2025-06-28 17:16:27 -04:00

This program opens a range of datastores between two specified dates and extracts all the data
for a set of user-defined publishers.

Add the gitea repo (this has to be done just once):
dotnet nuget add source --name Gitea https://www.gitea.econlabs.org/api/packages/markusmobiuspublic/nuget/index.json

Clone repo:
git clone https://www.gitea.econlabs.org/markusmobiuspublic/RetroIndex

The navigate to RetroIndex folder:
cd RetroSampleApp
dotnet publish
cd bin\Release\net9.0\publish
dotnet RetroSampleApp.dll


Expected Output
-------------

7 stores are available between 6/15/2025 12:00:00 AM and 6/22/2025 12:00:00 AM:
first available time stub: 2025_06_16_00_00_00
 last available time stub: 2025_06_22_00_00_00
Opening datastore 2025_06_16_00_00_00 from disk ...
Opening datastore 2025_06_17_00_00_00 from disk ...
Opening datastore 2025_06_18_00_00_00 from disk ...
Opening datastore 2025_06_19_00_00_00 from disk ...
Opening datastore 2025_06_20_00_00_00 from disk ...
Opening datastore 2025_06_21_00_00_00 from disk ...
Opening datastore 2025_06_22_00_00_00 from disk ...
publishers: 8594
nytimes.com: exists in data
washingtonpost.com: exists in data
Reading filtered publishers ..
-----------------------
Publisher: nytimes.com
7/7 days: ["2025_06_16_00_00_00","2025_06_17_00_00_00","2025_06_18_00_00_00","2025_06_19_00_00_00","2025_06_20_00_00_00","2025_06_21_00_00_00","2025_06_22_00_00_00"]
total pages: 36046
pages with publish data between 6/15/2025 12:00:00 AM and 6/22/2025 12:00:00 AM: 604
-----------------------
Publisher: washingtonpost.com
7/7 days: ["2025_06_16_00_00_00","2025_06_17_00_00_00","2025_06_18_00_00_00","2025_06_19_00_00_00","2025_06_20_00_00_00","2025_06_21_00_00_00","2025_06_22_00_00_00"]
total pages: 19560
pages with publish data between 6/15/2025 12:00:00 AM and 6/22/2025 12:00:00 AM: 828
-----------------------


In order to incorporate in your own project make sure you reference the BingScrapeReadLib (2.0.0+) nuget package:
https://www.gitea.econlabs.org/markusmobiuspublic/-/packages/nuget/bingscrapereadlib/2.0.0

The RetroSampleApp demonstrates how.
Description
This library opens a range of datastores between two specified dates and extracts all the data for a set of user-defined publishers.
Readme 88 KiB
Languages
C# 100%