phc/conferences-llm-crawler
6
2
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

chore: updated readme, minor changes to the code

Browse Source
This commit is contained in:
aziis98
2024-02-01 15:22:03 +01:00
parent 2a29c9e09f
commit bb6af6cf11
2 changed files with 56 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+45 -1
View File
@@ -37,4 +37,48 @@ The output is a list of json objects, one per line. To display the results with
```bash
$ jq -s '.' conferences.json
```
```
## Main idea / Explainaition
We need to parse strings like the following
```html
<p><a href="http://www.crm.sns.it/event/507/" target="_blank" rel="noreferrer
noopener">Statistical and Computational Aspects of Dynamics<br></a>Organized by Buddhima
Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi &#8211; SNS),
Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Centro De
Giorgi &#8211; SNS, Pisa. December 13 &#8211; 16, 2022.</p>
<p><a href="https://events.dm.unipi.it/event/126/" target="_blank" rel="noreferrer
noopener">Weekend di lavoro su Calcolo delle Variazioni<br></a>Organized by Giuseppe
Buttazzo, Maria Stella Gelli, and Aldo Pratelli. Grand Hotel Tettuccio Montecatini.
November 25 &#8211; 27, 2022.</p>
<p><a href="https://events.dm.unipi.it/event/109/" target="_blank" rel="noreferrer
noopener">Incontri di geometria algebrica ed aritmetica Milano – Pisa<br></a>Department
of Mathematics, Pisa. November 16 &#8211; 17, 2022.</p>
```
We compose the following conversation with an LLM, things between `{{ ... }}` are templates that will be replaced before starting generating.
```
INPUT:
<p><a href="http://www.crm.sns.it/event/507/" target="_blank" rel="noreferrer noopener">Statistical and Computational Aspects of Dynamics<br></a>Organized by Buddhima Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi &#8211; SNS), Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Centro De Giorgi &#8211; SNS, Pisa. December 13 &#8211; 16, 2022.</p>
OUTPUT:
{
"title": "Statistical and Computational Aspects of Dynamics",
"url": "http://www.crm.sns.it/event/507/",
"description": "Organized by Buddhima Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi – SNS), Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Location: Centro De Giorgi - SNS, Pisa.",
"startDate": "2022-12-13",
"endDate": "2022-12-16"
}
INPUT:
{{ conference_html }}
OUTPUT:
```
And the LLM will complete this conversation with a json representation of the `{{ conference_html }}`. The first example is needed to show the model how to convert information from the input html data.
main.py
+11 -3
View File
@@ -5,13 +5,21 @@ from bs4 import BeautifulSoup
import requests
import json
OUTPUT_FILE = "conferences.json"
HTML_EXAMPLE = r"""<p><a href="http://www.crm.sns.it/event/507/" target="_blank" rel="noreferrer noopener">Statistical and Computational Aspects of Dynamics<br></a>Organized by Buddhima Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi &#8211; SNS), Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Centro De Giorgi &#8211; SNS, Pisa. December 13 &#8211; 16, 2022.</p>"""
OUTPUT_EXAMPLE = json.dumps(
{ "title": "Statistical and Computational Aspects of Dynamics", "url": "http://www.crm.sns.it/event/507/", "description": "Organized by Buddhima Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi – SNS), Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Location: Centro De Giorgi - SNS, Pisa.", "startDate": "2022-12-13", "endDate": "2022-12-16" }
)
OUTPUT_EXAMPLE = json.dumps({
"title": "Statistical and Computational Aspects of Dynamics",
"url": "http://www.crm.sns.it/event/507/",
"description": "Organized by Buddhima Kasun Fernando Akurugodage (Centro di ricerca matematica Ennio De Giorgi – SNS), Paolo Giulietti, and Tanja Isabelle Schindler (Universität Wien, Austria). Location: Centro De Giorgi - SNS, Pisa.",
"startDate": "2022-12-13",
"endDate": "2022-12-16"
})
def translate_to_json(conference_html: str) -> str:
llm_answer = llm.create_chat_completion(