Quickstart
[1]:
from data_gatherer.data_gatherer import DataGatherer
Instantiate DataGatherer and select which LLM to use.
To provide the api key you can either add it in an .env file in the working directory or set an environment variable for that.
[2]:
data_gatherer = DataGatherer(llm_name="gemini-2.5-flash")
Extracting Dataset Citations from Papers
Given one or more scientific publications, you can provide the url to their full-text - or path to the PDF file - as input for the DataGatherer.
[3]:
input_articles = ["https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6141466"]
Then, with the process_articles method, you can extract information from the full-text of the scientific publication and have a summary of the Datasets used in the study.
You can use the section_filter parameter to specify which section you want to extract data from (supplementary_material or data_availability_statement).
[4]:
article_available_data = data_gatherer.process_articles(input_articles, section_filter='data_availability_statement', semantic_retrieval=True)
article_available_data["https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6141466"]
Token indices sequence length is longer than the specified maximum sequence length for this model (533 > 512). Running this sequence through the model will result in indexing errors
[4]:
| dataset_identifier | data_repository | dataset_webpage | source_section | retrieval_pattern | access_mode | pub_title | source_url | raw_data_format | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | PHS001049 | dbgap | https://www.ncbi.nlm.nih.gov/projects/gap/cgi-... | data_availability | data availability | Application to access | Recurrent WNT pathway alterations are frequent... | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | XML |
| 1 | PRJNA306801 | sra | https://www.ncbi.nlm.nih.gov/bioproject/?term=... | data_availability | data availability | Complex download | Recurrent WNT pathway alterations are frequent... | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | XML |
[5]:
article_supplementary_material = data_gatherer.process_articles(input_articles, section_filter='supplementary_material')
article_supplementary_material["https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6141466"]
[5]:
| link | source_url | download_link | title | content_type | id | caption | description | source_section | retrieval_pattern | context_description | file_extension | pub_title | raw_data_format | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 41467_2018_6162_MOESM1_ESM.pdf | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM1 | Supplementary Information | Supplementary Information | supplementary material | .//supplementary-material | Representative slides from these samples were ... | Recurrent WNT pathway alterations are frequent... | XML | |
| 1 | 41467_2018_6162_MOESM2_ESM.pdf | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM2 | Peer Review File | Peer Review File | supplementary material | .//supplementary-material | Recurrent WNT pathway alterations are frequent... | XML | ||
| 2 | 41467_2018_6162_MOESM3_ESM.pdf | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM3 | Description of Additional Supplementary Files | Description of Additional Supplementary Files | supplementary material | .//supplementary-material | Recurrent WNT pathway alterations are frequent... | XML | ||
| 3 | 41467_2018_6162_MOESM4_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM4 | Supplementary Data 1 | Supplementary Data 1 | supplementary material | .//supplementary-material | Representative slides from these samples were ... | xlsx | Recurrent WNT pathway alterations are frequent... | XML |
| 4 | 41467_2018_6162_MOESM5_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM5 | Supplementary Data 2 | Supplementary Data 2 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 5 | 41467_2018_6162_MOESM6_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM6 | Supplementary Data 3 | Supplementary Data 3 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 6 | 41467_2018_6162_MOESM7_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM7 | Supplementary Data 4 | Supplementary Data 4 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 7 | 41467_2018_6162_MOESM8_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM8 | Supplementary Data 5 | Supplementary Data 5 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 8 | 41467_2018_6162_MOESM9_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM9 | Supplementary Data 6 | Supplementary Data 6 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 9 | 41467_2018_6162_MOESM10_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM10 | Supplementary Data 7 | Supplementary Data 7 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 10 | 41467_2018_6162_MOESM11_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM11 | Supplementary Data 8 | Supplementary Data 8 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 11 | 41467_2018_6162_MOESM12_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM12 | Supplementary Data 9 | Supplementary Data 9 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 12 | 41467_2018_6162_MOESM13_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM13 | Supplementary Data 10 | Supplementary Data 10 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 13 | 41467_2018_6162_MOESM14_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM14 | Supplementary Data 11 | Supplementary Data 11 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML | |
| 14 | 41467_2018_6162_MOESM15_ESM.xlsx | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6... | https://pmc.ncbi.nlm.nih.gov/articles/instance... | No Title | local-data | MOESM15 | Supplementary Data 12 | Supplementary Data 12 | supplementary material | .//supplementary-material | xlsx | Recurrent WNT pathway alterations are frequent... | XML |
[ ]: