Docs Data Gatherer

class data_gatherer.data_gatherer.DataGatherer(llm_name='gpt-4o-mini', process_entire_document=False, log_level=40, clear_previous_logs=True, log_file_override=None, load_from_cache=False, save_to_cache=False, driver_path=None, save_dynamic_prompts=False, write_htmls_xmls=False, article_file_dir='tmp/raw_files/', download_data_for_description_generation=False, data_resource_preview=False, download_previewed_data_resources=False, embeds_cache_read=False, embeds_cache_write=False, data_repos_config='open_bio_data_repos.json', grobid_for_pdf=False, raw_data_df_parquet_filepath=None)

This class orchestrates the data gathering process by coordinating the publication fetcher, the publication parser, and the metadata processing in a single workflow.

Parameters:

llm_name – Name of the LLM model to use for parsing and classification.
process_entire_document – Flag to indicate if the model processes the entire document.
log_level – Logging level for the logger (‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’).
clear_previous_logs – Flag to clear previous logs before setting up logging.
log_file_override – Optional log file path to override the default logging configuration.
load_from_cache – Flag to indicate if results should be loaded from cache.
save_to_cache – Flag to indicate if results should be saved to cache.
driver_path – Path to the WebDriver executable for the data fetcher (if applicable).
save_dynamic_prompts – Flag to indicate if dynamically generated prompts should be saved.
write_htmls_xmls – Flag to indicate if raw HTML/XML files should be saved.
article_file_dir – Directory to save the raw HTML/XML/PDF files.
download_data_for_description_generation – Flag to indicate if data should be downloaded for description generation.
data_resource_preview – Flag to indicate if a preview of data resources should be generated.
embeds_cache_read – Flag to indicate if embeddings cache should be read.
embeds_cache_write – Flag to indicate if embeddings cache should be written.
download_previewed_data_resources – Flag to indicate if previewed data resources should be downloaded.

create_publication_corpus(metadata, sitemap: str)

Create a corpus of publications from the dataset/study page and sitemap.

Parameters:

metadata – Extracted metadata for the dataset.
sitemap – Sitemap of the dataset/study website containing internal links.

Returns:

A corpus of publications related to the dataset.

display_metadata(metadata, display_type='console', interactive=True)

Display extracted metadata as a clean table in both Jupyter and terminal environments.

Parameters:

metadata – Dictionary containing metadata to display.
display_type – Type of display for the preview. Options are ‘console’ or ‘ipynb’.
interactive – If True, allows user interaction for displaying data previews.

fetch_data(urls, search_method='url_list', driver_path=None, browser=None, headless=True, HTML_fallback=False, local_fetch_file=None, write_htmls_xmls=False, article_file_dir='scripts/tmp/raw_files/', write_df_to_path=False, sects_required=1, HTML_fallback_priority_list=['HTTPGetRequest', 'Selenium'])

Fetches data from the given URL using the configured data fetcher (WebScraper or EntrezFetcher).

Parameters:

urls – The list of URLs to fetch data from.
search_method – Optional method to override the default search method. Supported values are ‘url_list’, ‘cloudscraper’, ‘google_scholar’.
driver_path – Path to your local WebDriver executable (if applicable). When set to None, Webdriver manager will be used.
browser – Browser to use for scraping (if applicable). Supported values are ‘Firefox’, ‘Chrome’.
headless – Whether to run the browser in headless mode (if applicable).
HTML_fallback – Flag to indicate if HTML fallback should be used when fetching data. This will override any other fetching resource (i.e. API).
local_fetch_file – Optional file containing data to be used in the fetching process. Supported format is ‘parquet’ file.
write_htmls_xmls – Flag to indicate if raw HTML/XML files should be saved. Overwrites the default setting.
article_file_dir – Directory to save the raw HTML/XML/PDF files. Overwrites the default setting.
write_df_to_path – Optional path to save the fetched data as a DataFrame in Parquet format.
sects_required – List or Int indicating required sections to consider the fetched HTML data complete.

Returns:

Dictionary with URLs as keys and raw data as values.

fetch_publication_and_extract_dataset_references(url, full_document_read=False, semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, save_staging_table=False, article_file_dir='tmp/raw_files/', use_portkey=True, driver_path=None, browser='Firefox', headless=True, HTML_fallback=False, grobid_for_pdf=False, write_htmls_xmls=False)

Simplified interface for process_url, where you do not have to select prompt and response format and all the parameters are set automatically based on the LLM being used.

Parameters:

url – URL/PMCID of the publication to process
full_document_read – Flag to indicate if the model processes the entire document.
semantic_retrieval – Flag to indicate if semantic retrieval should be used.
top_k – Number of top documents to retrieve for semantic retrieval.
embeddings_retriever_model – Model to use for embeddings retrieval.
section_filter – Optional filter to apply to the sections (supplementary_material’, ‘data_availability_statement’).
save_staging_table – Flag to save the staging table.
article_file_dir – Directory to save the raw HTML/XML/PDF files.
use_portkey – Flag to use Portkey for Gemini LLM.
driver_path – Path to your local WebDriver executable (if applicable). When set to None, Webdriver manager will be used.
browser – Browser to use for scraping (if applicable). Supported values are ‘Firefox’, ‘Chrome’.
headless – Whether to run the browser in headless mode (if applicable).
HTML_fallback – Flag to indicate if HTML fallback should be used.
grobid_for_pdf – Flag to indicate if GROBID should be used for PDF processing.
write_htmls_xmls – Flag to indicate if raw HTML/XML files should be saved.

Returns:

DataFrame of classified data links.

get_params() → Dict[str, Any]

Get all public attributes and their values from the DataGatherer instance. Uses introspection to dynamically retrieve all non-private attributes.

Returns:: Dictionary of all public attributes and their values.

normalize_fulltext_input(fulltext): Normalize the fulltext input to ensure it’s a string.

parse_data(raw_data, raw_data_format='XML', full_document_read=False, prompt_name='GPT_FewShot', response_format={'name': 'GPT_responses_schema', 'schema': {'additionalProperties': False, 'properties': {'datasets': {'items': {'additionalProperties': False, 'properties': {'data_repository': {'description': 'A valid URI or string referring to the repository.', 'maxLength': 128, 'type': 'string'}, 'dataset_identifier': {'description': 'A unique identifier or accession code for the dataset.', 'maxLength': 64, 'type': 'string'}, 'dataset_webpage': {'description': "If present the URL of the dataset page, otherwise return 'n/a'.", 'maxLength': 128, 'type': 'string'}}, 'required': ['dataset_identifier', 'data_repository', 'dataset_webpage'], 'type': 'object'}, 'minItems': 1, 'type': 'array'}}, 'required': ['datasets'], 'type': 'object'}, 'type': 'json_schema'}, semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, publisher=None, current_url_address=None, parsed_data_dir='tmp/parsed_articles/', use_portkey=True, grobid_for_pdf=False, dedup=True, brute_force_RegEx_ID_ptrs=False)

Parses the raw data fetched from the source using the appropriate parser.

Parameters:

raw_data – The raw data to parse, typically string formatted as HTML or XML content, if local PDF it will accept the file path.
raw_data_format – The format of the raw data (e.g., ‘HTML’, ‘XML’).
full_document_read – Flag to indicate if the model processes the entire document.
prompt_name – Name of the prompt to use for LLM parsing.
response_format – The response schema to use as target schema for the output sequence.
semantic_retrieval – Flag to indicate if semantic retrieval should be used.
top_k – Number of top relevant sections to retrieve if semantic retrieval is enabled.
embeddings_retriever_model – The name of the embeddings model (from sentence-transformers: https://huggingface.co/sentence-transformers) to use for semantic retrieval.
section_filter – Optional filter to apply to the sections (supplementary_material’, ‘data_availability_statement’).
publisher – The publisher domain or identifier for the data source.
current_url_address – The URL of the current data source being processed.
parsed_data_dir – Directory to save the parsed HTML/XML/PDF files.
use_portkey – Flag to use Portkey for supported API providers (e.g., Gemini).
grobid_for_pdf – Flag to indicate if Grobid should be used for PDF parsing. Read more on GROBID PDF Parser here: https://grobid.readthedocs.io/en/latest/

Returns:

Parsed data as a DataFrame or dictionary, depending on the parser used.

process_articles(url_list, log_modulo=10, full_document_read=False, response_format={'name': 'GPT_responses_schema', 'schema': {'additionalProperties': False, 'properties': {'datasets': {'items': {'additionalProperties': False, 'properties': {'data_repository': {'description': 'A valid URI or string referring to the repository.', 'maxLength': 128, 'type': 'string'}, 'dataset_identifier': {'description': 'A unique identifier or accession code for the dataset.', 'maxLength': 64, 'type': 'string'}, 'dataset_webpage': {'description': "If present the URL of the dataset page, otherwise return 'n/a'.", 'maxLength': 128, 'type': 'string'}}, 'required': ['dataset_identifier', 'data_repository', 'dataset_webpage'], 'type': 'object'}, 'minItems': 1, 'type': 'array'}}, 'required': ['datasets'], 'type': 'object'}, 'type': 'json_schema'}, prompt_name='GPT_FewShot', semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, save_staging_table=False, write_htmls_xmls=False, article_file_dir='tmp/raw_files/', driver_path=None, browser='Firefox', headless=True, use_portkey=True, grobid_for_pdf=False, brute_force_RegEx_ID_ptrs=False, return_df_joint=False)

Processes a list of article URLs and returns parsed data.

Parameters:

url_list – List of URLs/PMCIDs to process.
log_modulo – Frequency of logging progress (useful when url_list is long).
full_document_read – Flag to indicate if the model processes the entire document.
response_format – The response schema to use for parsing the data.
prompt_name – Name of the prompt to use for LLM parsing.
semantic_retrieval – Flag to indicate if semantic retrieval should be used.
top_k – Number of top documents to retrieve for semantic retrieval.
embeddings_retriever_model – Model to use for embeddings retrieval.
section_filter – Optional filter to apply to the sections (supplementary_material’, ‘data_availability_statement’).
save_staging_table – Flag to save the staging table.
article_file_dir – Directory to save the raw HTML/XML/PDF files.
driver_path – Path to your local WebDriver executable (if applicable). When set to None, Webdriver manager will be used.
browser – Browser to use for scraping (if applicable). Supported values are ‘Firefox’, ‘Chrome’.
headless – Whether to run the browser in headless mode (if applicable).
use_portkey – Flag to use Portkey for Gemini LLM.
grobid_for_pdf – Flag to indicate if GROBID should be used for PDF processing.
brute_force_RegEx_ID_ptrs – Flag to indicate if brute force RegEx ID pointers should be used.
return_df_joint – If True, returns a joint DataFrame of all parsed data instead of a dictionary of DataFrames per URL.

Returns:

Dictionary with URLs as keys and DataFrames of classified data as values.

process_metadata(combined_df, pass_cols_to_prompt=[], force_js_load=False, display_type='console', interactive=True, return_metadata=False, write_raw_metadata=False, article_file_dir='tmp/raw_files/', use_portkey=True, prompt_name='gpt_metadata_extract', response_format={'json_schema': {'name': 'Dataset_metadata_response', 'schema': {'properties': {'dataset_description': {'description': 'Short summary of the dataset contents, plus - if mentioned - the use in the research publication of interes.', 'type': 'string'}, 'dataset_identifier': {'description': 'A unique identifier for the dataset.', 'type': 'string'}, 'download_type': {'description': 'Type of download (e.g., HTTP, FTP, API, ...).', 'type': 'string'}, 'file_format': {'description': 'Format of the file (e.g., CSV, FASTQ).', 'type': 'string'}, 'file_license': {'description': 'License under which the file is distributed.', 'type': 'string'}, 'file_name': {'description': 'Filename or archive name.', 'type': 'string'}, 'file_size': {'description': 'Cumulative file size or range.', 'type': 'string'}, 'file_type': {'description': 'Type or category of the file.', 'type': 'string'}, 'file_url': {'description': 'Direct link to the file.', 'type': 'string'}, 'number_of_files': {'description': 'Total number of files.', 'type': 'string'}, 'request_access_form_links': {'description': 'Links to forms or pages where access requests can be made.', 'items': {'description': 'Links to forms or pages where access requests can be made.', 'format': 'uri', 'type': 'string'}, 'type': 'array'}, 'request_access_needed': {'description': '[Yes or No] Whether access to the file requires a request.', 'type': 'string'}, 'sample_size': {'description': 'How many samples are recorded in the dataset.', 'type': 'string'}}, 'required': ['dataset_description'], 'type': 'object'}}, 'type': 'json_schema'}, timeout=1, profile_dir=None, browser='Firefox', add_sitemap_to_prompt=False, redirect_url=None, from_metadata_to_publication_corpus=False)

This method iterates through the combined_df DataFrame, checks for dataset webpages or download links,

Parameters:

combined_df – DataFrame containing the data to preview. It should contain columns like ‘dataset_webpage’, ‘download_link’, etc.
force_js_load – If True, forces JavaScript loading for all dataset webpages.
pass_cols_to_prompt – A list of column names in combined_df to pass to the prompt for metadata extraction.
display_type – Type of display for the preview. Options are ‘console’, ‘ipynb’.
interactive – If True, allows user interaction for displaying data previews.
return_metadata – If True, returns a list of metadata dictionaries instead of just displaying them.
write_raw_metadata – If True, saves raw metadata to the specified directory.
article_file_dir – Directory to save raw HTML/XML files if write_raw_metadata is True.
use_portkey – If True, uses Portkey for Gemini LLM.
prompt_name – Name of the prompt to use for LLM parsing.
response_format – The response schema to use for parsing the metadata.
timeout – Timeout for requests to fetch dataset webpages.
profile_dir – Path to a persistent Firefox profile directory. If set, the browser session (cookies, auth tokens) is saved to disk so that login only needs to happen once. On subsequent runs with the same profile_dir, headless mode is maintained automatically once the session is still valid.
add_sitemap_to_prompt – If True, adds a sitemap of the dataset webpage to the prompt for navigation.

Returns:

If return_metadata is True, returns a list of metadata dictionaries. Otherwise, displays the data preview.

process_url(url, full_document_read=False, prompt_name='GPT_FewShot', response_format={'name': 'GPT_responses_schema', 'schema': {'additionalProperties': False, 'properties': {'datasets': {'items': {'additionalProperties': False, 'properties': {'data_repository': {'description': 'A valid URI or string referring to the repository.', 'maxLength': 128, 'type': 'string'}, 'dataset_identifier': {'description': 'A unique identifier or accession code for the dataset.', 'maxLength': 64, 'type': 'string'}, 'dataset_webpage': {'description': "If present the URL of the dataset page, otherwise return 'n/a'.", 'maxLength': 128, 'type': 'string'}}, 'required': ['dataset_identifier', 'data_repository', 'dataset_webpage'], 'type': 'object'}, 'minItems': 1, 'type': 'array'}}, 'required': ['datasets'], 'type': 'object'}, 'type': 'json_schema'}, semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, save_staging_table=False, article_file_dir='tmp/raw_files/', use_portkey=True, driver_path=None, browser='Firefox', headless=True, HTML_fallback=False, grobid_for_pdf=False, write_htmls_xmls=False, dedup=True, brute_force_RegEx_ID_ptrs=False, sects_required=5)

Orchestrates the process for a single given source URL (publication).

Fetches raw data using the data fetcher (WebScraper or EntrezFetcher).
Parses the raw data using the parser (LLMParser).
Scrapes additional Metadata from dataset pages (optional).

Parameters:

url – The URL to process.
full_document_read – Flag to indicate if the model processes the entire document.
prompt_name – Name of the prompt to use for LLM parsing.
response_format – The response schema to use for parsing the data.
semantic_retrieval – Flag to indicate if semantic retrieval should be used.
top_k – The number of top results to return for semantic retrieval (embeddings similarity).
embeddings_retriever_model – The name of the embeddings model (from sentence-transformers: https://huggingface.co/sentence-transformers) to use for semantic retrieval.
section_filter – Optional filter to apply to the sections (supplementary_material’, ‘data_availability_statement’).
save_staging_table – Flag to save the staging table.
article_file_dir – Directory to save the raw HTML/XML/PDF files.
use_portkey – Flag to use Portkey for supported APIs (e.g., Gemini).
driver_path – Path to your local WebDriver executable (if applicable). When set to None, Webdriver manager will be used.
browser – Browser to use for scraping (if applicable). Supported values are ‘Firefox’, ‘Chrome’.
headless – Whether to run the browser in headless mode (if applicable).
HTML_fallback – Flag to indicate if HTML fallback should be used when fetching data. This will override any other fetching resource (i.e. API).
grobid_for_pdf – Flag to indicate if GROBID should be used for PDF processing.
write_htmls_xmls – Flag to indicate if raw HTML/XML files should be saved. Overwrites the default setting.
sects_required – List or Int indicating required sections to consider the fetched HTML data complete.

Returns:

DataFrame of classified links or None if an error occurs.

retrieve_dataset_context(full_paper, dataset_ID_ptrs=None, dataset_info=None, force_include_DAS=False): Retrieve context for datasets using the parser’s retrieval method. Use-case: AutoDDG

run(input_file='scripts/exp_input/test_input.txt', full_output_file='scripts/output/result.csv', semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, prompt_name='GPT_FewShot', response_format={'name': 'GPT_responses_schema', 'schema': {'additionalProperties': False, 'properties': {'datasets': {'items': {'additionalProperties': False, 'properties': {'data_repository': {'description': 'A valid URI or string referring to the repository.', 'maxLength': 128, 'type': 'string'}, 'dataset_identifier': {'description': 'A unique identifier or accession code for the dataset.', 'maxLength': 64, 'type': 'string'}, 'dataset_webpage': {'description': "If present the URL of the dataset page, otherwise return 'n/a'.", 'maxLength': 128, 'type': 'string'}}, 'required': ['dataset_identifier', 'data_repository', 'dataset_webpage'], 'type': 'object'}, 'minItems': 1, 'type': 'array'}}, 'required': ['datasets'], 'type': 'object'}, 'type': 'json_schema'})

This method orchestrates the entire data gathering process by performing the following steps:

Setup data fetcher (web scraper or API client)
Load URLs from input_file
Process each URL and return results as a dictionary like source_url: DataFrame_of_data_links
Write results to output file specified in configuration file

Parameters:

input_file – Path to the input file containing URLs or PMCIDs to process.
full_output_file – Path to the output file where results will be saved.
semantic_retrieval – Flag to indicate if semantic retrieval should be used.
top_k – Number of top documents to retrieve for semantic retrieval.
embeddings_retriever_model – Model to use for embeddings retrieval.
section_filter – Optional filter to apply to the sections (supplementary_material’, ‘data_availability_statement’).
prompt_name – Name of the prompt to use for LLM parsing.
response_format – Format of the response to return.

Returns:

Combined DataFrame of all processed data links.

run_integrated_batch_processing(url_list, batch_file_path, output_file_path=None, api_provider='openai', prompt_name='GPT_FewShot', response_format=None, temperature=0.0, semantic_retrieval=False, top_k=5, embeddings_retriever_model=None, section_filter=None, submit_immediately=True, wait_for_completion=False, poll_interval=60, batch_description=None, grobid_for_pdf=False, use_portkey=True, dedup=True, brute_force_RegEx_ID_ptrs=False, write_htmls_xmls=False, article_file_dir='scripts/tmp/raw_files/', url2id_mapping=None, local_fetch_file=None)

Complete integrated batch processing using LLMClient batch functionality.

This method leverages the new LLMClient batch processing capabilities for improved performance and proper separation of concerns.

Parameters:

url_list – List of URLs/PMCIDs to process
batch_file_path – Path for the batch JSONL file
output_file_path – Path for the results file (auto-generated if None)
api_provider – ‘openai’ or ‘portkey’
prompt_name – Name of the prompt template
response_format – Response schema
temperature – Model temperature
semantic_retrieval – Enable semantic retrieval
top_k – Number of top results to retrieve
embeddings_retriever_model – Model for embeddings retrieval
section_filter – Section filter
submit_immediately – Whether to submit the batch job immediately
wait_for_completion – Whether to wait for batch completion
poll_interval – Seconds between status checks
batch_description – Optional description for the batch job
grobid_for_pdf – Whether to use GROBID for PDF processing
use_portkey – Whether to use Portkey for Gemini LLM
dedup – Whether to deduplicate rule-based retrieved elements
brute_force_RegEx_ID_ptrs – Whether to include snippets with ID patterns using brute-force regex
write_htmls_xmls – Whether to write HTML and XML files during processing
article_file_dir – Directory to save raw HTML/XML/PDF files
url2id_mapping – Optional mapping from URL to custom ID
local_fetch_file – Optional local file for fetching data

Returns:

Dictionary with batch information and results