Skip to content
BranchesTags

Repository files navigation

TDM Studio Scripts

Author: Laura Vodden, Digital Media Research Centre, Queensland University of Technology


About this repository

This repository contains Python scripts for processing analysing datasets in the TDM Studio cleanroom environment.

Main functionality includes:

  • Processing XML data files by extracting XML data and saving to a CSV, to be read with e.g. Pandas.
  • Running several basic, preliminary analyses on the processed dataset.

How to use this repository

Follow the instructions carefully and in order!

Use this repository to guide you through from dataset creation to analysing the processed data.

Official TDM Studio documentation: https://proquest.libguides.com/tdmstudio




1. Creating a dataset in the TDM Studio Workbench

a. Log in to the workbench and click '+ Create New Dataset'

You may have up to 10 datasets visible in the Workbench at a time; if there is no room, kindly ask members of the team if any can be deleted. Deleting datasets from the Workbench will NOT delete them from your instance.

b. Follow the Search Tips to construct your query and filter your dataset.

You may save up to 2 million documents per dataset.

c) Create your dataset by clicking the 'Save Dataset' button.

You will be prompted to give your dataset a name and a description

Click 'Create Dataset'

You will be directed back to the Workbench, and your dataset should appear with STATUS as 'In-Progress'.




2. Enter the workspace

When you enter the workspace, you will see a file structure similar to the below:

🗂️SageMaker/
├── 🗂️ data/
├── 🗂️ Getting Started/
├── 🗂️ Getting Started R/
├── 🗂️ ...

The dataset you created is located in SageMaker/data by default.

! IMPORTANT: Do not try to open the files in these folders as the environment will crash. Raw data need to be accessed and proccessed via the terminal or a Jupyter Notebook.

The environment also contains some helpful tutorial-style (introductory) notebooks from TDM in Getting Started and Getting Started R, to guide you on the system usage.


3. Importing files to the Workspace

There is a multi-step process for uploading any files, including code, to the TDM Studio Workspace. The following instructions relate to importing this repository, but can be generalised to any file upload.

  1. Download this repository as a .zip file to your local machine.

  2. Upload to the TDM Studio workspace:

    1. Click My Files on the top navigation bar
    2. In the pop-up window, navigate to Temporary files
    3. Click Upload File(s) and upload the .zip file to the Temporary files folder from your local machine.
    4. Confirm that your file is uploaded to Temporary files, and close the pop-up window.

  3. In the Jupyter Notebook environment, ensure that you are in the top-level directory. It should look similar to the following:

    🗂️/
├── 🗂️ data/
├── 🗂️ Getting Started/
├── 🗂️ Getting Started R/
├── 🗂️ ...
├── 🗂️ ...

  4. In the top right of the Jupyter file view, click upload.

    1. Navigate to My Files -> Temporary Files and select your file.
    2. Your file will appear at the top of the list of files and directories; click Upload.
    3. Your file will upload, and you will see it appear in the directory.
    🗂️/
├── 🗂️ data/
├── 🗂️ Getting Started/
├── 🗂️ Getting Started R/
├── ⭐ TDM_Studio_Scripts.zip

The structure of this TDM_Studio_Scripts repository is as follows:

├── ️ 🗂️ TDM_Studio_Scripts
    ├── 🗂️ config
        ├── 🛠️ config.yml
    ├── 🗂️ input_files
    ├── 🗂️ output_files
    ├── 🗂️ TDM_pipeline
        ├── 🐍 __init__.py
        ├── 🐍 config.py
        ├── 🐍 pipeline.py
    ├── 🐍 process_dataset.py
    ├── 📖 README.md
    ├── 🐍 run_topic_model.py
    ├── 🐍 run_volume_analysis.py

Processing your datasets

The datasets created in the Workbench are provided as .XML files. In order to work with them, they need to be processed.

process_dataset.py extracts data from the XML tags and outputs a processed .CSV file to output_files











c. Setting up your config file

config/config.yml contains options to configure your project and search parameters. Below are the parameters and a description of each one.

# project params
project: 'setuptest'              # REQUIRED string e.g. 'ausvotes2025'
collection_id: None                # OPTIONAL string options: 'XXXX-XX-XX-XX-XX' | None

# query params
platform: 'facebook'               # REQUIRED string options: 'facebook' | 'instagram'
search_type: 'string'              # REQUIRED string options: 'prodlist' | 'string'
page_or_group: 'page'              # REQUIRED string options: 'page' | 'group'
search_string: 'brisbane auto'     # OPTIONAL string | None
since_date: '2024-12-31 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date: '2025-01-31 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date_comment_max_hrs: '48'  # max hours to capture for comments under post

d. Building your project structure

In the terminal, run the following command: python set_up_project.py --project This will use the information from your config to build the specific directories required for your project.

If you are working with more than one platform within the same project, you will need to update the config platform parameter and run this again. This will not do anything to your existing files.




2. Gathering data

Facebook

Workflow 1: Gather posts and comments from pages or groups using a search string and date parameters.

  1. Fill config.yml with your chosen parameters
    • search_type should be set to 'string'
    • search_string should contain your search string (see advanced string search rules for search syntax and accepted query formatting).
    • since_date and until_date should contain your search start and end dates, respectively.
  2. Run python run_gather.py --pages --gather --fromstring or python run_gather.py --groups --gather --fromstring

Example config.yml

# project params
project: 'ausvotes2025'              # REQUIRED string e.g. 'ausvotes2025'
collection_id: None                # OPTIONAL string options: 'XXXX-XX-XX-XX-XX' | None

# query params
platform: 'facebook'               # REQUIRED string options: 'facebook' | 'instagram'
search_type: 'string'              # REQUIRED string options: 'prodlist' | 'string'
page_or_group: 'page'              # REQUIRED string options: 'page' | 'group'
search_string: 'brisbane auto'     # OPTIONAL string | None
since_date: '2025-09-19 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date: '2025-12-03 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date_comment_max_hrs: '48'  # max hours to capture for comments under post



Workflow 2: Gather posts and comments from pages or groups using Producer Lists.

This uses the asynchronous method. Instructions for Instagram are the same as for Facebook.

  1. In your project/FACEBOOK/producers directory, create a text file called producer_ids_list.txt.
  2. Fill the template with your producer id values.
  3. Fill config.yml with your chosen parameters.
  4. Convert your producer_list_ids to page_ids: python run_gather.py --pages --gather --fromlist or python run_gather.py --groups --gather --fromlist.
    1. This will save your producer metadata to FACEBOOK/producers/producers.csv.
    2. While this runs, it will also get the page data and save them to FACEBOOK/pages/page_metrics.csv or FACEBOOK/groups/group_metrics.csv.
  5. You can now run an estimate of the posts by these pages or groups by running python run_gather.py --posts --estimate --fromlist in the terminal.
    1. This will print the total posts to the screen for your pages or groups that are available to collect during your specified timeframe.
  6. Gather posts if you are happy with this total: python run_gather.py --posts --gather --fromlist.
  7. Estimate number of comments for these posts: python run_gather.py --comments --estimate --fromlist
  8. Gather posts if you are happy with this total: python run_gather.py --comments --gather --fromlist

Note that comment gathering is likely to be costly - total budget per user is 500,000 comments per weekly rolling window. If you have a team, you can take advantage of your team's query budget, since this tool allows multiple users to gather data simultaneously. See '3. Post and comment batching' for an overview of how this works.


Example config.yml

# project params
project: 'ausvotes2025'              # REQUIRED string e.g. 'ausvotes2025'
collection_id: None                # OPTIONAL string options: 'XXXX-XX-XX-XX-XX' | None

# query params
platform: 'facebook'               # REQUIRED string options: 'facebook' | 'instagram'
search_type: 'prodlist'              # REQUIRED string options: 'prodlist' | 'string'
page_or_group: 'page'              # REQUIRED string options: 'page' | 'group'
search_string: ''     # OPTIONAL string | None
since_date: '2024-12-31 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date: '2025-05-03 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date_comment_max_hrs: '48'  # max hours to capture for comments under post



Workflow 3: PASSING PAGE, GROUP, POST AND COMMENT IDS TO THE QUERY AS STRING


Example config.yml [not yet accurate]

# project params
project: 'ausvotes2025'              # REQUIRED string e.g. 'ausvotes2025'
collection_id: None                # OPTIONAL string options: 'XXXX-XX-XX-XX-XX' | None

# query params
platform: 'facebook'               # REQUIRED string options: 'facebook' | 'instagram'
search_type: 'prodlist'              # REQUIRED string options: 'prodlist' | 'string'
page_or_group: 'page'              # REQUIRED string options: 'page' | 'group'
search_string: '['123654765868', '34635647585]'     # OPTIONAL string | None
since_date: '2024-12-31 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date: '2025-05-03 14:00:00'  # OPTIONAL utc date options: 'yyyy-mm-dd hh:mm:ss' | None
until_date_comment_max_hrs: '48'  # max hours to capture for comments under post



3. Outputs

All raw and processed data are stored in the relevant directories within your project, e.g. Facebook posts are stored in /project/FACEBOOK/posts


4. Post and comment batching

How it works:




5. Helpful shortcuts

Checking your budget

python run_gather.py --budget







TODO

  1. reduce redundancy in file paths
  2. add producer_ids_list.txt as empty file on setup
  3. allow user to specify batch size for estimate (comments) in cmd line - default is 250, cannot exceed this. if higher value entered, default to 250.
  4. test comment gathering from post list with additional query
  5. allow user to pass a list of ids to page, group, post or comment endpoint
    • allow post gathering from a list of page or group ids (as string in config)
    • allow comment gathering from a list of post ids (as string in config)

About

A selection of scripts that can be pasted or upload into TDM Studio to support data processing and analysis workflows.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

5 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages