DOCSEARCHbyAlgolia
Submit Docs

QMSum: Query-based Multi-domain Meeting Summarization

repository·main·Indexed 41 days ago

https://github.com/yale-lily/qmsum

A benchmark dataset and research repository for query-based multi-domain meeting summarization. It provides human-annotated query-summary pairs across Academic, Product, and Committee domains, including meeting transcripts and relevant text spans. The repository includes data processing tools for seq2seq models like BART and PGNet, as well as pre-computed extracted spans and model outputs for evaluation.

Tokens
2.7K
Snippets
6
Records
10
Agent score
50%

What's inside QMSum

  1. Understand the QMSum JSON data format

    main

    The QMSum data is structured as a JSON object containing topics, queries, and meeting transcripts.

    Key Fields:

    • topic_list: A list of topics. Each topic includes the topic string and relevant_text_span (a list of [start, end] string indices).
    • general_query_list: Queries that ask for a summary of the entire meeting. These do not have relevant_text_span because they cover the whole transcript.
    • specific_query_list: Queries targeting specific parts of the meeting. Each includes the query, the answer, and the relevant_text_span indices.
    • meeting_transcripts: A list of objects containing the speaker name and the content of their speech.

    Note: A topic or specific query may have multiple relevant text spans.

    {
    "topic_list": [
        {
            "topic": "Introduction of petitions and prioritization of governmental matters",
            "relevant_text_span": [["0","19"]]
        }
    ],
    "general_query_list": [
        {
            "query": "Summarize the whole meeting.",
            "answer": "The meeting of the standing committee took place to discuss matters pertinent to the Coronavirus pandemic..."
        }
    ],
    "specific_query_list": [
        {
            "query": "Summarize the discussion about introduction of petitions and prioritization of government matters.",
            "answer": "The Chair brought the meeting to order...",
            "relevant_text_span": [["0","19"]]
        }
    ],
    "meeting_transcripts": [
        {
            "speaker": "The Chair",
            "content": "I call the meeting to order..."
        }
    ]
}

  2. Convert data for seq2seq models

    main

    To use QMSum with sequence-to-sequence models like BART or PGNet, use the provided Jupyter notebook data_process.ipynb to convert the raw data into the required format.

    Important Constraint: During training for these models, the maximum source length is set to 2048.

  3. Implement supported models

    main

    The repository references several models used in the original paper. To implement them, refer to the following external resources:

  4. Access the QMSum dataset

    main

    The QMSum dataset is organized into a main directory containing all data and three domain-specific directories.

    Directory Structure:

    • data/ALL: Contains the full train/valid/test sets.
    • data/Academic: Data specific to the Academic domain.
    • data/Product: Data specific to the Product domain.
    • data/Committee: Data specific to the Committee domain.

    File Formats and Splits: Within each folder, you will find:

    • .jsonl: Data in JSON Lines format.
    • all: All data in a single .json file.
    • train: Training data split.
    • val: Validation data split.
    • test: Test data split.

  5. Process data for BART using gold span content

    main

    For training models to focus on specific relevant segments, use the relevant_text_span provided in the specific_query_list. Instead of the entire meeting, the source (src) is constructed only from the turns within the indices specified by the gold span. The input format remains <s> query </s> gold_span_content </s>.

    # For specific queries, extract only the relevant turns
for j in range(len(data[i]['specific_query_list'])):
    cur = {}
    query = tokenize(data[i]['specific_query_list'][j]['query'])
    src = []
    for span in data[i]['specific_query_list'][j]['relevant_text_span']:
        st, ed = int(span[0]), int(span[1])
        for k in range(st, ed + 1):
            cur_turn = data[i]['meeting_transcripts'][k]['speaker'].lower() + ': '
            cur_turn = cur_turn + tokenize(data[i]['meeting_transcripts'][k]['content'])
            src.append(cur_turn)
    src_str = ' '.join(src)
    cur['src'] = clean_data('<s> ' + query + ' </s> ' + src_str + ' </s>')
    target = tokenize(data[i]['specific_query_list'][j]['answer'])
    cur['tgt'] = target
    bart_data_gold.append(cur)

  6. Process data for BART using entire meeting content

    main

    To prepare data for BART where the model sees the entire meeting transcript, iterate through the dataset and construct a source string (src) by concatenating all speaker turns. For each query in general_query_list and specific_query_list, format the input as: <s> query </s> meeting_content </s>. The target (tgt) is the tokenized answer.

    # Example of the resulting structure for BART
# src: <s> query </s> speaker1: content speaker2: content </s>
# tgt: tokenized answer

bart_data = []
for i in range(len(data)):
    src = []
    for k in range(len(data[i]['meeting_transcripts'])):
        cur_turn = data[i]['meeting_transcripts'][k]['speaker'].lower() + ': '
        cur_turn = cur_turn + tokenize(data[i]['meeting_transcripts'][k]['content'])
        src.append(cur_turn)
    src = ' '.join(src)
    
    # Process both general and specific queries
    for query_list in [data[i]['general_query_list'], data[i]['specific_query_list']]:
        for j in range(len(query_list)):
            cur = {}
            query = tokenize(query_list[j]['query'])
            cur['src'] = clean_data('<s> ' + query + ' </s> ' + src + ' </s>')
            target = tokenize(query_list[j]['answer'])
            cur['tgt'] = target
            bart_data.append(cur)

  7. Read the QMSum dataset from JSONL

    main

    To load the QMSum dataset, read the .jsonl files for a specific split (e.g., 'train'). Each line in the file represents a single meeting containing transcripts and query lists.

    import json

split = 'train'
data_path = '/path/to/data/ALL/jsonl/' + split + '.jsonl'
data = []
with open(data_path) as f:
    for line in f:
        data.append(json.loads(line))

  8. Access extracted spans and model outputs

    main

    The repository provides pre-computed outputs for evaluation and research:

    • Extracted Spans: Spans extracted by the Locator (used as input for the Summarizer) are located in the /extracted_span directory.
    • Model Outputs: Summaries generated by HMNet (using golden input) are located in the /model_output directory. The ROUGE scores for these outputs are R-1: 36.51, R-2: 11.41, and R-L: 31.60.

  9. Tokenize text using NLTK

    main

    The tokenize function converts a string to lowercase and joins NLTK word tokens with spaces to ensure consistent formatting for model training.

    from nltk import word_tokenize

def tokenize(sent):
    tokens = ' '.join(word_tokenize(sent.lower()))
    return tokens

  10. Clean speech recognition noise from text

    main

    Use the clean_data function to remove speech recognition artifacts and non-verbal markers from the transcripts. This includes removing tags like { vocalsound }, { pause }, { nonvocalsound }, and { gap }, and normalizing specific speaker/sound tokens.

    def clean_data(text):
    text = text.replace('{ vocalsound } ', '')
    text = text.replace('{ disfmarker } ', '')
    text = text.replace('a_m_i_', 'ami')
    text = text.replace('l_c_d_', 'lcd')
    text = text.replace('p_m_s', 'pms')
    text = text.replace('t_v_', 'tv')
    text = text.replace('{ pause } ', '')
    text = text.replace('{ nonvocalsound } ', '')
    text = text.replace('{ gap } ', '')
    return text