Issues

• k

  EleutherAI/dps#82 · Filter by author NguyenNhoTrung opened on Jun 18, 2024

• dedup_job java.lang.UnsatisfiedLinkError

  EleutherAI/dps#81 · Filter by author syedhasnainrazashah opened on Apr 1, 2024
  3

• [ja] .filter is used instead of .map for non-filter methods

  EleutherAI/dps#74 · Filter by author mrorii opened on Jul 14, 2023
  1

• Chiese dedup memory error

  EleutherAI/dps#65 · Filter by author hyeinhyun opened on Apr 29, 2023
  1

• [ja] refactor MinHashLSH-based near deduplication method

  EleutherAI/dps#62 · Filter by author fujiki-1emon opened on Apr 19, 2023
  

• Refactor RDD process to Dataframe process

  enhancement
  EleutherAI/dps#57 · Filter by author Taekyoon opened on Apr 3, 2023

• Need to add ignore null or empty text during korean text process

  bug
  EleutherAI/dps#56 · Filter by author Taekyoon opened on Apr 1, 2023

• [ja] spam word filter

  EleutherAI/dps#51 · Filter by author fujiki-1emon opened on Mar 13, 2023

• [ja] replace Japanese PII

  EleutherAI/dps#49 · Filter by author fujiki-1emon opened on Mar 13, 2023

• Add scripts to run hadoop cluster

  enhancement
  EleutherAI/dps#20 · Filter by author Taekyoon opened on May 24, 2022
  

• Add build news paper dataset as long text data form

  add job
  EleutherAI/dps#9 · Filter by author Taekyoon opened on Apr 24, 2022