Round 2 / go-graphsplit

Go-Graphsplit divides large datasets into smaller pieces that fit the sector sizes of 32GiB or 64GiB, optimizing storage deals with miners. This process allows for storing substantial amounts of data, potentially reaching hundreds of TiB in a month. Traditional methods require retrieving and decompressing entire tarballs to access a single file, causing inefficiencies. Graphsplit addresses this by cutting datasets into smaller graphs using the IPLD protocol and Unixfs format. This approach maintains the original file system structure as much as possible, enabling efficient retrieval of individual files using payload CIDs through the Lotus client.Graphsplit generates a manifest.csv file that records the mapping between graph slice names, payload CIDs, piece CIDs, and the internal file structure. This ensures transparency and ease of access to stored data. In the past serveral months, Go-Graphsplit enhances the Filecoin Network's operational efficiency by streamlining both storage and retrieval processes. It allows for effective management of large datasets, which is crucial for the scalability of the Filecoin ecosystem. By simplifying data access and improving user experience, Go-Graphsplit supports broader adoption of decentralized storage solutions.