CSV file conversion to binary format for faster data loading

Blaa blaa.

Tests

These time measurement tests are done using Pythons time.perf_counter function and on a plugged-in laptop with Intel Core Ultra 5 125H processor.

Loading a 522 MB csv file takes aproximately 4,20 seconds. Loading that same data in binary format takes 0,12 seconds. On a 2 GB file loading time goes from 18 seconds to 1,5 seconds.

Code

# convert the columns to 16-bit integers
signal_1 = np.asarray(df["adc1"], dtype=np.int16)
signal_2 = np.asarray(df["adc2"], dtype=np.int16)
# interleave the two signals column-wise -> adc1, adc2 per sample
interleaved = np.column_stack((signal_1, signal_2))

# if no save path is given, place the .bin next to the CSV
if save_path is None:
    save_path = os.path.splitext(file_path)[0] + ".bin"

# write the data to disk and return save path
interleaved.tofile(save_path)
return save_path

Conclusion

Blaa blaa.