Efficient String Concatenation In Python 3

I am writing a Python 3 code where the task is to open about 550 files in a directory, read their contents and append it to a string variable 'all_text' which will be say around mi

Solution 1:

join() has a definition str.join(iterable) where iterable is a generator or a list or a set and so on. So it is helpful if you already have a list of strings read from the files and you are concatenating them using join. For example

numList = ['1', '2', '3', '4']
seperator = ', 'print(seperator.join(numList))

numTuple = ('1', '2', '3', '4')
print(seperator.join(numTuple))

s1 = 'abc'
s2 = '123'""" Each character of s2 is concatenated to the front of s1"""print('s1.join(s2):', s1.join(s2))

""" Each character of s1 is concatenated to the front of s2"""print('s2.join(s1):', s2.join(s1))

You can get all lines in a file using join like ''.join(readlines(f))

Now you can accomplish your task using join as follows using fileinput module

import fileinput
files= ['package-lock.json', 'sqldump.sql', 'felony.json', 'maindata.csv']
allfiles = fileinput.input(files)
all_text = ''.join(allfiles)

Refer to this answer to know the most efficient way to concat files into a string.

Suggestion: As you mentioned there would be millions of lines, did you consider the memory it is going to consume to store it in a variable? So it is better you do what you are planning to do on the fly while reading the lines instead of storing it in a variable.