Python:how To Split File Into Chunks By The Occurrence Of The Header Word

Have a file: NAME,ANDREW,AGE 20, BD 1979 NAT ENGLISH OCC LONDON INC 200$ NAME,SVEN,AGE 20, BD 1979 NAT SWEDISH OCC FALUN INC 100$ NAME,HANS,AGE 30, BD 1988 NAT GERMAN OCC BERLIN NO

Solution 1:

I think this would be an elegant solution:

token = 'foo'
chunks = []
current_chunk = []

for line in open('sample_csv.csv'):
   if line.startswith(token) and current_chunk: 
      # if line starts with token and the current chunk is not empty
      chunks.append(current_chunk[:]) #  add not empty chunk to chunks
      current_chunk = [] #  make current chunk blank# just append a line to the current chunk on each iteration
   current_chunk.append(line)

chunks.append(current_chunk)  #  append the last chunk outside the loop

So having file with contents:

foo
asdf
asdf
foo
foo
asdf
asdf
fooo

You'll get this result:

[
    ['foo\n', 'asdf\n', 'asdf\n'], 
    ['foo\n'], 
    ['foo\n', 'asdf\n', 'asdf\n'], 
    ['fooo\n']
]

Solution 2:

You can do it starting from the following code:

>>>"""NAME,ANDREW,AGE 20, BD 1979...NAT ENGLISH...OCC LONDON...INC 200$...NAME,SVEN,AGE 20, BD 1979...NAT SWEDISH...OCC FALUN...INC 100$...NAME,HANS,AGE 30, BD 1988...NAT GERMAN...OCC BERLIN...NOTE, HANDSOME ONE...NAME,LUDOVIC,AGE 40, BD 1955...NAT FRENCH...OCC BORDEAUX...INC 5000$...INTERESTS, FISHING...NAME,PETER...NAT DUTCH...SUMMARY,AGE:20,BD:1979,NAT:DUTCH,OCC:TILBURG,INC:1000$""".split('NAME,')
['', 'ANDREW,AGE 20, BD 1979\nNAT ENGLISH\nOCC LONDON\nINC 200$\n', 'SVEN,AGE 20, BD 1979\nNAT SWEDISH\nOCC FALUN\nINC 100$\n', 'HANS,AGE 30, BD 1988\nNAT GERMAN\nOCC BERLIN\nNOTE, HANDSOME ONE\n', 'LUDOVIC,AGE 40, BD 1955\nNAT FRENCH\nOCC BORDEAUX\nINC 5000$\nINTERESTS, FISHING\n', 'PETER\nNATDUTCH\nSUMMARY,AGE:20,BD:1979,NAT:DUTCH,OCC:TILBURG,INC:1000$']

Also you can use filter function to filter out '' values and list comprehension to make each item a list instead of a string.

Solution 3:

Try this:

token = 'NAME,'

my_list = []
data = []

for line inopen('test.csv').readlines():
    if line.startswith(token):
        if len(data) > 0:
            my_list.append(data)
        data = [line[len(token):]]
    else:
        data.append(line)


if len(data) > 0:
    my_list.append(data)

print my_list

Solution 4:

This one does the trick:

in_string = """NAME,ANDREW,AGE 20, BD 1979
NAT ENGLISH
OCC LONDON
INC 200$
NAME,SVEN,AGE 20, BD 1979
NAT SWEDISH
OCC FALUN
INC 100$
NAME,HANS,AGE 30, BD 1988
NAT GERMAN
OCC BERLIN
NOTE, HANDSOME ONE
NAME,LUDOVIC,AGE 40, BD 1955
NAT FRENCH
OCC BORDEAUX
INC 5000$
INTERESTS, FISHING
NAME,PETER
NAT DUTCH
SUMMARY,AGE:20,BD:1979,NAT:DUTCH,OCC:TILBURG,INC:1000$"""

out_list = []
for chunk in in_string.split('NAME,'):
    out_list.append(chunk.splitlines())
print out_list

Solution 5:

content = open('sample_csv.csv').read()
res = filter(None, [filter(None, i.split('\n')) for i in content.split('NAME,')])

[['ANDREW,AGE 20, BD 1979', 'NAT ENGLISH', 'OCC LONDON', 'INC 200$'], ['SVEN,AGE 20, BD 1979', 'NAT SWEDISH', 'OCC FALUN', 'INC 100$'], ['HANS,AGE 30, BD 1988', 'NAT GERMAN', 'OCC BERLIN', 'NOTE, HANDSOME ONE'], ['LUDOVIC,AGE 40, BD 1955', 'NAT FRENCH', 'OCC BORDEAUX', 'INC 5000$', 'INTERESTS, FISHING'], ['PETER', 'NAT DUTCH', 'SUMMARY,AGE:20,BD:1979,NAT:DUTCH,OCC:TILBURG,INC:1000$']]