2d Array To Represent A Huge Python Dict, Coordinate Like Solution To Save Memory

I try to update a dict_with_tuples_key with the data from an array: myarray = np.array([[0, 0], # 0, 1 [0, 1], [1, 1], # 1, 2

Solution 1:

Use dok_matrix from scipy; a dock_matrix is a dictionary Of Keys based sparse matrix. They allow you to build sparse matrices incrementally and they won't allocate huge empty darray = np.zeros((xlen, ylen)) that does not fit into your computer memory.

The only change to do is to import the right module from scipy and to change the definition of darray in your function convert_dict_to_darray.

It will look like this:

from scipy.sparse import dok_matrix

def convert_dict_to_darray(dict_with_tuples_key, myarray):
    idx_max_array = np.max(myarray, axis=0)
    idx_max_dict  = np.max(dict_with_tuples_key.keys(), axis=0)
    lens = np.max([list(idx_max_array), list(idx_max_dict)], axis=0)
    xlen, ylen = lens[0] + 1, lens[1] + 1
    darray = dok_matrix( (xlen, ylen) )
    for key, value in dict_with_tuples_key.items():
        darray[key[0], key[1]] = value
    return darray