Samat Jain's personal home page

I’m writing a load balancing algorithm for this cluster I’m working on, and have a bunch of things in a Python list sequence. So came the issue, how to divide the list into sublists for each node in the cluster to compute? I came up with the algorithm:

def split_seq_stride(l, n):
  """Splits a sequence l into a list of subsequences containing at least n
  elements each, not preserving order. The first few subsequences may contain
  n+1 elements, containing the last few elements. (Algorithm is good for load
  balancing)"""
  r=[]
  k=len(l)/n
  [r.append(l[i::k]) for i in range(k)]
  return r

which behaves like:

Assuming each task is an equal amount of work, this algorithm is a good load balancing algorithm. it prevents any node from having to do any less significant work than the others.

Probably useful for someone: what about splitting up a sequence in-order, with the last subsequence containing fewer elements? This code segment does just that:

def split_seq(l, n):
  """Splits a sequence l into a list of subsequences containing at most n
  elements each, preserving order. The last subsequence may contain less
  than n elements."""
  r=[]
  [r.append(l[s:s+n]) for s in range(0, len(l), n)]
  return r

which behaves like:

Thanks to yason and others on EFnet’s #python for pointers.