We use cookies to ensure you have the best browsing experience on our website. Please read our cookie policy for more information about how we use cookies.
A more useful way to think about partitions is as chunks of work to be done by workers. When a map worker has finished mapping a parition, it can grab a new one from a coordinator and start working on that. This approach has a number of advantages, including workload balance (if you have a really fast worker then it may finish before other workers and would just be sitting there idle), and it allows for workers to not pull in enormous chunks of data that they would have to swap in and out of disk whenever doing work. By partitioning the data into smaller chunks, the workers can keep more data in memory which is much faster.
Cookie support is required to access HackerRank
Seems like cookies are disabled on this browser, please enable them to open this website
MapReduce - 3
You are viewing a single comment's thread. Return to all comments →
A more useful way to think about partitions is as chunks of work to be done by workers. When a map worker has finished mapping a parition, it can grab a new one from a coordinator and start working on that. This approach has a number of advantages, including workload balance (if you have a really fast worker then it may finish before other workers and would just be sitting there idle), and it allows for workers to not pull in enormous chunks of data that they would have to swap in and out of disk whenever doing work. By partitioning the data into smaller chunks, the workers can keep more data in memory which is much faster.