This notebook provides a clear introduction to using map reduce for analyzing sales data, specifically in finding the maximum transaction value by store and effectively demonstrates it.
In MapReduce, the *reducer* function processes the output generated by the *mapper* function. Here’s a breakdown of how it works and what calculations it typically performs:
1. *Mapper Function*: This function processes input data and produces a set of key-value pairs. For example, if the task is to count words in a text, the mapper function would output pairs like `("word", 1)` for each occurrence of a word.
2. *Shuffle and Sort*: After mapping, the MapReduce framework shuffles and sorts these key-value pairs by key, grouping together all values associated with the same key. This organizes the data so that the reducer can work on each key individually.
3. *Reducer Function*: The reducer function then takes each key and the list of values associated with it. It performs a calculation, typically an aggregation, on this list of values.
- **Example Calculation (Sum)**: For a word count, if the key is `"word"`, the reducer would receive all the `1`s associated with that word and sum them up, giving the total count for that word.
- **Other Calculations**: The reducer might perform other aggregations like finding the maximum or minimum value, averaging, or concatenating values depending on the task.
In summary, the reducer aggregates or processes each key’s values returned by the mapper function, completing the overall transformation.
The reducer function takes the data returned by the mapper function, which includes the 'Store' and 'Cost' columns, and groups it by 'Store'. It then calculates the maximum 'Cost' for each store, effectively identifying the highest transaction value for each location.
