Fair multi-agent task allocation for large datasets analysis
Autor: | Jean-Christophe Routier, Maxime Morge, Quentin Baert, Anne-Cécile Caron |
---|---|
Přispěvatelé: | Systèmes Multi-Agents et Comportements (SMAC), Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS), This project is supported by the CNRS Challenge Mastodons. |
Jazyk: | angličtina |
Rok vydání: | 2018 |
Předmět: |
Computer science
Distributed computing Big data 02 engineering and technology Negotiation Task (project management) [INFO.INFO-AI]Computer Science [cs]/Artificial Intelligence [cs.AI] Artificial Intelligence 020204 information systems 0202 electrical engineering electronic engineering information engineering MapReduce Reducer business.industry Design pattern Multi-agent system Skew Workload Data structure [INFO.INFO-MO]Computer Science [cs]/Modeling and Simulation Human-Computer Interaction Hardware and Architecture [INFO.INFO-MA]Computer Science [cs]/Multiagent Systems [cs.MA] 020201 artificial intelligence & image processing ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE/I.2.11: Distributed Artificial Intelligence/I.2.11.3: Multiagent systems business Software Information Systems |
Zdroj: | Knowledge and Information Systems (KAIS) Knowledge and Information Systems (KAIS), 2018, 54 (3), pp.591-615. ⟨10.1007/s10115-017-1087-4⟩ Knowledge and Information Systems (KAIS), Springer, 2018, 54 (3), pp.591-615. ⟨10.1007/s10115-017-1087-4⟩ |
ISSN: | 0219-1377 0219-3116 |
Popis: | International audience; MapReduce is a design pattern for processing large datasets distributed on a cluster. Its performances are linked to the data structure and the runtime environment. Indeed, data skew can yield an unfair task allocation, but even when the initial allocation produced by the partition function is well balanced, an unfair allocation can occur during the reduce phase due to the heterogeneous performance of nodes. For these reasons, we propose an adaptive multi-agent system. In our approach, the reducer agents interact during the job and the task reallocation is based on negotiation in order to decrease the workload of the most loaded reducer and so the runtime. In this paper, we propose and evaluate two negotiation strategies. Finally, we experiment our multi-agent system with real-world datasets over heterogeneous runtime environment. |
Databáze: | OpenAIRE |
Externí odkaz: |