Methods, systems, and articles of manufacture for dynamic resource allocation in MapReduce are provided herein. A method includes partitioning input data into one or more sized items of input data associated with a MapReduce job; determining a total number of mapper components, and a total number of
Methods, systems, and articles of manufacture for dynamic resource allocation in MapReduce are provided herein. A method includes partitioning input data into one or more sized items of input data associated with a MapReduce job; determining a total number of mapper components, and a total number of reducer components for the MapReduce job based on said partitioning; dynamically determining an allocation of resources to each of the total number of mapper components and reducer components during run-time of the MapReduce job, wherein said dynamically determining the allocation of resources comprises monitoring one or more utilization parameters for each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job; and dynamically determining a number of concurrently executing mapper components and reducer components from the total number of mapper components and the total number of reducer components for the MapReduce job.
대표청구항▼
1. A method comprising steps of: partitioning input data into one or more sized items of input data associated with a MapReduce job, wherein said partitioning comprises changing the size of the input data based on an execution duration associated with the MapReduce job, wherein said changing compris
1. A method comprising steps of: partitioning input data into one or more sized items of input data associated with a MapReduce job, wherein said partitioning comprises changing the size of the input data based on an execution duration associated with the MapReduce job, wherein said changing comprises (i) increasing the size of the input data if said execution duration is less than a given threshold and (ii) decreasing the size of the input data if said execution duration is greater than the given threshold, and wherein said partitioning comprises ensuring that the number of input data items having a size below a given size is less than a predetermined threshold number;determining a total number of mapper components, and a total number of reducer components for the MapReduce job based on said partitioning;dynamically determining an allocation of resources to each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, wherein said dynamically determining the allocation of resources comprises (i) monitoring one or more utilization parameters for each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, (ii) characterizing a type of bottleneck from among multiple types of bottlenecks based on said monitoring, and (iii) dynamically determining the allocation of resources based at least on the type of bottleneck; anddynamically determining a number of concurrently executing mapper components and reducer components from the total number of mapper components and the total number of reducer components for the MapReduce job;wherein said steps are carried out by at least one computing device. 2. The method of claim 1, wherein said partitioning comprises monitoring the execution duration associated with the MapReduce job. 3. The method of claim 1, wherein said dynamically determining the number of concurrently executing mapper components and reducer components comprises determining the number of concurrently executing mapper components and reducer components to reduce overall execution duration of the MapReduce job. 4. The method of claim 1, wherein said dynamically determining the number of concurrently executing mapper components and reducer components comprises monitoring resource utilization during the MapReduce job. 5. The method of claim 4, wherein said dynamically determining the number of concurrently executing mapper components and reducer components comprises increasing the number of concurrently executing mapper components and reducer components if sufficient idle resources are available based on said monitoring. 6. An article of manufacture comprising a computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising: partitioning input data into one or more sized items of input data associated with a MapReduce job, wherein said partitioning comprises changing the size of the input data based on an execution duration associated with the MapReduce job, wherein said changing comprises (i) increasing the size of the input data if said execution duration is less than a given threshold and (ii) decreasing the size of the input data if said execution duration is greater than the given threshold, and wherein said partitioning comprises ensuring that the number of input data items having a size below a given size is less than a predetermined threshold number;determining a total number of mapper components, and a total number of reducer components for the MapReduce job based on said partitioning;dynamically determining an allocation of resources to each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, wherein said dynamically determining the allocation of resources comprises (i) monitoring one or more utilization parameters for each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, (ii) characterizing a type of bottleneck from among multiple types of bottlenecks based on said monitoring, and (iii) dynamically determining the allocation of resources based at least on the type of bottleneck; anddynamically determining a number of concurrently executing mapper components and reducer components from the total number of mapper components and the total number of reducer components for the MapReduce job. 7. The article of manufacture of claim 6, wherein said partitioning comprises monitoring the execution duration associated with the MapReduce job. 8. The article of manufacture of claim 6, wherein said dynamically determining the number of concurrently executing mapper components and reducer components comprises monitoring resource utilization during the MapReduce job. 9. A system comprising: at least one memory; andat least one processor coupled to the memory and configured for: partitioning input data into one or more sized items of input data associated with a MapReduce job, wherein said partitioning comprises changing the size of the input data based on an execution duration associated with the MapReduce job, wherein said changing comprises (i) increasing the size of the input data if said execution duration is less than a given threshold and (ii) decreasing the size of the input data if said execution duration is greater than the given threshold, and wherein said partitioning comprises ensuring that the number of input data items having a size below a given size is less than a predetermined threshold number;determining a total number of mapper components, and a total number of reducer components for the MapReduce job based on said partitioning;dynamically determining an allocation of resources to each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, wherein said dynamically determining the allocation of resources comprises (i) monitoring one or more utilization parameters for each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job, (ii) characterizing a type of bottleneck from among multiple types of bottlenecks based on said monitoring, and (iii) dynamically determining the allocation of resources based at least on the type of bottleneck; anddynamically determining a number of concurrently executing mapper components and reducer components from the total number of mapper components and the total number of reducer components for the MapReduce job. 10. The system of claim 9, wherein said partitioning comprises monitoring the execution duration associated with the MapReduce job. 11. The system of claim 9, wherein said dynamically determining the number of concurrently executing mapper components and reducer components comprises monitoring resource utilization during the MapReduce job.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (1)
Hardwick Jonathan C.,GBX, Dynamic load balancing among processors in a parallel computer.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.