One example method includes identifying synchronous code including instructions specifying a computing operation to be performed on a set of data; transforming the synchronous code into a pipeline application including one or more pipeline objects; identifying a first input data set on which to exec
One example method includes identifying synchronous code including instructions specifying a computing operation to be performed on a set of data; transforming the synchronous code into a pipeline application including one or more pipeline objects; identifying a first input data set on which to execute the pipeline application; executing the pipeline application on a first input data set to produce a first output data set; after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application; determining a set of differences between the first input data set and second input data set; and executing the pipeline application on the set of differences to produce a second output data set.
대표청구항▼
1. A computer-implemented method executed by one or more processors, the method comprising: identifying synchronous code including instructions specifying a computing operation to be performed on a set of data;transforming the synchronous code into a pipeline application including one or more pipeli
1. A computer-implemented method executed by one or more processors, the method comprising: identifying synchronous code including instructions specifying a computing operation to be performed on a set of data;transforming the synchronous code into a pipeline application including one or more pipeline objects, the pipeline application configured to be executed in parallel across a plurality of computing devices, each of the one or more pipeline objects configured to receive an input data set and produce an output data set;identifying a first input data set on which to execute the pipeline application;executing the pipeline application on a first input data set to produce a first output data set, the executing the pipeline application including executing each of the one or more pipeline objects in an order in which a previous pipeline object provides its output data set to a next pipeline object as its input data set;after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application;determining a set of differences between the first input data set and second input data set; andexecuting the pipeline application on the set of differences to produce a second output data set, the executing the pipeline application on the set of differences including executing each of the one or more pipeline objects includes each previous pipeline object in the order providing differences from its previous output data set to the next pipeline object as its input data set, and the second output data set including differences from the first output data set. 2. The method of claim 1, further comprising determining a pipeline state in response to executing the pipeline on the first input data set, the pipeline state including a representation of the first input data set and the first output data set. 3. The method of claim 2, further comprising updating the pipeline state in response to executing the pipeline on the set of differences from the first input data set to generate an updated pipeline state, the updated pipeline state including a representation of the second input data set and the second output data set. 4. The method of claim 1, further comprising determining a pipeline object state for each of the one or more pipeline objects in response to executing the pipeline on the first input data set, the pipeline object state including a representation of the input data set and the output data set for the pipeline object. 5. The method of claim 4, further comprising updating the pipeline object state in response to executing the pipeline on the set of differences from the first input data set to generate an updated pipeline object state, the updated pipeline object state including differences from the input data set and the output data set for the pipeline object. 6. The method of claim 1, wherein identifying the first input data set on which to execute the pipeline comprises: transforming the first input data set into a first set of key value pairs; andstoring the first set of key value pairs in a key value store. 7. The method of claim 1, wherein determining the set of differences between the first input data set and second input data set comprises: transforming the second input data set into a second set of key value pairs;comparing the second set of key value pairs to first set of key value pairs; andidentifying key value pairs that have been added or deleted from the second set of key value pairs relative to the first set of key value pairs. 8. The method of claim 1, wherein determining the set of differences between the first input data set and second input data set comprises: determining a last execution timestamp for the pipeline representing a time at which the pipeline was executed on the first input data set; andidentifying a set of items in the second input data set including timestamps after the last execution timestamp. 9. A non-transitory, computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations comprising: identifying synchronous code including instructions specifying a computing operation to be performed on a set of data;transforming the synchronous code into a pipeline application including one or more pipeline objects, the pipeline application configured to be executed in parallel across a plurality of computing devices, each of the one or more pipeline objects configured to receive an input data set and produce an output data set;identifying a first input data set on which to execute the pipeline application;executing the pipeline application on a first input data set to produce a first output data set, the executing the pipeline application including executing each of the one or more pipeline objects in an order in which a previous pipeline object provides its output data set to a next pipeline object as its input data set;after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application;determining a set of differences between the first input data set and second input data set; andexecuting the pipeline application on the set of differences to produce a second output data set, the executing the pipeline application on the set of differences including executing each of the one or more pipeline objects includes each previous pipeline object in the order providing differences from its previous output data set to the next pipeline object as its input data set, and the second output data set including differences from the first output data set. 10. The computer-readable medium of claim 9, the operations further comprising determining a pipeline state in response to executing the pipeline on the first input data set, the pipeline state including a representation of the first input data set and the first output data set. 11. The computer-readable medium of claim 10, the operations further comprising updating the pipeline state in response to executing the pipeline on the set of differences from the first input data set to generate an updated pipeline state, the updated pipeline state including a representation of the second input data set and the second output data set. 12. The computer-readable medium of claim 9, the operations further comprising determining a pipeline object state for each of the one or more pipeline objects in response to executing the pipeline on the first input data set, the pipeline object state including a representation of the input data set and the output data set for the pipeline object. 13. The computer-readable medium of claim 12, the operations further comprising updating the pipeline object state in response to executing the pipeline on the set of differences from the first input data set to generate an updated pipeline object state, the updated pipeline object state including differences from the input data set and the output data set for the pipeline object. 14. The computer-readable medium of claim 9, wherein identifying the first input data set on which to execute the pipeline comprises: transforming the first input data set into a first set of key value pairs; andstoring the first set of key value pairs in a key value store. 15. The computer-readable medium of claim 9, wherein determining the set of differences between the first input data set and second input data set comprises: transforming the second input data set into a second set of key value pairs;comparing the second set of key value pairs to first set of key value pairs; andidentifying key value pairs that have been added or deleted from the second set of key value pairs relative to the first set of key value pairs. 16. The computer-readable medium of claim 9, wherein determining the set of differences between the first input data set and second input data set comprises: determining a last execution timestamp for the pipeline representing a time at which the pipeline was executed on the first input data set; andidentifying a set of items in the second input data set including timestamps after the last execution timestamp. 17. A system comprising: memory for storing data; andone or more processors operable to perform operations comprising: identifying synchronous code including instructions specifying a computing operation to be performed on a set of data;transforming the synchronous code into a pipeline application including one or more pipeline objects, the pipeline application configured to be executed in parallel across a plurality of computing devices, each of the one or more pipeline objects configured to receive an input data set and produce an output data set;identifying a first input data set on which to execute the pipeline application;executing the pipeline application on a first input data set to produce a first output data set, the executing the pipeline application including executing each of the one or more pipeline objects in an order in which a previous pipeline object provides its output data set to a next pipeline object as its input data set;after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application;determining a set of differences between the first input data set and second input data set; andexecuting the pipeline application on the set of differences to produce a second output data set, the executing the pipeline application on the set of differences including executing each of the one or more pipeline objects includes each previous pipeline object in the order providing differences from its previous output data set to the next pipeline object as its input data set, and the second output data set including differences from the first output data set. 18. The system of claim 17, the operations further comprising determining a pipeline state in response to executing the pipeline on the first input data set, the pipeline state including a representation of the first input data set and the first output data set. 19. The system of claim 18, the operations further comprising updating the pipeline state in response to executing the pipeline on the set of differences from the first input data set to generate an updated pipeline state, the updated pipeline state including a representation of the second input data set and the second output data set. 20. The system of claim of claim 17, the operations further comprising determining a pipeline object state for each of the one or more pipeline objects in response to executing the pipeline on the first input data set, the pipeline object state including a representation of the input data set and the output data set for the pipeline object.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (5)
Wang,Albert Ren Rui; Ruddell,Richard; Goodwin,David William; Killian,Earl A.; Bhattacharyya,Nupur; Medina,Marines Puig; Lichtenstein,Walter David; Konas,Pavlos; Srinivasan,Rangarajan; Songer,Christop, Automated processor generation system for designing a configurable processor and method for the same.
Shih, Kathryn Marie; Moore, Eider Brantly; McKnight, Richard Rex; Aggarwal, Vaibhav; Sirota, Peter; Cole, Richard Jeffrey; Bartlett, James P.; Christofferson, Carl Louis, Connector interface for data pipeline.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.