Systems and methods for distributed system scanning
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06F-007/00
G06F-017/00
출원번호
US-0837439
(2010-07-15)
등록번호
US-8214334
(2012-07-03)
발명자
/ 주소
Mikesell, Paul A.
Anderson, Robert J.
Godman, Peter J.
Schack, Darren P.
Dire, Nathan E.
출원인 / 주소
EMC Corporation
대리인 / 주소
Knobbe Martens Olson & Bear, LLP
인용정보
피인용 횟수 :
3인용 특허 :
235
초록▼
Systems and methods are provided for scanning files and directories in a distributed file system on a network of nodes. The nodes include metadata with attribute information corresponding to files and directories distributed on the nodes. In one embodiment, the files and directories are scanned by c
Systems and methods are provided for scanning files and directories in a distributed file system on a network of nodes. The nodes include metadata with attribute information corresponding to files and directories distributed on the nodes. In one embodiment, the files and directories are scanned by commanding the nodes to search their respective metadata for a selected attribute. At least two of the nodes are capable of searching their respective metadata in parallel. In one embodiment, the distributed file system commands the nodes to search for metadata data structures having location information corresponding to a failed device on the network. The metadata data structures identified in the search may then be used to reconstruct lost data that was stored on the failed device.
대표청구항▼
1. A method for handling a failure in a distributed file system, the method comprising: for at least one file of a plurality of files, storing a the file across one or more memory devices of a plurality of nodes in a network, at least some of the files distributed in blocks across multiple nodes;cre
1. A method for handling a failure in a distributed file system, the method comprising: for at least one file of a plurality of files, storing a the file across one or more memory devices of a plurality of nodes in a network, at least some of the files distributed in blocks across multiple nodes;creating a data block location information structure associated with the file and listing, for all of the blocks in the file, the memory devices and physical addresses where the blocks are stored;creating a consolidated device list which is associated with the file and which comprises an abbreviated listing of each of the one or more memory devices that are listed in the data block location information structure, but which does not include a duplicate listing of a particular memory device for all of the blocks in the file that are stored on the particular memory device, wherein the device list is separate from and includes less information than the data block location information structure;storing metadata data structures across the plurality of nodes, at least two of the metadata data structures stored on different nodes, each of the metadata data structures referencing information about a corresponding one of the plurality of files, and the metadata data structures comprising the data block location information structure and the consolidated device list of the corresponding file;detecting that one of the memory devices in the network has failed;using one or more processors, commanding the nodes to search each consolidated device list stored in each of their respective metadata data structures to determine whether the failed memory device is used to store information referenced by their respective metadata data structures, wherein searching each consolidated device list allows the nodes to make the determination without searching through the data block location information structure, which includes more information than the consolidated device list; andreceiving an indication from the nodes of which metadata data structures reference information stored on the failed device. 2. The method of claim 1, further comprising commanding the nodes to access the identified metadata data structures to reconstruct the information stored on the failed device. 3. The method of claim 1, wherein the files comprise files and directories. 4. The method of claim 1, wherein at least some of the nodes perform at least a portion of the search of the consolidated lists of their respective metadata data structures in parallel with at least a portion of the search of one or more other nodes. 5. The method of claim 1, wherein accessing the identified metadata data structures comprises reading pointers to at least one of file content blocks, metadata data blocks, and protection data blocks stored on the failed memory device. 6. The system of claim 1, wherein the consolidated device list includes a single reference to each of the one or more memory devices that is listed in the data block location information structure, regardless of the number of blocks of the associated file that are stored on the particular memory device. 7. A distributed file system, comprising: a plurality of nodes in a network, each comprising: one or more processors; andat least one memory device, wherein: a plurality of files is stored across the memory devices of the plurality of nodes, at least some of the files distributed in blocks across multiple nodes, andmetadata data structures are stored across the memory devices of the plurality of nodes, at least two of the metadata data structures stored on different nodes, each of the metadata data structures referencing information about a corresponding one of the files, and the respective metadata data structures comprising: a data block location information structure associated with the file corresponding to the respective metadata data structure and listing, for all of the blocks in the file, the memory devices and physical addresses where the blocks in the file are stored; anda consolidated device list associated with the file corresponding to the respective metadata data structure and which comprises an abbreviated listing of each of one or more memory devices that are listed in the data block location information structure, wherein the consolidated device list does not include a duplicate listing of a particular memory device for all of the blocks in the file that are stored on the particular memory device, and is separate from and includes less information than the data block location information structure; andwherein the one or more processors of at least one of the nodes are configured to: detect that one of the memory devices in the network has failed;command the nodes to search each consolidated device list stored in each of their respective metadata data structures to determine whether the failed memory device is used to store information referenced by their respective metadata data structures, wherein searching each consolidated device list allows the nodes to make the determination without searching through the data block location information structure, which includes more information than the consolidated device list; andreceive an indication from the nodes of which metadata data structures reference information stored on the failed device. 8. The system of claim 7, wherein the one or more processors of the at least one of the nodes are further configured to command the nodes to access the identified metadata data structures to reconstruct the information stored on the failed device. 9. The system of claim 7, wherein the files comprise files and directories. 10. The system of claim 7, wherein at least some of the nodes perform at least a portion of the search of the consolidated lists of their respective metadata data structures in parallel with at least a portion of the search of one or more other nodes. 11. The system of claim 7, wherein the nodes are configured to access the identified metadata data structures at least in part by reading pointers to at least one of file content blocks, metadata data blocks, and protection data blocks stored on the failed memory device. 12. The system of claim 7, wherein the consolidated device lists each include a single reference to each one or more memory devices that is listed in the corresponding data block location information structure, regardless of the number of blocks of the associated file that are on the particular memory device.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (235)
McGoveran,David O., Adaptive transaction manager for complex transactions and business process.
Stirpe Paul Alan ; Verma Dinesh Chandra ; Nadas Stephen Joseph ; Gupta Manish ; Hervatic Elizabeth A., Automatic reconfiguration of multipoint communication channels.
Alverson, Gail A.; Callahan, II, Charles David; Kahan, Simon H.; Koblenz, Brian D.; Porterfield, Allan; Smith, Burton J., Detecting access to a memory location in a multithreaded environment.
Watanabe Naoki,JPX ; Kakuta Hitoshi,JPX ; Takamoto Yoshifumi,JPX, Disk array system having adjustable parity group sizes based on storage unit capacities.
Smith Robert M. (Andover MA) Ting David M. T. (Sudbury MA) Boer Jan H. (Lexington MA) Mendelssohn Marvin (Melrose MA), Document management and production system.
Leblang David B. (Wayland MA) Allen Larry W. (Cambridge MA) Chase ; Jr. Robert P. (Newton MA) Douros Bryan P. (Framingham MA) Jabs David E. (Sudbury MA) McLean ; Jr. Gordon D. (Brookline MA) Minard D, Dynamic rule-based version control system.
Solter,Nicholas A.; Kong,Wei; Rao,Anil; Tripathi,Ashutosh, Facilitating event notification through use of an inverse mapping structure for subset determination.
Napolitano Richard ; Silverman Herbert W. ; Juzsczak Chester ; Panner Bryan K. ; Franklin Chris ; Noya Eric S. ; Hoskins Timothy Lee ; Luke Stanley ; Shaughnessy Paul Richard ; McLeod Alexander C. ; , File array storage architecture having file system distributed across a data processing platform.
Belsan Jay S. (Nederland CO) Laughlin Jeffrey S. (Nederland CO) Pedersen Mogens H. (Longmont CO) Raicer Robert J. (Niwot CO) Rudeseal George A. (Boulder CO) Schafer Charles P. (Louisville CO) Steele , File server having snapshot application data groups.
Akizawa Mitsuru (Hachioji JPX) Yamashita Hirofumi (Yokohama JPX) Kawaguchi Hisamitsu (Sagamihara JPX) Tada Katsumi (Yokohama JPX) Kato Kanji (Yokohama JPX) Kito Akira (Ebina JPX) Yamada Hidenori (Had, File server system and file access control method of the same.
Christensen Steve W. ; Dasher John M. ; Martherus Robin E., File wrapper containing cataloging information for content searching across multiple platforms.
Nunnelley Lewis L. (San Jose CA) Williams Larry L. (Los Altos CA) Wood ; Jr. Leighton C. (Morgan Hill CA), High capacity data storage system using disk array.
Calvignac Jean (La Gaude FRX) Saint Georges Eric (La Gaude FRX) Orsatti Daniel (Cagnes sur Mer FRX) Toubol Gilles (Villeneuve Loubet FRX) Verplanken Fabrice (Le Haute de Cagnes FRX) Nicolas Francois , Hybrid switching system for a communication node.
Perycz,Krzysztof S.; Iwanojko,Bohdan T.; Kamiński,Adam; Kogut,Jaroslaw; Oriol,Mariusz; Przekop,Zbigniew, Initialization, reconfiguration, and shut down of a module function.
Lincoln, Patrick D.; Dawson, Steven M.; Samarati, Pierangela; De Capitani di Vimercati, Sabrina, Lattice-based security classification system and method.
Humlicek, Donald R.; DeKoning, Rodney A.; Delaney, William P., Managing a snapshot volume or one or more checkpoint volumes with multiple point-in-time images in a single repository.
Cabrera Luis Felipe ; Long Darrell Don Earl, Method and apparatus for establishing and maintaining the status of membership sets used in mirrored read and write inpu.
Gondi, Albert C.; Klein, Johannes; de Roo, John S.; Lanka, Sitaram V.; Sripada, Ramprasad K. L., Method and apparatus for handling failures of resource managers in a clustered environment.
Philip E. Tamer ; Terry Seto Lee, Method and apparatus for identifying changes to a logical object based on changes to the logical object at physical level.
Liu,Yu Jih; Visvader,Joseph John; Schnabel,Jon William, Method and apparatus for on demand multicast and unicast using controlled flood multicast communications.
Matthew J. D'Errico ; Steven M. Blumenau ; Erez Ofer, Method and apparatus for providing a host computer with information relating to the mapping of logical volumes within an intelligent storage system.
Bond Milton F. (Rochester MN) Clark Brian E. (Rochester MN) McRoberts Raymond S. (Rochester MN), Method and apparatus for recovering parity protected data.
Fagen Scott Andrew ; Frey Jeffrey Alan ; Fulkerson ; Jr. Carroll Eugene ; Kowalski Mark Albert ; North Benjamin John,IEX, Method and apparatus for serializing resource access requests in a multisystem complex.
Menon Jaishankar M. (San Jose CA) Mattson Richard L. (San Jose CA) Ng Spencer W. (San Jose CA), Method and means for distributed sparing in DASD arrays.
Huffman,William A.; Anderson,Michael L.; Thorson,Gregory M.; Garcia,Susan; Kunkel,Daniel L., Method and system for covering multiple resourcces with a single credit in a computer system.
Litwin Witold,FRX ; Menon Jaishankar Moothedath ; Risch Tore Johan Martin,SEX, Method and system for data recovery using a distributed and scalable data structure.
Brent Cameron Beardsley ; Michael Thomas Benhase ; Douglas A. Martin ; Robert Louis Morton ; Kenneth Wayne Todd, Method and system for managing meta data.
Frey ; Jr. Alexander H. (Pasadena CA) Mosteller Richard C. (Sierra Madre CA), Method for balancing of distributed tree file structures in parallel computing systems to enable recovery after a failur.
Stallmo David C. ; Hall Randy K., Method for organizing storage devices of unequal storage capacity and distributing data using different raid formats depending on size of rectangles containing sets of the storage devices.
Gentry Timothy W. (Wichita KS) Fredin Gerald J. (Wichita KS) Riedl Daniel A. (Andover KS), Method for partitioning disk drives within a physical disk array and selectively assigning disk drive partitions into a.
Burton, David Alan; Morton, Robert Louis, Method, system, program, and data structures for mapping logical units to a storage space comprises of at least one array of storage units.
Bingham, Scott Forrest; Buchman, Matthew D.; Singhal, Upanshu; Rokicki, John C.; Murthy, Venkatesha, Mirrored storage architecture using continuous data protection techniques.
Marcelo Weinberger ; Tomas G. Rokicki ; Gadiel Seroussi ; Rajiv Gupta ; Neri Merhav IL; Joesp M. Ferrandiz, Optimizing computer performance by using data compression principles to minimize a loss function.
Row Edward J. (Mountain View CA) Boucher Laurence B. (Saratoga CA) Pitts William M. (Los Altos CA) Blightman Stephen E. (San Jose CA), Parallel I/O network file server architecture.
Popelka Paul ; Tripathy Tarun Kumar ; Walter Richard Allen ; Del Fante Paul Brian ; Repakula Murali Sundaramoorthy ; Narayanaswamy Lakshman ; Sterk Donald Wayne ; Bodas Amod Prabhakar ; McCutcheon Le, Processing system with dynamically allocatable buffer memory.
Hinshaw, Foster D.; Meyers, David L.; Zane, Barry M., Programmable streaming data processor for database appliance having multiple processing unit groups.
Blackmon, Herman Lee; Drehmel, Robert Allen; Haselhorst, Kent Harold; Marcella, James Anthony, Reordering and flushing commands in a computer memory subsystem.
Bauer,Andreas L.; Gruttadauria,Brian R.; Lazar,Gregory W.; Dobberpuhl,Walter T., Scalable communication within a distributed system using dynamic communication trees.
Rafert,James Lee; Martin,Marcia R.; Abramovitz,Michael Paul, Storage backup system for backing up data written to a primary storage device to multiple virtual mirrors using a reconciliation process that reflects the changing state of the primary storage device.
Manley,Stephen L.; Owara,Shane S., System and method for asynchronous mirroring of snapshots at a destination using a purgatory directory and inode mapping.
Benson Max L. ; Morais Dinarte ; Norin Scott ; Champion William P. ; Fakes Thomas F. ; Joshi Milind M., System and method for incremental change synchronization between multiple copies of data.
Heil Thomas F. ; Francis Martin H. ; DeKoning Rodney A. ; Weber Bret S., System and method for peer-to-peer accelerated I/O shipping between host bus adapters in clustered computer network.
Kekre,Anand A.; Pendbarkar,Niranjan S., System and method for recording the order of a change caused by restoring a primary volume during ongoing replication of the primary volume.
Mark S. Day ; Donald J. Brady ; Deric S. Horn, System and method for storing and retrieving filenames and files in computer memory using multiple encodings.
Zayas, Edward R.; Haynes, Thomas; Gillono, John Francis; Kahn, Andy C., System and method for verifying and restoring the consistency of inode to pathname mappings in a filesystem.
Patel, Sujal M.; Mikesell, Paul A., System and methods for providing a distributed file system utilizing metadata to track information about data stored throughout the system.
Klein Johannes (San Francisco CA) Lutgardo Alberto (Santa Clara CA) Chang Edward Y. (Santa Clara CA) Cheng Edward C. (S. San Francisco CA) Lee Dora L. (San Francisco CA) Lu Edward S. (San Bruno CA), System for distributed computation processing includes dynamic assignment of predicates to define interdependencies.
Bertin Olivier (Nice FRX) Chobert Jean-Paul (Carros FRX) Pruvost Alain (Valauris FRX), System for managing topology of a network in spanning tree data structure by maintaining link table and parent table in.
Anderson, Robert J.; Fachan, Neal T.; Husted, Justin M.; Lemar, Eric M.; Passey, Aaron J.; Schack, Darren P., Systems and methods for a snapshot of data.
Anderson, Robert J.; Fachan, Neal T.; Husted, Justin M.; Lemar, Eric M.; Passey, Aaron J.; Schack, Darren P., Systems and methods for a snapshot of data.
Anderson, Robert J.; Fachan, Neal T.; Lemar, Eric M.; Passey, Aaron J.; Richards, David W.; Schack, Darren P., Systems and methods for a snapshot of data.
Anderson, Robert J.; Fachan, Neal T.; Lemar, Eric M.; Passey, Aaron J.; Richards, David W.; Schack, Darren P., Systems and methods for a snapshot of data.
Passey, Aaron J.; Schack, Darren P.; Godman, Peter J.; Anderson, Robert J.; Fachan, Neal T., Systems and methods for accessing and updating distributed data.
Patel,Sujal M.; Mikesell,Paul A.; Schack,Darren P.; Passey,Aaron J., Systems and methods for providing a distributed file system incorporating a virtual hot spare.
Patel, Sujal M.; Mikesell, Paul A.; Schack, Darren P., Systems and methods for providing a distributed file system utilizing metadata to track information about data stored throughout the system.
Mikesell, Paul A.; Anderson, Rob; Passey, Aaron James; Godman, Peter John; Khan, Hassan F.; Schack, Darren P., Systems and methods for restriping files in a distributed file system.
Akidau, Tyler Arthur; Dire, Nate E.; Fachan, Neal T.; Godman, Peter J.; Loafman, Zachary M., Systems and methods of managing resource utilization on a threaded computer system.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.