Method and apparatus for a local competitive learning rule that leads to sparse connectivity
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06N-003/10
G06N-003/08
G06N-003/06
G05B-013/02
출원번호
US-0166269
(2011-06-22)
등록번호
US-9129222
(2015-09-08)
발명자
/ 주소
Aparin, Vladimir
출원인 / 주소
QUALCOMM Incorporated
대리인 / 주소
Patel, Rupit M.
인용정보
피인용 횟수 :
0인용 특허 :
2
초록▼
Certain aspects of the present disclosure support a local competitive learning rule applied in a computational network that leads to sparse connectivity among processing units of the network. The present disclosure provides a modification to the Oja learning rule, modifying the constraint on the sum
Certain aspects of the present disclosure support a local competitive learning rule applied in a computational network that leads to sparse connectivity among processing units of the network. The present disclosure provides a modification to the Oja learning rule, modifying the constraint on the sum of squared weights in the Oja rule. This constraining can be intrinsic and local as opposed to the commonly used multiplicative and subtractive normalizations, which are explicit and require the knowledge of all input weights of a processing unit to update each one of them individually. The presented rule provides convergence to a weight vector that is sparser (i.e., has more zero elements) than the weight vector learned by the original Oja rule. Such sparse connectivity can lead to a higher selectivity of processing units to specific features, and it may require less memory to store the network configuration and less energy to operate it.
대표청구항▼
1. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local trainin
1. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the computational network comprises a neural network, and the processing unit comprises a neuron circuit. 2. The method of claim 1, wherein changing the at least one weight using the local training rule comprises updating the at least one weight based on at least one of: one or more inputs in the processing unit, the output, or the at least one existing weight before the change. 3. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero, wherein changing the at least one weight using the local training rule comprises updating the at least one weight based on at least one of: one or more inputs in the processing unit, the output, or the at least one existing weight before the change, and wherein the sparse connectivity is created after multiple updating of the at least one weight. 4. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule imposes a constraint on a sum of absolute values of the weights. 5. The method of claim 4, wherein the constrained weights are associated with input connections to the processing unit. 6. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule constrains an input weight vector associated with input connections to the processing unit at an equilibrium point. 7. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule constrains a number of nonzero weights. 8. The method of claim 7, wherein the constrained weights are associated with input connections to the processing unit. 9. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the processing unit utilizes one or more non-linear operations. 10. A method of training a computational network, comprising: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule imposes bounds on individual weights associated with input connections to the processing unit. 11. The method of claim 10, wherein a maximum value of the individual weights is bounded by an upper bound. 12. The method of claim 10, wherein a minimum value of the individual weights is bounded by a lower bound. 13. The method of claim 10, wherein both maximum and minimum values of the individual weights are bounded. 14. The method of claim 10, wherein the bounds vary for the individual weights. 15. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the computational network comprises a neural network, and the apparatus comprises a neuron circuit. 16. The apparatus of claim 15, wherein the at least one processing unit is also configured to update the at least one weight based on at least one of: one or more inputs in the apparatus, the output, or the at least one existing weight before the change. 17. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero, wherein the at least one processing unit is also configured to update the at least one weight based on at least one of: one or more inputs in the apparatus, the output, or the at least one existing weight before the change, and wherein the sparse connectivity is created after multiple updating of the at least one weight. 18. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule imposes a constraint on a sum of absolute values of the weights. 19. The apparatus of claim 18, wherein the constrained weights are associated with input connections to the apparatus. 20. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule constrains an input weight vector associated with input connections to the apparatus at an equilibrium point. 21. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; anda change the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule constrains a number of nonzero weights. 22. The apparatus of claim 21, wherein the constrained weights are associated with input connections to the apparatus. 23. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the apparatus utilizes one or more non-linear operations. 24. An apparatus of a computational network, comprising: at least one processing unit configured to:compute an output of the apparatus in the computational network based at least in part on at least one existing weight; andchange the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule imposes bounds on individual weights associated with input connections to the apparatus. 25. The apparatus of claim 24, wherein a maximum value of the individual weights is bounded by an upper bound. 26. The apparatus of claim 24, wherein a minimum value of the individual weights is bounded by a lower bound. 27. The apparatus of claim 24, wherein both maximum and minimum values of the individual weights are bounded. 28. The apparatus of claim 24, wherein the bounds vary for the individual weights. 29. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the computational network comprises a neural network, and the apparatus comprises a neuron circuit. 30. The apparatus of claim 29, wherein the means for changing the at least one weight using the local training rule comprises means for updating the at least one weight based on at least one of: one or more inputs in the apparatus, the output, or the at least one existing weight before the change. 31. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero, wherein the means for changing the at least one weight using the local training rule comprises means for updating the at least one weight based on at least one of: one or more inputs in the apparatus, the output, or the at least one existing weight before the change, and wherein the sparse connectivity is created after multiple updating of the at least one weight. 32. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule imposes a constraint on a sum of absolute values of the weights. 33. The apparatus of claim 32, wherein the constrained weights are associated with input connections to the apparatus. 34. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule constrains an input weight vector associated with input connections to the apparatus at an equilibrium point. 35. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule constrains a number of nonzero weights. 36. The apparatus of claim 35, wherein the constrained weights are associated with input connections to the apparatus. 37. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the apparatus utilizes one or more non-linear operations. 38. An apparatus of a computational network, comprising: means for computing an output of the apparatus in the computational network based at least in part on at least one existing weight; andmeans for changing the at least one weight of the apparatus using a local training rule, wherein the local training rule creates sparse connectivity between apparatuses of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the apparatuses to a defined value when an average change of the weights equals zero,wherein the local training rule imposes bounds on individual weights associated with input connections to the apparatus. 39. The apparatus of claim 38, wherein a maximum value of the individual weights is bounded by an upper bound. 40. The apparatus of claim 38, wherein a minimum value of the individual weights is bounded by a lower bound. 41. The apparatus of claim 38, wherein both maximum and minimum values of the individual weights are bounded. 42. The apparatus of claim 38, wherein the bounds vary for the individual weights. 43. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the computational network comprises a neural network, and the processing unit comprises a neuron circuit. 44. The computer program product of claim 43, wherein the computer-readable medium further comprising code for updating the at least one weight based on at least one of: one or more inputs in the processing unit, the output, or the at least one existing weight before the change. 45. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero, wherein the computer-readable medium further comprising code for updating the at least one weight based on at least one of: one or more inputs in the processing unit, the output, or the at least one existing weight before the change, and wherein the sparse connectivity is created after multiple updating of the at least one weight. 46. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule imposes a constraint on a sum of absolute values of the weights. 47. The computer program product of claim 46, wherein the constrained weights are associated with input connections to the processing unit. 48. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule constrains an input weight vector associated with input connections to the processing unit at an equilibrium point. 49. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule constrains a number of nonzero weights. 50. The computer program product of claim 49, wherein the constrained weights are associated with input connections to the processing unit. 51. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the processing unit utilizes one or more non-linear operations. 52. A computer program product for training a computational network, comprising a non-transitory computer-readable medium comprising code for: computing an output of a processing unit in the computational network based at least in part on at least one existing weight; andchanging the at least one weight of the processing unit using a local training rule, wherein the local training rule creates sparse connectivity between processing units of the computational network by constraining a norm of a weight vector comprising weights associated with connections between the processing units to a defined value when an average change of the weights equals zero,wherein the local training rule imposes bounds on individual weights associated with input connections to the processing unit. 53. The computer program product of claim 52, wherein a minimum value of the individual weights is bounded by a lower bound. 54. The computer program product of claim 52, wherein both maximum and minimum values of the individual weights are bounded. 55. The computer program product of claim 52, wherein the bounds vary for the individual weights. 56. The computer program product of claim 52, wherein a maximum value of the individual weights is bounded by an upper bound.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (2)
Robert W. Means ; Richard Calmbach, Cortronic neural networks with distributed processing.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.