At a recent presentation by Jay Kreps of LinkedIn at the Hadoop Sumit 2010. Jay describes how LinkedIn crunches 120 billion relationships per day and blends large scale data computation with high volume, low latency site serving.
The Search, Network, and Analytics (SNA) team at LinkedIn works on LinkedIn's information retrieval systems, the social graph system, data driven features, and supporting data infrastructure. The system uses a number of open source software products such as- Apache Hadoop - framework for running applications on large clusters of commodity hardware.
- Zoie - Faceted search
- Bobo - Real-time search indexing
- Decomposer - Very large matrix decomposition routines
- Norbert - Partition aware cluster management & RPC
- Voldermort - Key/Value storage
- Kamikaze - Compression package
- Sensei - Distributed realtime search
- Azkaban - Hadoop workflow