FRIHOST FORUMS SEARCH FAQ TOS BLOGS COMPETITIONS
You are invited to Log in or Register a free Frihost Account!


Storage of Massive Amounts of Data





polly-gone
I was wondering how places like Facebook probably store massive amounts of data.

For example, sometime today, the 200 Millionth member is going to join, and that is obviously a lot of people. But then there are probably hundreds of thousands if not millions of groups and people can join as many groups as they want.

How would a place like Facebook go about storing all the group members for each group?

Thanks,

-Nick Smile Smile Smile
vong
Many dedicated servers, properly indexed and linked tables.

This way you can store gigabytes of information in your database, but once your database gets large, you have to make sure all your links are indexed for quick searching

Storing group members for groups is relatively simple (in design) - just create a table with a many to one link to the user table and the group table. (groupid, memberid)

so you can have something like
1,1
1,2
1,3
2,2
2,4

where the id 1 links to the group frihost.com and id 2 links to frihost.net
and the various user id's can link to the user table to get vong, jim, bob, barthelemue, etc

there are ways to shape your tables to make the access of data quick and easy, if you know how its going to be used.
rvec
they'll probably have multiple servers for the database. You wouldn't want to have all the groups, users and cache data on the same server. They most likely have something better, but I've seen drupal sites work with a database spread over 2 servers.
vong
rvec wrote:
they'll probably have multiple servers for the database. You wouldn't want to have all the groups, users and cache data on the same server. They most likely have something better, but I've seen drupal sites work with a database spread over 2 servers.


oh yea, defiantly. they probably have a few servers world wide, and assign you to the closest server.

it would probably function like a distributed cloud. data spread out, but every server knows where to go to get the data it needs.
polly-gone
Well, all of Facebook's servers are based in California.

What I mean is there are 200 Million Members on Facebook. There are probably millions of groups. Groups have no member limits and you can join as many or as few groups as you want.

So say all millions groups have 100,000 members each. How is that stored?

Would they have a separate table for each group's members? One table seems excessive for storing all 100,000,000,000 membership records.

Thanks,

-Nick Smile Smile Smile
Stubru Freak
The way Facebook stores its data is really complicated. They have tried explaining it on the developer blog, here: http://blog.facebook.com/
pumpin
vong wrote:
Many dedicated servers, properly indexed and linked tables.

This way you can store gigabytes of information in your database, but once your database gets large, you have to make sure all your links are indexed for quick searching

Storing group members for groups is relatively simple (in design) - just create a table with a many to one link to the user table and the group table. (groupid, memberid)

so you can have something like
1,1
1,2
1,3
2,2
2,4

where the id 1 links to the group frihost.com and id 2 links to frihost.net
and the various user id's can link to the user table to get vong, jim, bob, barthelemue, etc

there are ways to shape your tables to make the access of data quick and easy, if you know how its going to be used.


I think He said it all, so far so good i think that's still the best form of indexing and table-use Wink
AftershockVibe
Facebook has actually presented how it's architecture works. It's not the best presentation in the world but informative if you're interested in that sort of thing. Reasonably technical.

See here:
http://linux.slashdot.org/article.pl?sid=09/04/11/1142246
sheedatali
AftershockVibe wrote:
Facebook has actually presented how it's architecture works. It's not the best presentation in the world but informative if you're interested in that sort of thing. Reasonably technical.

See here:
http://linux.slashdot.org/article.pl?sid=09/04/11/1142246


Yea the presenter was bit strange since did not seem very well prepared, but hey it is very good information. I was impressed to see what they are doing and techniques they are using to solve issues, some very clever people working in their development teams. This is the beauty of open source, you don't have to wait for Microsoft for 7 years for them to give you something, you just something off the market, change it, mold it suit your needs and then share your work back so that others can do the same again.
Related topics
Gmail?
What messenger/email service do you use
Think it's safe to trust your data on DVDs?
When Jesus was born...
Looking For A good Databasing Program
Google's Philosopy
Is this piece of PHP code efficent?
A Good Web Host
Americans want universal health care. Why can't we get it?
Storing Massive Amounts of Data
10 Reasons To Buy Windows 7 Professional (!)
Christmas Lights
Obamacare Fiasco
Reply to topic    Frihost Forum Index -> Scripting -> Php and MySQL

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.