twitter-archive/flockdb: A distributed, fault-tolerant ... - GitHub

FlockDB is a distributed graph database for storing adjancency lists, with goals of supporting: ... Non-goals include: ... FlockDB is much simpler than other graph ... Skiptocontent {{message}} Thisrepositoryhasbeenarchivedbytheowner.Itisnowread-only. twitter-archive / flockdb Publicarchive Notifications Fork 264 Star 3.3k Adistributed,fault-tolerantgraphdatabase License Viewlicense 3.3k stars 264 forks Star Notifications Code Pullrequests 0 Actions Projects 0 Security Insights More Code Pullrequests Actions Projects Security Insights Thiscommitdoesnotbelongtoanybranchonthisrepository,andmaybelongtoaforkoutsideoftherepository. master Branches Tags Couldnotloadbranches Nothingtoshow {{refName}} default Couldnotloadtags Nothingtoshow {{refName}} default 71 branches 21 tags Code Latestcommit grimreaper Formallyarchivetwitter/flockdb … d0e72e8 Mar16,2017 Formallyarchivetwitter/flockdb d0e72e8 Gitstats 740 commits Files Permalink Failedtoloadlatestcommitinformation. Type Name Latestcommitmessage Committime config Fixconfig Mar23,2012 doc Demousesgizzmotopologyinsteadofforwardings.Deprecationwarning… Sep28,2011 project org=com.twitter,name=flockdb,version=1.8.16-SNAPSHOT Apr2,2012 src changeshard_idtovalidformat.gizzmo-0.15.1expandedshard_id Apr28,2012 .ensime ensimeprojectfile Aug3,2011 .gitignore upgradetogizzard3.0.0-beta6,plugingaugesforthrottledconnecti… Aug26,2011 LICENSE ModifyLICENSE Aug15,2011 README.markdown Formallyarchivetwitter/flockdb Mar16,2017 TODO initialsourceimport Apr12,2010 Viewcode STATUS FlockDB Itdoeswhat? Building Running Community Contributors README.markdown STATUS TwitterisnolongermaintainingthisprojectorrespondingtoissuesorPRs. FlockDB FlockDBisadistributedgraphdatabaseforstoringadjancencylists,with goalsofsupporting: ahighrateofadd/update/removeoperations potientiallycomplexsetarithmeticqueries pagingthroughqueryresultsetscontainingmillionsofentries abilityto"archive"andlaterrestorearchivededges horizontalscalingincludingreplication onlinedatamigration Non-goalsinclude: multi-hopqueries(orgraph-walkingqueries) automaticshardmigrations FlockDBismuchsimplerthanothergraphdatabasessuchasneo4jbecauseit triestosolvefewerproblems.Itscaleshorizontallyandisdesignedfor on-line,low-latency,highthroughputenvironmentssuchasweb-sites. TwitterusesFlockDBtostoresocialgraphs(whofollowswhom,whoblocks whom)andsecondaryindices.AsofApril2010,theTwitterFlockDBcluster stores13+billionedgesandsustainspeaktrafficof20kwrites/secondand 100kreads/second. Itdoeswhat? If,forexample,you'restoringasocialgraph(userAfollowsuserB),and it'snotnecessarilysymmetrical(AcanfollowBwithoutBfollowingA),then FlockDBcanstorethatrelationshipasanedge:nodeApointstonodeB.It storesthisedgewithasortposition,andinbothdirections,sothatitcan answerthequestion"WhofollowsA?"aswellas"WhomisAfollowing?" Thisiscalledadirectedgraph.(Technically,FlockDBstorestheadjacency listsofadirectedgraph.)Eachedgehasa64-bitsourceID,a64-bit destinationID,astate(normal,removed,archived),anda32-bitposition usedforsorting.Theedgesarestoredinbothaforwardandbackward direction,meaningthatanedgecanbequeriedbasedoneitherthesourceor destinationID. Forexample,ifnode134pointstonode90,anditssortpositionis5,then therearetworowswrittenintothebackingstore: forward:134->90atposition5 backward:90