Sensei

Sensei is a distributed database that is designed to handle the following type of query:

SELECT f1,f2...fn FROM members 
WHERE c1 AND c2 AND c3..
MATCH (fulltext query, e.g. "java engineer")
GROUP BY fx,fy,fz...
ORDER BY fa,fb...
LIMIT offset,count
							

Architecture:

Some Features

• Full-text search
• Faceted Search
• Dynamic Sorting
• Streaming index
• Fast realtime update
• Distributed - sharded
• Dynamic cluster support
• ...

Design considerations

• data:
  • Fault tolerance - when one replication is down, data is still accessible
  • Durability - N copies of data is stored
  • Through-put - Parallelizable request-handling on different nodes/data replicas, designed to handle internet traffic
  • Consistency - Eventally consistent
  • Data recovery - each shared/replica is noted with a watermark for data recovery
  • Large dataset - designed to handle 100s millions - billions of rows
• horizontally scalable:
  • Data is partitioned - so work-load is also distributed
  • Elasticity - Nodes can be added to accomodate data growth
  • Online expansion - Cluster can grow while handling online requests
  • Online cluster management - Cluster topology can change while handling online requests
  • Low operational/maintenance costs - Push it, leave it and forget it.
• performance:
  • low indexing latency - real-time update
  • low search latency - millisecond query response time
  • low volatility - low variance in both indexing and search latency
• customizability:
  • plug-in framework - custom query handling logic
  • routing factory - custom routing logic, default: round-robin
  • index sharding strategy - different sharding strategy for different applications, e.g. time, mod etc.

  Comparing to a traditional RDBMS

  RDBMS:

  • scales vertically
  • strong ACID guarantee
  • relational support
  • performance cost with full-text integration
  • high query latency with large dataset, esp. Group By
  • indexes needs to be built for all sort possibilities offline

  Sensei:

  • scales horizontally
  • relaxed Consistency with high durability guarantees
  • data is streamed in, so Atomicity and Isolation is to be handled by the data producer
  • full-text support
  • low query latency with arbitrarily large dataset
  • dynamic sorting, index is already built for all sortable fields and their combinations

        Learn about Sensei

Technology stack:

Sensei is built on top of an abundance of awesome-ness from the open-source community:

• Bobo: a faceted search library
• Zoie: a realtime stream search library
• Lucene: full-text search engine
• ZooKeeper: distribute resource synchronization system
• Netty: NIO client server framework

Getting Started

Things to help you get started:

• sample application
• getting started

Gateways

Gateways specify where Sensei should be fetching data from.