Possible way to distribute NFS?

From: mike (mike503_at_gmail.com)
Date: 11/28/05

  • Next message: kylin: "Re: to list all the devices in freebsd &definition analyse" 
    Date: Sun, 27 Nov 2005 20:05:03 -0800
To: freebsd-hackers@freebsd.org

    (Note: also sent this to -fs, but it's relatively low traffic, and a
    lot of people on -hackers have the knowledge of whether or not this is
    even an idea worth giving a shot)

    I've been looking for clustered FS or (if I have to) clustered NFS
    solutions, and there's not many out there. Unless you want to go with
    an expensive appliance. Currently you can't "roll your own" at all
    with FreeBSD.

    However, I think I've figured out a way that might work; basically
    using the memcached[1] idea of hashing the requests to identify which
    server is serving the data could alleviate any issues with NFS not
    being "globally" aware, and allow for multiple NFS servers to read
    from the same physical shared storage (note that the expectation is
    the shared storage is being accessed from multiple physical servers,
    and redundancy is done below the actual FS layer)

    If someone could write an NFS client using FUSE (perhaps?) and have it
    integrate with memcached[1], couldn't that basically allow for n+1 scaling
    out of NFS servers and storage? Basically any time the FS would need
    to issue or check a file lock, it would hit the memcache API to check
    if it's been locked or "represented" yet by a specific NFS server.

    (Note: whether or not it actually needs a "lock" or it's just saying
    "I am NFS server #1 and /foo/bar/baz.txt will be served from my
    server" is something I'd let someone smarter figure out - perhaps just
    taking ownership of files regardless of the need to lock it would
    work?)

    I'm just brain dumping here. Note that I am not that advanced of a
    programmer (I'm a PHP scripter) so filesystem and hardware I/O
    semantics are not my specialty; this is just what I've got from
    Googling until I'm blue in the face and why multi-path storage
    solutions are still not very cheap/open source (only a couple
    possibilities, each with limitations and caveats...)

    Perhaps someone could shed some light, tell me it's stupid, or tell me
    it's worthwhile?

    NFS is pretty much tried and true, adding a small middle layer in
    between to allow for distribution of NFS requests and alleviating the
    lock conflicts might be a much more usable solution? Perhaps this
    could fill the void of no GFS for FreeBSD?

    My idea for design would be N number of NFS servers behind a single
    virtual IP (or this "middleware" layer would be listening on this
    virtual IP intercepting all the NFS requests and routing them
    appropriately to the NFS servers behind them - also, for all intents
    and purposes, this middleware daemon *could* just be on the same
    physical machine as each NFS server - saving the need for separate
    middleware servers too)

    Thanks,
    mike

    [1] "memcached" is a distributed memory caching system -
    http://www.danga.com/memcached/ - it can be scaled up or out, and can
    handle failover inside of the protocol (if a memcached server
    disappears, it's info is forgotten and freed up for another active
    memcached server)
    _______________________________________________
    freebsd-hackers@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
    To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

  • Next message: kylin: "Re: to list all the devices in freebsd &definition analyse"