Do you believe given Docker’s history of poor strategy (e.g., monetizing late requiring changes that negatively impact reputation such as docker desktop and hub), communication (e.g., the short notice changes a couple of months ago around free groups resulting in backlash) and in some cases destruction of successful businesses (see other comments in this thread) that this will result in a positive result for end users?
I hope you made bank while the ship continues to sink.
Mutagen's core users are primarily Docker users, and I honestly think they'll be much better served by the tighter integrations that we'll be able to offer now. If I didn't think that were the case, and I didn't think there were a net win (for Docker, Mutagen, and their users), then I wouldn't have joined.
Fair, I appreciate the answer. I guess I’ve just seen enough purchases where it’s just for the payday that I’m a bit cynical. Thanks again for taking the time to read and answer, wishing you the best
Thanks! I totally get it — I think a lot of time with acquisitions in general there's also a lot of good faith intentions that never manifest for whatever practical reason(s), so I can totally understand why the cynicism arises. I appreciate the positive wishes and I will do my best to convert those into positive outcomes!
> there's also a lot of good faith intentions that never manifest for whatever practical reason(s)
This is the risk blindness that exists in people that find success.
No one will perceive risk correctly. They will either over or under estimate. Those there's a huge amount of survivorship bias and those who underestimate risk and get lucky will be those who experience the most outsized success.
This isn't to say the cynical "I told you so"s are right, they're blind to the benefits. It's just that it's easy to say you can see the win-win. From the outside it's easy to see that the underlying benefit for Docker is reducing the power and input of other sources of influence in their technology, and that's a greater benefit to them than any benefit that this move could possibly provide to the users. But if anyone could make it work it's you.
You started an open source business making and publishing free software. How do you feel about going to work for a proprietary software vendor that ships lots of closed source software under restrictive licenses?
Was it a change in your own views toward user freedom (and an abandonment of free software ideology) or a practical matter?
I'd point out that most of the foundational components of Docker's technology stack are FOSS (e.g. [0] [1]). Mutagen also has (and had, prior to acquisition) closed-source components and components licensed under non-OSI licenses (e.g. SSPL), so evaluate my response in that context (and also please note that these are my responses, and not representative of Docker's opinions/positions). Also, as mentioned in the acquisition FAQ [2], we don't have any plans to change the open-source licensing structure, and we're still open to contributions.
It's a delicate balance to strike, and almost more delicate to discuss. At the end of the day, an open source business is still a business, and you have to make money to eat, shelter, and continue writing code. I can certainly appreciate that there are different approaches to balancing that with open-source (e.g. consulting rather than close-sourcing), but so long as you're setting natural (non-contrived) boundaries between open-source and proprietary, then I don't think you're doing anything wrong. In fact, several companies were embedding and making money from Mutagen before I was, but that never really bothered me — that was the freedom I was affording them as users.
I have nothing but the utmost respect for user freedom and I've always strived to offer that with Mutagen (e.g. allowing people to disable components that might not fit the OSI definition of FOSS). I also have nothing but the utmost respect for the authors of open-source software; Mutagen stands on the shoulders of many different dependencies and I've always strived to ensure that we're acknowledging those in a manner that is compliant with their licensing requirements (or going above and beyond that).
In the end, it's a tough but fair question. I don't think my views have changed and I don't think there's any real incompatibility. It's a balance I'll necessarily continue to assess on a daily basis, both at Docker and in any other FOSS I write.
> Also, as mentioned in the acquisition FAQ [2], we don't have any plans to change the open-source licensing structure, and we're still open to contributions.
Well, assuming you require contributors to sign a CLA, that is requiring anyone who donates software to your organization to allow you to release it in nonfree products (something Docker requires today, given that they ship proprietary software).
I'm personally really tired of this sort of free software cosplay, where companies pretend to respect user freedoms but are actually just pretending to embrace free software ideology with a license here or there but don't actually support user freedoms (as evidenced by their continued use of nonfree licenses and CLAs to enable dual licensing).
> but so long as you're setting natural (non-contrived) boundaries between open-source and proprietary, then I don't think you're doing anything wrong
I do. You can't be said to respect user freedoms if you, you know, don't respect user freedoms on a regular basis by promoting proprietary software that users can't easily modify and redistribute freely. That's actively user-hostile, just like Docker Inc has been with Docker Desktop and other products.
I'm really happy with this outcome! I've been a Docker Captain for almost 3 years now, so being an employee at Docker was a very easy switch to flip (at least mentally). As I said in another comment, this really felt like the logical step given that almost all of Mutagen's users are Docker users, and I think this is the best possible outcome for the project too!
Well, that's one of those things I won't answer, but Docker is actually really great about listing nominal salary ranges for their positions on their Docker Careers page [1] - I'd encourage anyone to check out the openings there!
It's pretty uncommon to post one's salary on a public forum like this one, let alone to answer a random outburst question in a thread about a company acquisition.
I think the norm you're suggesting goes against the original novelty of the "AMA." There is no act, and they're not the ones behaving poorly. Every AMA attracts odd questions, but it's even stranger to get mad about it.
Ah the original novelty of the AMA, back in those quaint gatherings at the town circle. Naturally the subjects were allowed to ask the minister literally anything. They all had a good laugh and much was learned. I harken back to those rosary tinted times of old. Good jolly.
If anyone has any questions about Mutagen (or integrating it into their Docker-based workflows), I'm happy to help.
Just one clarification on the article: Mutagen offers Docker Compose integration, not Composer integration (Composer is a PHP package manager). However, as mentioned, DDEV is a great option if you're looking to do containerized PHP development while using Mutagen for performance improvements.
There's no conflict resolution UI at the moment (either graphical or command-line based). Mutagen's conflict resolution is primarily performed via the specification of its synchronization mode (which can automate the resolution of most conflicts) and (in the case where conflicts can't be resolved automatically) by manually deleting the file that should "lose" the conflict (because Mutagen will synchronize a modification over a deletion).
Mutagen provides fairly detailed reporting of synchronization status, file staging progress, and change application problems via its "mutagen sync monitor" and "mutagen sync list" commands. These also support JSON output (and Go-template-style formatting), so you can pipe this information into other tooling. If you need REALLY detailed information, you can look at the debug or trace-level logs from the Mutagen daemon, but that's typically only for debugging during development of Mutagen itself.
Mutagen is somewhere in between Unison and Syncthing on the topology front, but closer to Unison. It still only supports two endpoints per synchronization session, but unlike Unison it doesn't require that one is local (i.e. you can do remote-to-remote sync using your local system as a proxy). But with both Mutagen (and Unison), you can set up a hub-and-spoke topology if you want to sync with multiple nodes.
One option I often recommend is setting up multiple synchronization sessions targeting different parts of a particular codebase with different synchronization configurations (kept mutually exclusive via ignore specifications). Quite often people want unidirectional replicas for a certain folder (e.g. a build directory) with more standard bidirectional synchronization for the rest of the code. It can be a little more complex to orchestrate, but you can use Mutagen's project functionality or a shell script to automate the setup a bit.
Thanks, very interesting read. One-sided installation and low-latency sound particularly appealing.
Two features I didn't see addressed:
- Is there a GUI of any kind? I use Syncthing's web GUI + a tray applet for monitoring syncing on desktop, and on mobile there's an app for syncing also. Makes it easy to detect if there's a problem with any of the folders being synced, and easy to add new folders this way also.
- Is there an equivalent to Syncthing's untrusted devices feature? It basically allows for client-side encrypting the files, so you can have say one remote with the encrypted files and another remote with unencrypted files.
- Mutagen performs bidirectional synchronization (though it can also operate unidirectionally); rsync is unidirectional
- Mutagen uses recursive filesystem watching to avoid full filesystem rescans (whereas rsync always does a full filesystem rescan). This allows Mutagen to provide a more "real time" sync.
- Mutagen has an active synchronization loop that doesn't require manual invocation.
- Mutagen has more idiomatic Windows support.
- Mutagen doesn't require that it be pre-installed on both endpoints.
Both use differential transfers (i.e. the "rsync algorithm") for transferring individual files.
There are other differences, of course, as well as similarities. Mutagen's design is tuned for development work, rsync's design is tuned for replication. I still use rsync for archival operations on a daily basis - it's great!
Conceptually speaking, Mutagen and Unison are very similar (and actually I mentioned Benjamin Pierce's work in another comment here asking about the sync algorithm - fantastic stuff!). I tend to avoid direct comparisons because they always come across one-sided, but some cursory differences:
- Mutagen tries to integrate recursive filesystem watching very tightly into its synchronization loop to drive synchronization and allow for near-instant filesystem rescans
- Mutagen automatically copies an "agent" binary to remote systems to support synchronization, so no remote install is required
- Mutagen uses Protocol Buffers for its data storage, so synchronization sessions created with older versions continue to work with newer versions
- Mutagen written in Go, Unison in OCaml (which allows Mutagen broader platform support "for free")
- Mutagen tries to treat Windows as a first-class citizen
Obviously the internal implementations are different, but both use differential (rsync-style) file transfers, both use the same reconciliation concepts, etc.
Mutagen has the advantage of Go, recursive filesystem watching, and modern POSIX/Windows APIs that didn't exist when Unison was originally written, though some of that functionality has been brought into Unison.
For a comparison with Syncthing (and to some extent Unison), check out this comment[0].
It should work fine. Many users use Mutagen on multi-GB codebases. If we're talking something larger (say 10s of GBs or TB-sized monorepos), then there are some tweaks you can do to make life with Mutagen a little easier. Feel free to reach out to jacob[-at-]mutagen.io if you have a specific use case, or pop over to the Mutagen Community Slack Workspace[0] to chat.
The general philosophy with Mutagen is to (a) delegate encryption to other tools and (b) use secure defaults (especially for permissions).
So, for example, Mutagen doesn't implement any encryption, instead relying on transports like OpenSSH to provide the underlying transport encryption. In the Docker case, Mutagen does rely on the user securing the Docker transport if using TCP, but works to make this clear in the docs, and Mutagen is generally using the Docker Unix Domain Socket transport anyway. When communicating with itself, Mutagen also only uses secure Unix Domain Sockets and Windows Named Pipes.
When it comes to permissions, Mutagen doesn't do a blanket transfer of file ownership and permissions. Ownership defaults to the user under which the mutagen-agent binary is operating and permissions default to 0700/0600. The only permission bits that Mutagen transfers are executability bits, and only to entities with a corresponding read bit set. The idea is that synchronizing files to a remote, multi-user system shouldn't automatically expose your files to everyone on that system. These settings can be tweaked, of course, and in certain cases (specifically the Docker Desktop extension), broader permissions are used by default to emulate the behavior of the existing virtual filesystems that Mutagen is replacing.
For files at rest on the remote, I guess I assumed files would be encryted on the remote with a local key since GP said "one can develop on untrusted remote machine" and "VSCode remote always assumes that the remote part is trusted".
On an actually untrusted remote, removing group read permissions doesn't do much to secure my code.
The only scenario where it's helpful is a system with multiple non-admin users, perhaps like a university lab computer but who's doing sensitive work on those anyway?
In many ways Mutagen and VSCode's remote extensions are the same idea, with trade-offs in terms of flexibility vs. integration.
Shared systems with multiple non-admin users was one of the original motivating use cases for tighter default permissions.
I don't think there's any scenario where one can perform truly secure development work on an untrusted system. You could certainly store encrypted code in an untrusted location, but there's not much you could do with it on that system (without a hypothetical compiler or tool that maybe supported some sort of homomorphic-encryption compilation operations?). Even decryption on-the-fly for processing by regular tools wouldn't be secure on an untrusted system. And running any code there would be equally insecure.
I'd imagine that for any seriously sensitive work, one would only want to work in highly controlled, trusted, and firewalled environments. If there's a scenario I'm missing though, definitely let me know.
Just to be clear, when I mention sensitive work, I'm not necessarily talking about national security, military, etc. kind of work. Any work for a client is sensitive enough that I wouldn't do it on any remote I (or my client assuming there is approval) don't control.
I will try Mutagen at some point. The fact that it's editor agnostic is certainly a big sell!
The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conflict resolution[0] modes that can be specified. You can find the reconciliation algorithm here[1] (and there are an exhaustive set of test cases in the corresponding _test.go file).
To avoid a large class of race conditions (at least to the extent possible allowed by POSIX and Windows), Mutagen will use `*at` style system calls for all filesystem traversal on POSIX systems, with a similar strategy on Windows.
Also, to avoid race conditions due to filesystem changes between scan time and change-application time, Mutagen will perform just-in-time checks that filesystem contents haven't changed from what was fed into the reconciliation algorithm.
Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1].
I’ve been using unison for what feels like 14 years. Once working it was great but it always took me a while to figure out the exact command line options I wanted. Beautiful tool.
