In the first part of this blog, I discussed a bit the change process and the development process we’re aiming for.
It’s interesting to realise that we’re going through the same process both in KDE and Qt. It’s also interesting that it’s happening at the same time for both — well, slightly ahead for Qt. That’s not, however, a coincidence.
I read a comment on the dot article about opposing a VCS switch every two years. Well, we’re not switching just for the fun of switching. We’re doing it because there are many compelling reasons to do so. (Also, it’s been over 3 years since the Subversion switch, about 4 since we started seriously considering it; Trolltech has been using Perforce for almost 9 years)
And it’s happening now because the DVCS tools have matured enough that we can migrate massive repositories into it. The Qt repository right now has about 170,000 commits and is over 900 MB in size (in Git, well packed). And the KDE repositories I imported had 79404, 71892, 60182 commits for kdelibs, kdebase and koffice, respectively.
And because the old processes and tools have become outdated. And, finally, because we’re people and we talk 
(that is, exchange of experience and opinions)
Switching to Git
So far I have not approached the second part of my blog title. There’s a very good reason for that: most of what we want to do, the workflow we want to introduce, does not depend on any specific tool. There are feature requirements that many tools do not fulfill, but many do fit the bill.
So why Git, in specific?
We had a lengthy process internally at Trolltech trying to decide whether we should switch to a DVCS and, if so, which tool it should be. That was a very interesting process, but one that would deserve an entire blog on the subject. We had a restricted set at the beginning, but only two serious contenders at the end: Git and Mercurial.
At one point, we came up with a list of what were the criteria we were going to use to make the decision. And then we took a look at which criteria were showstoppers. At the end of the process, there was a clear winner.
The most important criteria in that decision were:
- Quality of the conversion process (from Perforce)
- In-house knowledge of the tool
- Ability of the tool to support our proposed workflow
While the latter criterion was a tie between Mercurial and Git, the first two is where there was a clear winner.
(Disclaimer: there were other reasons, with a varying degree of importance, including some where Git lost to Mercurial)
And if you look at it, those are the exact same criteria that are of importance to KDE now. Sure, there are other tools that also do the job, but who’s going to do the conversion? Who’s going to support and help the developers who aren’t familiar with the tool?
For close to a year now, we’ve had the kde-scm-interest mailing list, whose mandate is to propose a plan to convert from Subversion to another VCS (or Source Control Management - SCM) and how the layout and workflow would be with this tool. Which VCS, it’s not specified. Any is welcome, barring CVS (we’re not going back), Subversion (it’s not conversion if we stay where we are) and solutions that aren’t Free and Open Source.
If you inspect the archives, you’ll see that there’s only one contender. So far, anyways. Others are more than welcome to join and repeat the process for other tools. But mind you: you’re a year behind schedule.
This weekend, I finished converting the KDE Subversion repository into Git. The process created no less than 493 different repositories: I decided that each application in extragear, kdereview, kdesupport or playground should get its own repository. There will be more when I run it again.
The process isn’t correct yet. There are definite import errors found: for instance, Thomas Zander inspected the the KOffice repository and realised that several branches were missing. What’s more, I still have a few ideas to do copy-with-history, which means preserving correctly the moves in SVN. Many of our projects originated in the old kdenonbeta, which I didn’t import.
I am forced to realise that Git isn’t completely ready for us yet. I’ve been using Git for well over a year now and I can see it improving build after build. 1.6.0 is very nice, but not there yet. So I ask the Git developer community: how do we handle 493 separate but related repositories? Mind you, we want to maintain the ability to build them all with very simple commands.
So, while Git is by no means decided for KDE — well advanced, I’d say — it is decided for Qt. We have kickstarted the process to completely switch to Git. Very soon now, we’ll be in the final leg of that process and will hopefully have some news on it. Our goal is to stop using Perforce for Qt 4.5 by October 1st.
19 Responses to “Workflow and switching to Git, part 2: The tools”
We’re facing the same problem at work (despite we’re having much less modules than KDE, I’d say we have 10 of them)
The solution I could think about was having a repository with many submodules in it (so you can init/clone/build it with a single command, while letting people use alternative sources for some modules)
but I don’t know how it would scale with nearly 500 modules…
The git community is great. When we (Source Mage) were switching to git from perforce a few years back, someone volunteered and wrote the p4 importer for us - it didn’t exist before. We didn’t have that many submodules, so we just use a repository for each. I know that can’t work here. Maybe have a top repository for each kde k^Wcomponent and then “submodules” in them, which would split the count a bit.
> Any is welcome, barring […] Subversion (it’s not conversion if we stay where we are) […]
Now that’s really broken logic. You make the assumption that a conversion is needed in the first place and use that assumption to completely ignore the option of not converting. The fallacy there is that the assumption you made is what people are arguing against, so you can’t use that assumption to simply discount their arguments.
You’re also completely ignoring factors where SVN is clearly superior to Git, such as the availability of complete, easy to use GUIs such as kdesvn. It is possible to use only the kdesvn GUI to interact with KDE SVN (and this is essentially what I do, I only use the svn command line when I want to pipe svn diff straight to a file). Git, on the other hand, requires you to use the command line pretty frequently, and the commands are also more complex (and thus harder to learn) than in SVN.
> the assumption that a conversion is needed in the first place
a conversion is desired so that we can support a workflow that better supports our efforts. the conversion isn’t for conversion’s sake, however.
> requires you to use the command line pretty frequently,
there are some git frontends available now, including ones done with Qt even. qgit 2.2 seems to be alright, but i can also see room for improvement here as well.
speaking of tools, i’ve been rather impressed with the various tools that have grown up around git, such as gitossis and gitorious.
> and the commands are also more complex (and thus harder to learn) than in SVN.
as someone who really didn’t feel comfortable with dvcs at first, and who found git really annoying to use when i tried it a while back, it’s actually gotten to the point where it’s really about the same complexity for the common tasks. things like bisecting are something git makes easy and svn can’t even touch, really; while merging is also a lot less of a nightmare.
git can be more complex in usage, ime, but that’s primarily when you start doing things with it that aren’t really feasible with svn in the first place, so it’s not useful as a comparison point.
for the usual diffs, commits, histories and commit reviews it’s all very simple with git. it’s as simple as with svn, only a lot faster, offline and with more possibilities.
i was dead set against dvcs, and esp git, for kde back when it was decided we needed to move on from cvs .. but times, and the software, has changed for the better in the last 4 years.
Kevin:
The purpose of the kde-scm-interest mailing list is to discuss what would happen if we converted and what it would take to do it. There’s no point in discussing a conversion to SVN.
However, kde-scm-interest is not about making the decision to change. No, that has to happen at a wider audience. So it’s a two-step program:
- kde-scm-interest comes up with a plan, a tool, a layout
- Wider audience approves of it and we begin implementing it
Regarding building everything easily..
Have you looked a jhbuild? It is really sexy. Gnome use it (officially) and so does xorg (somewhat unofficially).
I’ve only used it for xorg development, but basically xorg is split up into about 250 separate git trees. To checkout and install the xserver along the 60 or so dependancies (the other 190 modules aren’t required for a bare minimum xserver):
* Install the jhbuild executable
* git checkout url_for_a_modules_file_describing_dependencies
* git checkout url_for_jhbuildrc_files
* run: jhbuild build xserver
And that’s it! jhbuild resolves the dependencies and checks out all the modules that I require and builds them in order. If there are errors it brings up a text menu to help resolve problems and so on.
I think that this is the best way to go - it works pretty well for xorg development so I see no reason that it shouldn’t work for us.
While I can deduct the business case for switching from the different workflows, it would be great if you could publicly explain it in plain terms a businessperson can appreciate, as well as follow through with an article afterwards evaluating which objectives were met as expected, and which weren’t.
> And if you look at it, those are the exact same criteria that are of importance to KDE now
Sorry.. but there’s something i really don’t understand here. The only point where i can understand git is a winner it the “migrate from perforce” (i don’t know though, but I’ll trust you on this one). And this is definitely not a point for KDE, is it ?
I have no idea about the second point (in-house knowledge), all I can say is that I’ve been through the same evaluation process one year ago, knowing none of them, and mercurial was a clear winner for me. I needed one full day to be able to do what i want with git, using too much pages/tutorials on the web, while using only ‘hg help’ I manage to do what i want with mercurial in less than 15 minutes. This let me think that even if (say) 50% of kde users know git and only 10% of them know mercurial, it still would need, as a whole, less work to use mercurial.
Someone raised the point about GUI. I don’t know about git, but mercurial has a very nice tool (even if written in tcl/tk, which i don’t like), that I like to use a lot more than kdesvn, for example.
Finally, considering the point of switching from svn (which did a very good job for many years) to a DVCS, I absolutely support it. I’ve done the switch (using mercurial) for both professional and free software projects, and it’s really worth it. Although it is indeed a (very) little bit more complicated at the beginning to use.
Thomas, you apparently missed the point that the people interested in converting KDE into a DVCS (eg Thiago) are also Git people. Not git partisans, just people using git.
From what I know, Mercurial sounds like a better choice too. But I don’t know, since Mercurial doesn’t have a git-svn equivalent so its impossible to play with really.
I prefer Mercurial over git, too, because it has good documentation, sane (and often even helpful) error messages and in general seems to have usability as the main goal. Its speed is in the same league as git.
Ian, have you tried hg convert? It has built-in svn support since a couple of versions. It does not support committing back to svn, though. One hint if you are going to test it: it is confused by the round-robin DNS for anonsvn.kde.org. If you give it a svn URL with an IP it will work fine.
Also have a look at mercurial.intuxication.org/hg/kde(libs/pimlibs/base/pim) for mirrors of some important KDE components created using hg convert.
Thomas: Ian has the answer there.
The three reasons I posted for our internal conversion for Qt apply to KDE the following manner:
- The quality of the conversion (from Subversion)
- Knowledge of the tool among the active KDE developers
- Ability of the tool to support the proposed workflow
Like in the internal case, the third point is a tie between Git and Mercurial. But in the first two, Git is a clear winner. Just try to find out anyone who has tried converting the KDE Subversion repository to Mercurial. Or try to count how many of the KDE developers are now familiar with Mercurial as opposed to those familiar with Git.
Also note that familiarity is necessary to a very deep level, not just basic usage of the tool. We’ll need to do serious hacking with the tool to maintain our enormous repositories. We need a tool we understand intimately. I understand Git, who in our team understands Mercurial?
re separate-but-related repos - did you check out git’s submodule support? It’s basic, but might be adequate for your needs. git submodule –help
Mind you, the x.org project just uses a big bunch of separate git repos and shell scripts to grab them and build them, though.
http://www.x.org/wiki/ModularDevelopersGuide#head-a085b56a4b98f5418566ff5f575ca47805bd54c6
http://www.x.org/wiki/ModularDevelopersGuide#head-c35264aacf77e77caeca305a1a9a3da3be6e4fa5
(just in case, bit more clarity - I mean you’d still have your 493 repos for each project, but also overrarching superrepo(s) with a bunch of submodules referencing the project repos that it versioned so that one can do also-versioned bundles-of-projects for releases)
http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
glad to see a move to git!
> Also note that familiarity is necessary to a very deep level, not just basic usage of the tool. We’ll need to do serious hacking with the tool to maintain our enormous repositories. We need a tool we understand intimately. I understand Git, who in our team understands Mercurial?
You are right, developers must know how to do their work with the selected SCM, but this is a 2 ways road.
1.- Users coming from SVN (this covers 100% of kde-developers): they can pick Mercurial in a day, Mercurial’s ui is saner than Git and very similar to SVN. The can be working as they were only knowing about 2 new actions: push, pull.
2.- Users coming from Git: They can pick Mercurial in about an hour. To put it in poetic terms, they are descending one step in the stair of Difficult_to_grasp-ness
For some corner cases Git is bit more powerful but it comes with the cost is a great complexity. All its users sure can remember the WTF when first seeing things like “commit -a” (I have crossed that river…), now think about a kde n00b developer who only knows about SVN. Ok, we’re all Geek alpha males and can’t grasp concepts like the index and what-not, but imposing these kind of difficulties from the beginning maybe is going to discourage someone.
Of course, this kind of discussion is akin to a Vim-Emacs war (Vim won that FYI 
but I just try to point some concerns when seeing this post.
As the “switch every two years” comment, let me apologize for being off a year. My main problem is how this changes the developer’s workflow. I can certainly understand git’s desirability from a project management perspective, but how will it affect the day to day workflow of non-core developers? The classic centralized workflow is simple: [update, commit]*. I’m hoping I can keep that same workflow in a decentralized environment. I would also like to see some thorough documentation when this gets rolled out.
@JWilk: switching to Mercurial is and will remain a closed road until someone volunteers to do the conversion work.
No one has, so Git is the only contender for the moment. Canonical wants to sponsor a conversion to Bazaar, but it hasn’t happened yet.
While git is now working reliably on unix, I must say it’s not really the case on Windows. It boils down to the cygwin based version which is slow and forces the use of cygwin, or the MSYS based version, which is still unstable and also slow. In addition there’s no graphical frontend on windows, at best an Eclipse plugin which is still incomplete.
I know Windows is not a main target for KDE at the moment (although it’s about to change), but I’m more wondering about Qt. I guess that at least some trolls are developing on windows, if at least for testing, so I’m wondering what’s gonna happen there. Well, it _does_ work on windows too, but just not as reliably and comfortably as svn.
Anyway if this means it’ll come with some work for a better git support on Windows then I’m all for it.
Git works on Windows very well, thank you (so say our Windows developers).