I forked the “real” i2p repository in addition the I2P web-site from before. As this failed on my first attempt due to the connection, I had a look at the alternative way of using git-bundles …

i2p.i2p

I wanted to clone the repository containing the code of I2P as well. This is named i2p.i2p on git.idk.i2p and it worked on my second attempt:

lbt@go:~/synced-git.idk.i2p$ time git clone git@git.idk.i2p:lbt/i2p.i2p.git
Cloning into 'i2p.i2p'...
remote: Enumerating objects: 232741, done.
remote: Counting objects: 100% (102/102), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 232741 (delta 37), reused 102 (delta 37), pack-reused 232639
Receiving objects: 100% (232741/232741), 428.19 MiB | 150.00 KiB/s, done.
Resolving deltas: 100% (156119/156119), done.

real	48m41.029s
user	0m19.951s
sys	0m4.807s

On second attempt means that my first attempt (same command) failed as the connection wasn’t stable enough. git just reported an unexpected end-of-file and started from scratch when invoking it again. The 48 minutes this took give rather plenty of time for something like this to happen.

Anyways I did add “upstream” as a remote to my new repository:

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ git remote add upstream git@git.idk.i2p:i2p-hackers/i2p.i2p.git

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ git remote -v
origin	git@git.idk.i2p:lbt/i2p.i2p.git (fetch)
origin	git@git.idk.i2p:lbt/i2p.i2p.git (push)
upstream	git@git.idk.i2p:i2p-hackers/i2p.i2p.git (fetch)
upstream	git@git.idk.i2p:i2p-hackers/i2p.i2p.git (push)

git-bundle

While it’s probably good news, that I was able to do this “just like that” in the end, I started to look at the alternative way of getting this “first checkout” done. The preffered way seems to be git-bundles, a way to bundle a repository into a single file.

From the man page of git-bundle:

Create, unpack, and manipulate “bundle” files. Bundles are used for the “offline” transfer of Git objects without an active “server” sitting on the other side of the network connection.

As the repositories aren’t exactly small:

lbt@go:~/synced-git.idk.i2p$ du -hs *
543M	i2p.i2p
327M	i2p.www

this does sound sensible actually. For a moment I thought about how integrity/authenticity is guaranteed in this. But I didn’t further this much, as the scenario here is to link it back to the original and any differences would have to be noticed by git when trying to “put something back in” basically. So probably not a concern unless you were to use the downloaded repository without ever checking back to the original.

So I searched (using postman) for “git bundle”. And there was [i2p.i2p] [git bundle] Fully-Unshallowed Bundle, March 18, 2020.torrent to be found with a size of 366.21 MB. 2020 starts to sound kinda old in 2023 to me, so I started wondering if a newer one could be helpful. While “not sure”, I just went on diving further into this, as the official I2P documentation of this also explains how to generate such a bundle:

Once you have that, simply run the corresponding ant target: ant bundle

ant bundle

Of course, java devs use java make (in case you aren’t aware, ant is basically the java version to spell make - with build.xml instead of Makefile etc.). Ok, well, then I’ll need to start to get java stuff here:

root@go:~# apt install ant
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  ant-optional ca-certificates-java default-jre-headless java-common openjdk-11-jre-headless
Suggested packages:
  ant-doc default-jdk | java-compiler | java-sdk antlr javacc junit junit4 jython
  libactivation-java libbcel-java libbsf-java libcommons-logging-java libcommons-net-java
  libmail-java libjaxp1.3-java libjdepend-java libjsch-java liblog4j1.2-java liboro-java
  libregexp-java libxalan2-java libxml-commons-resolver1.1-java libxz-java default-jre
  fonts-wqy-microhei | fonts-wqy-zenhei fonts-indic
The following NEW packages will be installed:
  ant ant-optional ca-certificates-java default-jre-headless java-common
  openjdk-11-jre-headless
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 39.9 MB of archives.
After this operation, 175 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Let’s go:

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ ant bundle
Buildfile: /home/lbt/synced-git.idk.i2p/i2p.i2p/build.xml

BUILD FAILED
Target "bundle" does not exist in the project "i2p". 

Total time: 0 seconds

Hmpf, that was a bit too fast. Checking into build.xml there is no “bundle” target. That program was right to say so ;)

ant git-bundle

There is, however, a target “git-bundle”. So let’s try that, it sounds so right, right?

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ ant git-bundle
Buildfile: /home/lbt/synced-git.idk.i2p/i2p.i2p/build.xml

deletepkg-temp:

pkgclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

clean:

distclean:

checkForGit:

getGitRev:

getReleaseNumber:
     [echo] Release number: 2.0.0
     [echo] API version:    0.9.56

getBuildNumber:
     [echo] Build number is 16-rc

setBuildTimestamp:

disableManifestClasspath:

buildProperties:
     [echo] JDK: Debian 11.0.16 (OpenJDK Runtime Environment 11.0.16+8-post-Debian-1deb11u1)
     [echo] Building version 2.0.0-16-rc (revision 1e34738fca856fed7da17459c33af8c835eda85c)

buildCore:

depend:

compile:
    [mkdir] Created dir: /home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/build
    [mkdir] Created dir: /home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/build/obj
    [javac] Compiling 414 source files to /home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/build/obj
    [javac] Ignoring source, target and bootclasspath as release has been set
    [javac] /home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/src/net/i2p/I2PAppContext.java:35: warning: [deprecation] SimpleScheduler in net.i2p.util has been deprecated
    [javac] import net.i2p.util.SimpleScheduler;
    [javac]                    ^
    [javac] 1 warning

bundle:
    [mkdir] Created dir: /home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/build/messages-src
     [exec] Generating net.i2p.util.messages_ar ResourceBundle...
     [exec] ERROR - msgfmt failed on ../locale/messages_ar.po, not updating translations
     [exec] ./bundle-messages.sh: 104: msgfmt: not found
     [exec] ./bundle-messages.sh: 109: msgfmt: not found

BUILD FAILED
/home/lbt/synced-git.idk.i2p/i2p.i2p/build.xml:421: The following error occurred while executing this line:
/home/lbt/synced-git.idk.i2p/i2p.i2p/core/java/build.xml:137: exec returned: 1

Total time: 2 seconds

Still fast, a lot of cleaning from the looks, but still unsuccessfull. What’s going on?

Searching, finding, and looking into the script ./core/java/bundle-messages.sh (I found 10 scripts with that name :o) shows

	# Requires installed programs xgettext, msgfmt, msgmerge, and find.

Right …

root@go:~# apt install xgettext msgfmt msgmerge find
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package xgettext
E: Unable to locate package msgfmt
E: Unable to locate package msgmerge
E: Unable to locate package find

Was (maybe) worth a try, though ;)

Update: Build Environment

The page for new-developers clearly points to the GNU gettext package in addition to … java, of course. If I’m not mistaken that should be as easy as:

root@go:~# apt install default-jdk
[...]
root@go:~# apt install gettext
[...]

They do strongly recommend the SUN JDK on that page, though. Wonder how current that is … Anyways, let’s re-try with those installed now …

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ ant git-bundle
[...]
BUILD FAILED
/home/lbt/synced-git.idk.i2p/i2p.i2p/build.xml:642: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: java.io.IOException: Cannot run program "/home/lbt/synced-git.idk.i2p/i2p.i2p/installer/lib/launch4j/bin/windres": error=2, No such file or directory
[...]

Still no good. As I should satisfy what the new-developer page suggested as a complete build environment, I felt a bit helpless and started to ask at this point. If I understood correctly the mktorrent part relies on some libraries that need to be built/pointed at first, currently. Sounded like it’s not supposed to be like that. So let’s delay this step for a bit again …

git-bundle and mktorrent

However I started wondering “why all this, isn’t this basically a git feature?”. So I looked into that build.xml for what it is doing:

    <target name="git-bundle" depends="pkg">
        <exec executable="git" failonerror="true">
            <arg value="bundle" />
            <arg value="create" />
            <arg value="i2p.i2p.${full.version}.bundle" />
            <arg value="--all" />
            <arg value="--branches" />
            <arg value="--tags" />
        </exec>
        <mktorrent file="i2p.i2p.${full.version}.bundle" />
    </target>

So this target does actually call the git command, but then also makes a torrent file for the created git-bundle. That mktorrent is defined in the same file as:

    <macrodef name="mktorrent">
        <attribute name="file" />
        <sequential>
            <delete file="@{file}.torrent" quiet="true"/>
            <java classname="org.klomp.snark.Storage" fork="true" failonerror="true">
                <classpath>
                    <pathelement location="build/i2p.jar" />
                    <pathelement location="build/i2psnark.jar" />
                </classpath>
                <arg value="-a" />
                <arg value="http://tracker2.postman.i2p/announce.php" />
                <arg value="-c" />
                <arg value="${build.built-by}" />
                <arg value="-m" />
                <arg value="Official torrent for version ${full.version}" />
                <arg value="@{file}" />
            </java>
        </sequential>
    </macrodef>

And somewhere within that must be the call to that script bundle-messages.sh then.

So let me try the git part on it’s own by putting parameters in as the build.xml does:

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ git bundle create i2p.i2p.2023-01-09.git.bundle --all --branches --tags
Enumerating objects: 232741, done.
Counting objects: 100% (232741/232741), done.
Compressing objects: 100% (44664/44664), done.
Total 232741 (delta 156119), reused 232741 (delta 156119), pack-reused 0

Ok, so that actually works fine, it’s just creation of the torrent file for which I’m missing the prequisites. From a look at the parameters this could probably be done with the Debian package mktorrent as well?

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ mktorrent --announce="http://tracker2.postman.i2p/announce.php" --name="i2p.i2p.2023-01-09.git.bundle" --comment="git-bundle to use with git clone" i2p.i2p.2023-01-09.git.bundle 
mktorrent 1.1 (c) 2007, 2009 Emil Renner Berthing

Hashed 1713 of 1713 pieces.
Writing metainfo file... done.

Well, kind of. The java-generation seems to put a file as a description in there etc. And anyways should find out how to do it the i2p way. And more importantly, still need to check if it works ;)

Testing - git-fetch vs. git-pull

lbt@go:~/synced-git.idk.i2p/tmp$ git clone ../i2p.i2p/i2p.i2p.2023-01-09.git.bundle i2p.i2p
Cloning into 'i2p.i2p'...
Receiving objects: 100% (232741/232741), 428.19 MiB | 89.67 MiB/s, done.
Resolving deltas: 100% (156119/156119), done.
fatal: multiple updates for ref 'refs/remotes/origin/master' not allowed

Uha? Let’s check the bundle first:

lbt@go:~/synced-git.idk.i2p/i2p.i2p$ git bundle verify i2p.i2p.2023-01-09.git.bundle | grep master
1e34738fca856fed7da17459c33af8c835eda85c refs/heads/master
1e34738fca856fed7da17459c33af8c835eda85c refs/remotes/origin/master
1e34738fca856fed7da17459c33af8c835eda85c refs/heads/master
i2p.i2p.2023-01-09.git.bundle is okay

So there are duplicates, but actually not for the one mentioned in the error message? Searching the clearnet I found an old (2018) thread [1] stating:

Just like a remote repository that reports the same ref more than once in its initial advertisement (i.e. “git ls-remote $remote” gives duplicate entries), a bundle file that records the same ref more than once is a bug, I would think. A “git bundle create” command that creates such a bundle file shouldn’t. It is not very useful to diagnose it as an error; it probably makes more sense to dedup the refs instead when writing the bundle file. Of course, we should abort with an error if the code ever tries to store the same ref twice with different object name (i.e. attempt to dedup, in vain). Also, “git clone” from such a bundle file (or for that matter, a remote repository that advertises the same ref twice) probably should do a similar deduping, with a warning message.

All sounds like nothing is doing what it should, lol. But well …

The I2P documentation describes another way if git-clone errors out:

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git init
Initialized empty Git repository in /home/lbt/synced-git.idk.i2p/tmp/i2p.i2p/.git/

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git fetch ../../i2p.i2p/i2p.i2p.2023-01-09.git.bundle
Receiving objects: 100% (232741/232741), 428.19 MiB | 90.05 MiB/s, done.
Resolving deltas: 100% (156119/156119), done.
From ../../i2p.i2p/i2p.i2p.2023-01-09.git.bundle
 * branch            HEAD       -> FETCH_HEAD

But that left me with an empty directory? What seemed to work fine was to just pull from the file instead:

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git pull ../../i2p.i2p/i2p.i2p.2023-01-09.git.bundle
Receiving objects: 100% (232741/232741), 428.19 MiB | 90.44 MiB/s, done.
Resolving deltas: 100% (156119/156119), done.
From ../../i2p.i2p/i2p.i2p.2023-01-09.git.bundle
 * branch            HEAD       -> FETCH_HEAD

Looks exactly the same, but did give me files. Not quite sure about the difference here then …

Now as this was only the means to get the repository for the first time, I still need to “connect” it to the real one. The set-url variations suggested in the documentation didn’t work for me, but the following did:

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git remote add origin git@git.idk.i2p:lbt/i2p.i2p

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git pull origin master
From git.idk.i2p:lbt/i2p.i2p
 * branch                master     -> FETCH_HEAD
Already up to date.

lbt@go:~/synced-git.idk.i2p/tmp/i2p.i2p$ git status
On branch master
nothing to commit, working tree clean

So looking good. Once one has the details figured out it does indeed work fine this way it seems.

i2p.www

The way without java should work for any repository, though. As i2p.www isn’t much smaller, maybe it would make sense to provide such a download?

lbt@go:~/synced-git.idk.i2p/i2p.www$ git bundle create i2p.www.bundle --all --branches --tags
Enumerating objects: 53316, done.
Counting objects: 100% (53316/53316), done.
Compressing objects: 100% (10985/10985), done.
Total 53316 (delta 38740), reused 53277 (delta 38715), pack-reused 0

lbt@go:~/synced-git.idk.i2p/i2p.www$ mktorrent --announce="http://tracker2.postman.i2p/announce.php" --name="i2p.www.bundle as per 2023-01-09" --comment="git-bundle to use with git clone" i2p.www.bundle 
mktorrent 1.1 (c) 2007, 2009 Emil Renner Berthing

Hashed 986 of 986 pieces.
Writing metainfo file... done.

Hm, as security is a no-concern due to “joining” this to the official repos later, maybe it’s worth setting this up as a service? Like automatically providing monthly updated bundles for the two repositories?

TODO:

  • Provide feedback concerning the steps that (didn’t) work, possibly update docu if this “just has changed” - I’m unclear about the details here, though. It could be something about having/not having an entry in the .gitconf and alike.
  • Check if this reset before creating the bundle helps as suggested by Swipe5337 
    19:42:12 < Swipe5537> lbt: I created a git bundle, can you see if `git clone` 
                    works directly on it? 
http://tracker2.postman.i2p/index.php?view=TorrentDetail&id=68273
19:52:39 < Swipe5537> `git checkout master` `git reset --hard i2p-2.1.0` `git 
                    bundle create ../i2p.i2p_2.1.0.bundle master`

Clearnet Links

[1] https://www.spinics.net/lists/git/msg326380.html