speedy seeding info
Posted: Fri Aug 02, 2024 11:54 pm
Some of the same things apply for downloading but i'm speaking from a seeding perspective, specifically for newly seeded torrents where bandwidth demands are the highest. The slowness of i2p is something that comes up a lot, so here's some info on making it better.
The issue is from a couple areas, the limit of 16 tunnels means you should identify your max upload speed and make additional instances of your client if you max it out.
Sometimes the limiting factor is on the peers side, or otherwise a network constraint downstream.
If you're not running near your max, and hitting the peers from multiple instances and/or routers, it's a network issue.
On i2pd slowness
i've posted before about slowness with using i2pd to torrent, it's still the same.
I've tried running snark instances with the same exact torrents using different routers (both well connected and up to date), and the max i can usually push is 500KBs on the one using i2pd, while the average is usually closer to 300. Other people have reported the same thing, if you're seeing different results do share. Meanwhile in that scenario, snark was happily running with i2p+ router with peak of 2.4MBs and average closer to 1.7.
It sounds like this is an issue in the i2cp code in i2pd. original was working on it recently though, so maybe this will improve.
Snark is inefficient sometimes
In a perfect situation a single seeder uploading to a swarm would upload a little more than the size of the file being shared. The upload ratio you'll see on snark will typically be very high mainly because as far as i know snark does not put a priority on rare pieces. If that is the case pieces will be uploaded several times from the initial seeder, as snark seems to just greedily take whatever it can get. This is very noticeable in large swarms but doesn't matter much in small swarms.
Mixing snark with other clients
I wouldn't advise seeding with a snark client along with a biglybt/qbittorrent client for a new swarm. You can use multiple biglybt's to seed a swarm and still be efficient.
Biglybt is best for big swarms
If you use super-seeding mode you can get very low upload ratios because it tracks the available pieces in the swarm and sends out the rarest pieces first. This way the swarm will reach a point when a 1gb file is distributed among everyone in the swarm, although nobody has completed yet, and the peers can trade pieces among themselves instead of overloading a single seeder. Technically at this point a lone seeder could exit the swarm and the peers can all complete the download from each other if nobody leaves. When there are 20 peers downloading, the ability of bbt focusing on getting a full "distributed" copy out quickly makes a huge difference. The comparison with snark may be several times the bandwidth bbt would have used for the same file and swarm size.
Bad peer behavior that will skew results further in favor of bbt: because snark can be "greedy" a couple peers may hog a lot of bandwidth and complete early. If they leave immediately on completion (and many will) all that "investment" is lost. Since bbt spreads the pieces around more it handles this situation better and the upload ratio increase won't be as drastic as you'd see with snark.
Super-seeding is pointless with clearnet speeds, but it makes a major difference in low bandwidth situations since it more effectively distributes bandwidth. It's also pointless if the swarm is already seeded and bigly will just ignore it.
Bbt also allows for multiple destinations per tunnel which makes a difference (where snark is one tunnel to one destination). Max speeds with biglybt can be much higher because the tunnels are more effectively utilized, i've seen max speeds with one bigly instance around 5-6MB a couple times.
In reality what that means is with biglybt and a healthy swarm a 1gb file can be distributed among the swarm in less than an hour, sometimes as quick as 30 minutes (not completed downloads, piece distribution). The usefulness of multiple destinations per tunnel and efficient distribution of pieces becomes more useful with bigger swarms and won't be as useful with small ones.
To further leverage the bandwidth of the swarm, set ignore rules under queu --> seeding. You can ignore torrents that have x amount of seeders that way, so once the torrent is seeded you stop seeding unless the seeder count dips under that number (ensure auto-start selected)
There are priority rules there for prioritizing torrents with the least amount of seeds or high peer:seed ratios. The super-seed option is in that section too.
Snark standalone: use more tunnels easily
Snark isn't the brightest client but it's low overhead, low maintenance, simple. It might seem like i'm ragging on snark but i'm not. For most situations in i2p snark will do great for seeding and is highly reliable. A major utility is that you can run many snark instances very easily and the efficiency gap between biglybt and snark can be closed that way by brute forcing with speed of upload.
Also, a big one: seeding to 1-2 peers, you're not going to see the big efficiency gains bbt will get you. The larger the swarm you're seeding to at a time, the bigger the efficiency gap will be between the clients. So if you're normally seeding to a small number of peers per torrent might as well stick with snark.
For downloading, i don't think there a huge difference with snark and bigly. For uploading, 1 snark instance != biglybt. Observationally, i think it takes 2-3 snark instances to replace a bigly bt one.
Comparison is difficult because a snark instance could be uplading faster than bigly-bt, but bigly is uploading the pieces more efficiently. By nature snark will expend more bandwidth, so i'm looking at time to first completion to make a comparison.
Luckily, all you need to do to get multiple snarks running is unzip and set a download directory (outside of the snark folder), and run. Once it checks all the torrents in the download folder, copy the snark folder several times. Then change the ports to 8001, 8002, 8003...etc in config files.
It wouldn't be super difficult to make a script to do this and i've thought about making a kind of load balancing script to open up new snarks when needed, but haven't gotten there.
All i got for now.
The issue is from a couple areas, the limit of 16 tunnels means you should identify your max upload speed and make additional instances of your client if you max it out.
Sometimes the limiting factor is on the peers side, or otherwise a network constraint downstream.
If you're not running near your max, and hitting the peers from multiple instances and/or routers, it's a network issue.
On i2pd slowness
i've posted before about slowness with using i2pd to torrent, it's still the same.
I've tried running snark instances with the same exact torrents using different routers (both well connected and up to date), and the max i can usually push is 500KBs on the one using i2pd, while the average is usually closer to 300. Other people have reported the same thing, if you're seeing different results do share. Meanwhile in that scenario, snark was happily running with i2p+ router with peak of 2.4MBs and average closer to 1.7.
It sounds like this is an issue in the i2cp code in i2pd. original was working on it recently though, so maybe this will improve.
Snark is inefficient sometimes
In a perfect situation a single seeder uploading to a swarm would upload a little more than the size of the file being shared. The upload ratio you'll see on snark will typically be very high mainly because as far as i know snark does not put a priority on rare pieces. If that is the case pieces will be uploaded several times from the initial seeder, as snark seems to just greedily take whatever it can get. This is very noticeable in large swarms but doesn't matter much in small swarms.
Mixing snark with other clients
I wouldn't advise seeding with a snark client along with a biglybt/qbittorrent client for a new swarm. You can use multiple biglybt's to seed a swarm and still be efficient.
Biglybt is best for big swarms
If you use super-seeding mode you can get very low upload ratios because it tracks the available pieces in the swarm and sends out the rarest pieces first. This way the swarm will reach a point when a 1gb file is distributed among everyone in the swarm, although nobody has completed yet, and the peers can trade pieces among themselves instead of overloading a single seeder. Technically at this point a lone seeder could exit the swarm and the peers can all complete the download from each other if nobody leaves. When there are 20 peers downloading, the ability of bbt focusing on getting a full "distributed" copy out quickly makes a huge difference. The comparison with snark may be several times the bandwidth bbt would have used for the same file and swarm size.
Bad peer behavior that will skew results further in favor of bbt: because snark can be "greedy" a couple peers may hog a lot of bandwidth and complete early. If they leave immediately on completion (and many will) all that "investment" is lost. Since bbt spreads the pieces around more it handles this situation better and the upload ratio increase won't be as drastic as you'd see with snark.
Super-seeding is pointless with clearnet speeds, but it makes a major difference in low bandwidth situations since it more effectively distributes bandwidth. It's also pointless if the swarm is already seeded and bigly will just ignore it.
Bbt also allows for multiple destinations per tunnel which makes a difference (where snark is one tunnel to one destination). Max speeds with biglybt can be much higher because the tunnels are more effectively utilized, i've seen max speeds with one bigly instance around 5-6MB a couple times.
In reality what that means is with biglybt and a healthy swarm a 1gb file can be distributed among the swarm in less than an hour, sometimes as quick as 30 minutes (not completed downloads, piece distribution). The usefulness of multiple destinations per tunnel and efficient distribution of pieces becomes more useful with bigger swarms and won't be as useful with small ones.
To further leverage the bandwidth of the swarm, set ignore rules under queu --> seeding. You can ignore torrents that have x amount of seeders that way, so once the torrent is seeded you stop seeding unless the seeder count dips under that number (ensure auto-start selected)
There are priority rules there for prioritizing torrents with the least amount of seeds or high peer:seed ratios. The super-seed option is in that section too.
Snark standalone: use more tunnels easily
Snark isn't the brightest client but it's low overhead, low maintenance, simple. It might seem like i'm ragging on snark but i'm not. For most situations in i2p snark will do great for seeding and is highly reliable. A major utility is that you can run many snark instances very easily and the efficiency gap between biglybt and snark can be closed that way by brute forcing with speed of upload.
Also, a big one: seeding to 1-2 peers, you're not going to see the big efficiency gains bbt will get you. The larger the swarm you're seeding to at a time, the bigger the efficiency gap will be between the clients. So if you're normally seeding to a small number of peers per torrent might as well stick with snark.
For downloading, i don't think there a huge difference with snark and bigly. For uploading, 1 snark instance != biglybt. Observationally, i think it takes 2-3 snark instances to replace a bigly bt one.
Comparison is difficult because a snark instance could be uplading faster than bigly-bt, but bigly is uploading the pieces more efficiently. By nature snark will expend more bandwidth, so i'm looking at time to first completion to make a comparison.
Luckily, all you need to do to get multiple snarks running is unzip and set a download directory (outside of the snark folder), and run. Once it checks all the torrents in the download folder, copy the snark folder several times. Then change the ports to 8001, 8002, 8003...etc in config files.
It wouldn't be super difficult to make a script to do this and i've thought about making a kind of load balancing script to open up new snarks when needed, but haven't gotten there.
All i got for now.