pyBM eats too many CPU cycles

BM-2cU6H4GiPoJKh7E8w8gCxE9WUujuyyPd5w
Sep 22 12:54

CPU load gets to high after 12 hrs running pyBM with green light. erasing knownNodes will give you 1% CPU load for hours, with just a few connections. CPU load should not get that high ! How to throttle it for good ? limit green connection number too ?

[chan] bitmessage
Sep 22 13:47

When running green nodes be aware that there a number of negligent/malicious nodes that connect to you exhibiting some of the following behaviors: - continuously flooding you with fake nodes - advertising objects with expire times outside the acceptable range - repeatedly connect, download all objects, disconnect These can have negative impacts on network traffic, CPU load, and memory use.

[chan] bitmessage
Sep 22 13:50

I want to be able to limit connect to TLS 0.6.2 nodes and ignore 0.4.4 bullshit nodes. also I want an 8 green nodes limit. cannot be hard to do, just change a number, right ?

[chan] bitmessage
Sep 22 14:22

Actually the few 0.4.4 nodes I encountered were quite polite compared to a stock 0.6.2 node. The worst offenders are alternative implementations (wire/bmd) but attacks do come from TLS 0.6.2 as well - making decisions based on user agent string is just as useless as in http. Set maxtotalconnections to 8 more than maxoutboundconnections (note the minimum maxoutboundconnections is 8). For your other criteria you will need to modify the code where the version message is processed to close the connection when the criteria isn't met.

[chan] Crypto-Anarchist Federation
Sep 22 14:28

thanks - I will report back.

[chan] bitmessage
Sep 22 14:39

looks good so far , changed 200 to 17 like 5 times or so

[chan] bitmessage
Sep 22 14:44

well - not much if any change. 17 threads, 17% total load rescheduling renice is futile too lag is a bit too high I think, tho it still kinda is workable, but significantly slower.

[chan] bitmessage
Sep 22 15:01

Did you restart? Maybe restart is still required for maxtotalconnections to take affect.

[chan] bitmessage
Sep 22 15:06

oh my - now after 20 min. load is very low 1% , no lag, only 5 green conn. if it stays that way, its fine. then it was an effective change ... :-) thanx

[chan] bitmessage
Sep 22 15:09

and yes, I did restart & killed knownnodes.

BM-2cU6H4GiPoJKh7E8w8gCxE9WUujuyyPd5w
Sep 22 15:31

seems like abusive green nodes ate my cycles. went back to yellow light deliberately, with less than 1% load. screw it.

[chan] broadcast
Sep 22 16:01

8444 tcp & udp rule in GUFW firewall gave me green light and much lower CPU load. :-)

[chan] bitmessage
Sep 22 16:29

€urAsia has all it needs to take over planet Earth.

[chan] bitmessage
Sep 22 16:30

And what does that have to do with this thread?

[chan] bitmessage
Sep 22 20:33

but not for long. we need a better green connection throttle. this sucks.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Sep 23 07:41

It would be helpful to know which thread causes the problems. Install the prctl python module and do top -Hp `cat ~/.config/PyBitmessage/singleton.lock` Peter Surda Bitmessage core developer

[chan] bitmessage
Sep 23 08:50

If they have unique IPs, you can ban them. If they use TOR, not without blocking every good TOR user as well.

[chan] AMERICHAN
Sep 23 09:33

OK hang on

[chan] bitmessage
Sep 23 09:42

top -H -p`cat ./singleton.lock` is better

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Sep 23 09:43

That would work for portable mode if the previous one wouldn't. Peter Surda Bitmessage core developer

[chan] bitmessage
Sep 23 10:11

rr@rr ~/AA/PATA40/fern/BMuTB/BM/BMsep17/src $ top -H -p`cat ./singleton.lock` receive q ! NSA versucht meine ports zu auszuknocken ... top - 12:09:26 up 19 min, 2 users, load average: 1,20, 1,10, 0,95 Threads: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie %Cpu(s): 14,2 us, 5,5 sy, 0,0 ni, 80,2 id, 0,0 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 3897324 total, 474144 free, 938008 used, 2485172 buff/cache KiB Swap: 0 total, 0 free, 0 used. 2599584 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5946 rr 20 0 1823772 386480 63532 R 90,0 9,9 2:45.76 ReceiveQueue_2 5935 rr 20 0 1823772 386480 63532 S 3,7 9,9 0:54.37 SQL 5945 rr 20 0 1823772 386480 63532 S 2,7 9,9 0:24.54 ReceiveQueue_1 5944 rr 20 0 1823772 386480 63532 S 2,3 9,9 0:30.38 ReceiveQueue_0 5923 rr 20 0 1823772 386480 63532 S 1,7 9,9 0:21.61 MainThread 5943 rr 20 0 1823772 386480 63532 S 1,3 9,9 0:28.25 Asyncore 5949 rr 20 0 1823772 386480 63532 S 1,3 9,9 0:05.01 AddrBroadcaster 5948 rr 20 0 1823772 386480 63532 S 0,3 9,9 0:01.76 InvBroadcaster 5933 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 addressGenerato 5934 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.05 singleWorker 5938 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:01.10 objectProcessor 5939 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:04.36 singleCleaner 5942 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 singleAPI 5947 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:01.22 Announcer 5950 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:02.50 Downloader 5951 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 QInotifyFileSys 5967 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.32 UISignaler

[chan] bitmessage
Sep 23 10:36

that q is out of whack

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 9 11:12

I doubt that maxtotalconnections handled properly. I see where it read in code but that has no effect for me: with maxtotalconnections=16 set in keys.dat PyBitmessage establishes ~ 40 connections.

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 9 11:46

Got it probably: next connection will be closed if PyBitmessage already have more than (maxtotalconnections + maxbootstrapconnections) connections, where maxbootstrapconnections defaults to 20. So if you want maximum 8 connections, set maxbootstrapconnections = 4 maxtotalconnections = 4

[chan] bitmessage
Oct 13 15:29

The simplest solution is to stick to yellow light mode and delete knownnodes.dat before starting the program. Unfortunately there are some d1cks flooding the network, this always happens with peer to peer.

[chan] bitmessage
Oct 13 20:30

set it in which file , mate ? maxbootstrapconnections = 4 maxtotalconnections = 4 where in ?

[chan] bitmessage
Oct 13 22:34

keys.dat oviously, PyBitmessage hasn't other configs.

[chan] bitmessage
Oct 14 06:52

The simplest solution is to stick to yellow light mode and delete keys.dat before starting the program.

[chan] bitmessage
Oct 14 07:04

"The simplest solution is to stick to yellow light mode and delete keys.dat before starting the program." Agreed, but this doesn't solve the problems if a person wants to use green light mode. It's too difficult to ask an average user to edit the config files manually, as this is for advanced users only. Bitmessage needs to have easier GUI controls for the amount of connections in green light mode, as well as banning nodes if they make excessive or incorrect demands on the program, which some do.

[chan] bitmessage
Oct 14 07:44

The simplest solution doesn't work. pyBM still maxes out a CPU core

[chan] bitmessage
Oct 14 08:09

... and delete keys.dat ... ?!? DUDE WTF

[chan] bitmessage
Oct 14 08:26

This is old troll trick. When discussion touches the subject of deleting KNOWNNODES.DAT, troll replaces this filename in thread text with keys.dat and has fun.

[chan] bitmessage
Oct 14 08:36

Replacing name KNOWNNODES.DAT with messages.dat is also used by trolls.

[chan] bitmessage
Oct 14 10:41

If you delete knownnodes.dat it won't do any harm and the file will be recreated the next time the program starts (trust me, this is safe as milk). Deleting anything else will screw up the program. Anyway, you could keep a copy of the old knownnodes.dat instead of deleting it. My old knownnodes.dat was 1,469 KB (well over a megabyte) and had slowed down the program to a crawl. It's a pity to have to recommend deleting knownnodes.dat, but it is the only simple way to get rid of the crap nodes that slow down the program.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 11:44

Before you delete files and fiddle around with settings whose purpose you don't understand, try the latest development code. There were significant performance optimisations. Have a nice day, Peter Surda Bitmessage core developer

[chan] bitmessage
Oct 14 11:49

Newest build I tried barks about inability to compile pow library and doesn't work at all.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 11:49

You didn't write which version you're using. Try the latest development code. If you want to stick to 0.6.2, don't change the maxbootstrapconnections, you probably don't understand what it does. Also, don't use maxtotalconnections to reduce your CPU load. It may have unintended consequences. You can use maxtotalconnections to reduce the amount of memory PyBitmessage uses. If you want to reduce the CPU load, reduce the bandwidth. But there's a high chance you're better off trying the development code instead of changing these variables. Delete these variables if you're using the development code to make sure they're reset to default values. You still may end up with PyBitmessage maxing out a core, but it will simply mean that the bottleneck shifted, you'll get a better overall performance. If you're running from source code, just pull the v0.6 branch. If you're on Windows, try the daily development snapshots: https://bitmessage.org/download/snapshots/ Have a nice day, Peter Surda Bitmessage core developer

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 12:01

You didn't write what platform you're using. Have you gone through the compiling instructions, in particular the first section which talks about automated detection of dependencies? https://bitmessage.org/wiki/Compiling_instructions#setuptools Peter Surda Bitmessage core developer

[chan] bitmessage
Oct 14 12:47

I downloaded BM on Debian through git and then simply started pybitmessage.py

[chan] bitmessage
Oct 14 12:48

Use py2dsc-deb on debian, dude: https://pypi.python.org/pypi/stdeb/0.8.5

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 12:57

> I downloaded BM on Debian through git and then simply started > pybitmessage.py This will work sufficiently well, assuming you have all the dependencies sorted out. Run checkdeps.py to check your dependencies, it supports Debian and will tell you which packages you have to install. Peter Surda Bitmessage core developer

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 13:12

If that works, that would be great. Last time I tried to create a debian package I was unsuccessful, but multiple fixes have been applied to setup.py so maybe it works now. Ideally I would like to have two packages, one for daemon and one for GUI but maybe there are other options how to allow people who don't what the GUI to install PyBitmessage without the unnecessary dependencies like pyqt. Peter Surda Bitmessage core developer

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 14 18:14

Yes, It works. At least in my Ubuntu 17.04. I just do the following: python2.7 setup.py sdist sudo apt-get install python-stdeb python-all-dev build-essential libssl-dev python-msgpack python-qt4 py2dsc-deb dist/pybitmessage-0.6.2.tar.gz sudo dpkg -i deb_dist/python-pybitmessage_0.6.2-1_amd64.deb For daemon it is probably needed some systemd unit or what's used now in Debian/Ubuntu.

[chan] bitmessage
BM-2cWy7cvHoq3f1rYMerRJp8PT653jjSuEdY

Subject Last Count
BM slow suddenly? Oct 20 10:19 3
How long until streams work? Oct 19 20:35 1
demanded difficulty of messages Oct 19 13:41 11
Python host to run bitmessage Oct 19 11:21 5
chanbot Oct 17 18:01 3
Ukraine Kiev Now Oct 17 13:47 2
how to resend pubkey Oct 17 08:02 4
What's a status of DevTalk pseudo-mailing list? Oct 17 00:54 10
Question on message decryption Oct 16 11:43 22
How they are trying to derail Bitmessage development Oct 16 07:32 4
Question on message decryption ( Edge Case Best config ) Oct 15 16:21 2
new BM mod via 0install Oct 14 21:53 4
pyBM eats too many CPU cycles Oct 14 18:14 43
Sending ACKs from hidden messages Oct 13 16:25 12
Using whatsapp, anybody? Oct 13 13:52 1
Message to Julian Assange. Oct 13 00:47 18
Future, philosophy and guns (was: Re: Message by Julian Assange) Oct 12 09:04 1
Message by Julian Assange. Oct 12 02:24 2
bitmessage implementation in any other programming language Oct 10 03:45 29
Message to a friend, with copy to Julian Assange. Oct 8 17:20 1
Advantage of using namecoin? Oct 8 01:16 3
ISS Space Station - Augmented Virtual Reality Oct 7 20:54 8
shadowchat vs. BM comparison Oct 7 13:54 14
why not use Lazarus to make a BM GUI ? Oct 5 16:02 5
unicode support in bitmessage (total msg size) Oct 4 17:33 5
Node list fuckery, mitigations? Oct 3 18:14 23
unicode support in bitmessage Oct 3 17:50 11
PGP integration with bitmessage Oct 2 19:08 23
Why getopt? Oct 2 14:42 5
monkeys with typewriters Sep 30 23:26 7
RFC: HTTP/S-based secondary peer wire protocol (proposal) Sep 30 23:07 4
Secret ANTIFA Handbook Exposes Real Agenda of AntiFa Sep 29 15:57 9
Building instructions for Windows Sep 29 07:29 25
tor browser sandbox 0.0.20 r48240 Sep 29 02:16 1
BM GUI via API using monkey studio Sep 29 01:22 19
addon to pyBM : otp crypter -- soon to be integrated into BM for extra safety ! Sep 28 22:39 2
I stick with pyBM 0.6.0 Sep 28 12:52 4
( WAS ..addon to pyBM ) Announce Sep 28 08:23 3
alternative to Qt-Designer ? none it seems Sep 27 23:44 2
(no subject) Sep 27 23:37 1
pyBM eats too many CPU cycles Sep 27 23:33 4
Why Bitmessage has waves of online activity? Sep 27 23:16 5
beamstat.com is being cached by google Sep 27 23:15 7
turn MiNode into a full client - simple task ! Sep 27 00:32 2
BitText XHKhFPCDzj: ultimate bitmessage forum Sep 26 21:53 2
GET XHKhFPCDzj Sep 26 21:44 1
28 days Sep 25 07:18 3
peer rating Sep 25 06:44 5
TTL Tweaks Sep 25 06:34 13
How to.. Sep 24 16:43 2
BitText Error Sep 24 16:32 1
BitText LIST Sep 24 16:16 1
HELP Sep 24 15:49 2
BitText ADD confirmation Sep 24 15:34 1
use MiNode py3 app to route a BM via "stream 7" Sep 24 14:07 1
BM GUI via API using KDevelop Sep 23 18:21 1
Email client integration Sep 23 16:34 10
got green light Sep 23 10:59 2
Git pull error Sep 23 08:49 13
Kdevelop + qt-designer for python Sep 22 21:47 1