pyBM eats too many CPU cycles

BM-2cU6H4GiPoJKh7E8w8gCxE9WUujuyyPd5w
Sep 22 12:54

CPU load gets to high after 12 hrs running pyBM with green light. erasing knownNodes will give you 1% CPU load for hours, with just a few connections. CPU load should not get that high ! How to throttle it for good ? limit green connection number too ?

[chan] bitmessage
Sep 22 13:47

When running green nodes be aware that there a number of negligent/malicious nodes that connect to you exhibiting some of the following behaviors: - continuously flooding you with fake nodes - advertising objects with expire times outside the acceptable range - repeatedly connect, download all objects, disconnect These can have negative impacts on network traffic, CPU load, and memory use.

[chan] bitmessage
Sep 22 13:50

I want to be able to limit connect to TLS 0.6.2 nodes and ignore 0.4.4 bullshit nodes. also I want an 8 green nodes limit. cannot be hard to do, just change a number, right ?

[chan] bitmessage
Sep 22 14:22

Actually the few 0.4.4 nodes I encountered were quite polite compared to a stock 0.6.2 node. The worst offenders are alternative implementations (wire/bmd) but attacks do come from TLS 0.6.2 as well - making decisions based on user agent string is just as useless as in http. Set maxtotalconnections to 8 more than maxoutboundconnections (note the minimum maxoutboundconnections is 8). For your other criteria you will need to modify the code where the version message is processed to close the connection when the criteria isn't met.

[chan] Crypto-Anarchist Federation
Sep 22 14:28

thanks - I will report back.

[chan] bitmessage
Sep 22 14:39

looks good so far , changed 200 to 17 like 5 times or so

[chan] bitmessage
Sep 22 14:44

well - not much if any change. 17 threads, 17% total load rescheduling renice is futile too lag is a bit too high I think, tho it still kinda is workable, but significantly slower.

[chan] bitmessage
Sep 22 15:01

Did you restart? Maybe restart is still required for maxtotalconnections to take affect.

[chan] bitmessage
Sep 22 15:06

oh my - now after 20 min. load is very low 1% , no lag, only 5 green conn. if it stays that way, its fine. then it was an effective change ... :-) thanx

[chan] bitmessage
Sep 22 15:09

and yes, I did restart & killed knownnodes.

BM-2cU6H4GiPoJKh7E8w8gCxE9WUujuyyPd5w
Sep 22 15:31

seems like abusive green nodes ate my cycles. went back to yellow light deliberately, with less than 1% load. screw it.

[chan] broadcast
Sep 22 16:01

8444 tcp & udp rule in GUFW firewall gave me green light and much lower CPU load. :-)

[chan] bitmessage
Sep 22 16:29

€urAsia has all it needs to take over planet Earth.

[chan] bitmessage
Sep 22 16:30

And what does that have to do with this thread?

[chan] bitmessage
Sep 22 20:33

but not for long. we need a better green connection throttle. this sucks.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Sep 23 07:41

It would be helpful to know which thread causes the problems. Install the prctl python module and do top -Hp `cat ~/.config/PyBitmessage/singleton.lock` Peter Surda Bitmessage core developer

[chan] bitmessage
Sep 23 08:50

If they have unique IPs, you can ban them. If they use TOR, not without blocking every good TOR user as well.

[chan] AMERICHAN
Sep 23 09:33

OK hang on

[chan] bitmessage
Sep 23 09:42

top -H -p`cat ./singleton.lock` is better

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Sep 23 09:43

That would work for portable mode if the previous one wouldn't. Peter Surda Bitmessage core developer

[chan] bitmessage
Sep 23 10:11

rr@rr ~/AA/PATA40/fern/BMuTB/BM/BMsep17/src $ top -H -p`cat ./singleton.lock` receive q ! NSA versucht meine ports zu auszuknocken ... top - 12:09:26 up 19 min, 2 users, load average: 1,20, 1,10, 0,95 Threads: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie %Cpu(s): 14,2 us, 5,5 sy, 0,0 ni, 80,2 id, 0,0 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 3897324 total, 474144 free, 938008 used, 2485172 buff/cache KiB Swap: 0 total, 0 free, 0 used. 2599584 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5946 rr 20 0 1823772 386480 63532 R 90,0 9,9 2:45.76 ReceiveQueue_2 5935 rr 20 0 1823772 386480 63532 S 3,7 9,9 0:54.37 SQL 5945 rr 20 0 1823772 386480 63532 S 2,7 9,9 0:24.54 ReceiveQueue_1 5944 rr 20 0 1823772 386480 63532 S 2,3 9,9 0:30.38 ReceiveQueue_0 5923 rr 20 0 1823772 386480 63532 S 1,7 9,9 0:21.61 MainThread 5943 rr 20 0 1823772 386480 63532 S 1,3 9,9 0:28.25 Asyncore 5949 rr 20 0 1823772 386480 63532 S 1,3 9,9 0:05.01 AddrBroadcaster 5948 rr 20 0 1823772 386480 63532 S 0,3 9,9 0:01.76 InvBroadcaster 5933 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 addressGenerato 5934 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.05 singleWorker 5938 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:01.10 objectProcessor 5939 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:04.36 singleCleaner 5942 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 singleAPI 5947 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:01.22 Announcer 5950 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:02.50 Downloader 5951 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.00 QInotifyFileSys 5967 rr 20 0 1823772 386480 63532 S 0,0 9,9 0:00.32 UISignaler

[chan] bitmessage
Sep 23 10:36

that q is out of whack

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 9 11:12

I doubt that maxtotalconnections handled properly. I see where it read in code but that has no effect for me: with maxtotalconnections=16 set in keys.dat PyBitmessage establishes ~ 40 connections.

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 9 11:46

Got it probably: next connection will be closed if PyBitmessage already have more than (maxtotalconnections + maxbootstrapconnections) connections, where maxbootstrapconnections defaults to 20. So if you want maximum 8 connections, set maxbootstrapconnections = 4 maxtotalconnections = 4

[chan] bitmessage
Oct 13 15:29

The simplest solution is to stick to yellow light mode and delete knownnodes.dat before starting the program. Unfortunately there are some d1cks flooding the network, this always happens with peer to peer.

[chan] bitmessage
Oct 13 20:30

set it in which file , mate ? maxbootstrapconnections = 4 maxtotalconnections = 4 where in ?

[chan] bitmessage
Oct 13 22:34

keys.dat oviously, PyBitmessage hasn't other configs.

[chan] bitmessage
Oct 14 06:52

The simplest solution is to stick to yellow light mode and delete keys.dat before starting the program.

[chan] bitmessage
Oct 14 07:04

"The simplest solution is to stick to yellow light mode and delete keys.dat before starting the program." Agreed, but this doesn't solve the problems if a person wants to use green light mode. It's too difficult to ask an average user to edit the config files manually, as this is for advanced users only. Bitmessage needs to have easier GUI controls for the amount of connections in green light mode, as well as banning nodes if they make excessive or incorrect demands on the program, which some do.

[chan] bitmessage
Oct 14 07:44

The simplest solution doesn't work. pyBM still maxes out a CPU core

[chan] bitmessage
Oct 14 08:09

... and delete keys.dat ... ?!? DUDE WTF

[chan] bitmessage
Oct 14 08:26

This is old troll trick. When discussion touches the subject of deleting KNOWNNODES.DAT, troll replaces this filename in thread text with keys.dat and has fun.

[chan] bitmessage
Oct 14 08:36

Replacing name KNOWNNODES.DAT with messages.dat is also used by trolls.

[chan] bitmessage
Oct 14 10:41

If you delete knownnodes.dat it won't do any harm and the file will be recreated the next time the program starts (trust me, this is safe as milk). Deleting anything else will screw up the program. Anyway, you could keep a copy of the old knownnodes.dat instead of deleting it. My old knownnodes.dat was 1,469 KB (well over a megabyte) and had slowed down the program to a crawl. It's a pity to have to recommend deleting knownnodes.dat, but it is the only simple way to get rid of the crap nodes that slow down the program.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 11:44

Before you delete files and fiddle around with settings whose purpose you don't understand, try the latest development code. There were significant performance optimisations. Have a nice day, Peter Surda Bitmessage core developer

[chan] bitmessage
Oct 14 11:49

Newest build I tried barks about inability to compile pow library and doesn't work at all.

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 11:49

You didn't write which version you're using. Try the latest development code. If you want to stick to 0.6.2, don't change the maxbootstrapconnections, you probably don't understand what it does. Also, don't use maxtotalconnections to reduce your CPU load. It may have unintended consequences. You can use maxtotalconnections to reduce the amount of memory PyBitmessage uses. If you want to reduce the CPU load, reduce the bandwidth. But there's a high chance you're better off trying the development code instead of changing these variables. Delete these variables if you're using the development code to make sure they're reset to default values. You still may end up with PyBitmessage maxing out a core, but it will simply mean that the bottleneck shifted, you'll get a better overall performance. If you're running from source code, just pull the v0.6 branch. If you're on Windows, try the daily development snapshots: https://bitmessage.org/download/snapshots/ Have a nice day, Peter Surda Bitmessage core developer

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 12:01

You didn't write what platform you're using. Have you gone through the compiling instructions, in particular the first section which talks about automated detection of dependencies? https://bitmessage.org/wiki/Compiling_instructions#setuptools Peter Surda Bitmessage core developer

[chan] bitmessage
Oct 14 12:47

I downloaded BM on Debian through git and then simply started pybitmessage.py

[chan] bitmessage
Oct 14 12:48

Use py2dsc-deb on debian, dude: https://pypi.python.org/pypi/stdeb/0.8.5

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 12:57

> I downloaded BM on Debian through git and then simply started > pybitmessage.py This will work sufficiently well, assuming you have all the dependencies sorted out. Run checkdeps.py to check your dependencies, it supports Debian and will tell you which packages you have to install. Peter Surda Bitmessage core developer

BM-2cX62WCeFcUwzXWqxTBfaAzNy4j1y8yZVm
Oct 14 13:12

If that works, that would be great. Last time I tried to create a debian package I was unsuccessful, but multiple fixes have been applied to setup.py so maybe it works now. Ideally I would like to have two packages, one for daemon and one for GUI but maybe there are other options how to allow people who don't what the GUI to install PyBitmessage without the unnecessary dependencies like pyqt. Peter Surda Bitmessage core developer

BM-NBCPnifzmbz6aRN51nwMedSfRbVBfqYz
Oct 14 18:14

Yes, It works. At least in my Ubuntu 17.04. I just do the following: python2.7 setup.py sdist sudo apt-get install python-stdeb python-all-dev build-essential libssl-dev python-msgpack python-qt4 py2dsc-deb dist/pybitmessage-0.6.2.tar.gz sudo dpkg -i deb_dist/python-pybitmessage_0.6.2-1_amd64.deb For daemon it is probably needed some systemd unit or what's used now in Debian/Ubuntu.

[chan] bitmessage
BM-2cWy7cvHoq3f1rYMerRJp8PT653jjSuEdY

Subject Last Count
mass extinction of BM users Dec 14 18:46 4
force zeronet trough tor only? Dec 14 18:45 9
please post your onion and uptime in UTC London time in this list Dec 14 16:43 2
zeromail QUESTION Dec 14 13:07 12
torIRC QUESTION Dec 14 12:40 3
RICOCHET QUESTION Dec 14 12:34 15
the green light Dec 14 00:43 21
Safe chat software? Dec 13 23:07 18
torIRC 2.33 Dec 13 11:02 2
anyone uses Nyx for tor ? Dec 12 15:22 1
torIRC TBB-9150 version via port 11009 as always Dec 12 15:16 2
update torIRC , TBB-9150 version via port 1801 instead of 11009 Dec 12 09:28 1
torIRC , TBB-9150 version via port 1801 instead of 11009 Dec 12 09:14 1
torIRC latest - TBB-9150 version via port 1801 instead of 11009 Dec 12 09:13 7
bm as hidden service with stem Dec 12 03:11 10
sent via API : torIRC9050.py Dec 11 12:59 1
run bm in cloud: https://mybinder.org Dec 11 09:12 6
BitText XHKhFPCDzj: ultimate bitmessage forum Dec 11 03:10 1
BitText LIST Dec 11 02:39 1
Jupyter BM notebook with py2.7 Dec 11 00:34 1
firewall vs. pyBM green mode Dec 10 09:33 8
BREXIT IS BULLSHIT Dec 10 02:34 3
run bm in cloud Dec 9 19:13 6
The Stallman Tax Dec 9 14:47 1
using eclipse for pyBM & github Dec 7 15:28 7
BM w/ GUI : Jupyter BM notebook Dec 7 15:19 10
BM in python3? Dec 6 17:52 8
Listen for incoming on Tor hidden service Dec 6 15:43 12
questionable attitude Dec 6 08:09 21
why are all the channels blank Dec 6 06:24 11
Feature idea Dec 5 23:53 5
include the mammoth into pyBM source code Dec 5 11:18 1
include the mammoth into pyBM source code Dec 5 11:15 2
running the 800+ chan super mammoth Dec 5 10:20 32
the myth of the "secret chans" on BM Dec 4 20:23 1
[chan] DevTalk Dec 4 20:13 6
Hacking Bitmessage Dec 4 19:11 3
The complete full mammoth "keys.dat" file - replace your KEYS.DAT in pyBitMessage with this 800+ chan text file -- run the super mammoth ! Dec 4 16:23 2
clean up pyBM github landing page Dec 3 20:28 14
audit - squash this lil bug Dec 3 20:04 1
Error Dec 3 18:47 1
34C3 Dec 3 18:44 3
48 Dirty Little Secrets Cryptographers Don't Want You to Know Dec 3 17:50 1
mailchuck = OK ; onionmail = problematic Dec 3 16:27 6
mammoth chan list .py Dec 3 06:53 3
SUPER MAMMOTH -- 806 unique BM chans in one keys.dat file Dec 3 01:39 1
questionable attitude web.archive.org/web/20171202215905/http://bitmessage.mybb.im/viewtopic.php?id=30%23p106 Dec 2 22:42 1
ACK : Persistence via Dead Letter drop clearnet access w/Op separation Dec 2 12:42 1
OMEGA release 39 is available for download Dec 2 05:15 1
I get no reply from mailchuck upon registration -- fixed ! all working again. Dec 2 02:13 4
beamstat.com is fucking fast to serve broadcasts Dec 2 01:47 1
27'000 users who use OnionMail - more than BM ? sure ... Dec 1 10:40 1
beamstat.com is fucking slow to serve broadcasts Dec 1 10:34 4
Patch 4 Dec 1 10:31 5
crowdfunding : whack Trump Dec 1 10:31 11
27'000 users who use OnionMail - more than BM ? Dec 1 07:16 10
dotsendack =True *** stealth mode Dec 1 07:16 5
Pubkey request Dec 1 07:15 6
NEW ! subscribe to this pml : BM-NB4TuMJmrHeo1p1FsfuMCrkZBUPGLjnq Dec 1 07:15 7
Bitmessage feature request for timing attack mitigation Dec 1 07:15 14
How to disable ack packets Nov 30 13:07 36
Easter egg Nov 30 12:13 2
I get no reply from mailchuck upon registration Nov 30 11:40 8
easy mailer , just add MX line in DNS Nov 30 10:59 2
Feature request Nov 30 08:44 3
Bitmessage to Shell Nov 30 05:01 4
Patch 3 Nov 30 04:36 1
Patch 2 Nov 30 04:32 1
Patch Nov 30 04:18 1
Long delay making connections Nov 29 09:44 25
onion mail Nov 29 00:56 1
Flatpak Linux Distribution for Bitmessage Nov 28 20:19 1
sue corporations like stman sues postal services Nov 28 13:11 1
backbone member Nov 28 13:10 9
pyBM light load even on old laptops Nov 28 07:37 2
BM now with ding dong Nov 27 03:05 3
encryptpad Nov 27 02:52 2
The Homosexual Perversion: A Jewish Criminal Simhke for the Postmodern Corporate Conformist Nov 25 03:04 1
where is latest mammoth BM chan list ? larger than beamstat.com ? Nov 24 12:21 3
wayback archive for beamstat Nov 24 01:35 2
Where is PyBitmessage git signing key fingerprint? Nov 23 23:10 16
pyBM eating too many cycles still ----- fixed ! much better ! Nov 23 22:06 1
I'm back. Nov 23 21:50 3
pyBM eating too many cycles still Nov 23 21:34 4
wikileaks vault 8 Nov 23 17:02 3
very few active chans Nov 23 01:10 17
audit BM via web GUI Nov 22 20:37 6
But I'm using Tor!! Nov 22 12:38 2
LUKS! Nov 22 12:12 1
New Bitmessage CLI Nov 22 00:22 2
mailchuck question Nov 21 08:52 2
Incoming connections Nov 21 02:18 5
(no subject) Nov 20 17:07 7
(no subject) Nov 20 13:14 4
BitMessage crash Nov 19 12:50 7
Alternative Bitmessage port for official assignment with IANA? Nov 19 08:28 5
Tor replacement Nov 18 23:04 5
codewordtest2 Nov 17 21:52 1